cpubroker man page




SYNOPSIS

       The  CPU  Broker  is a reservation-based resource manager for CPU time.
       The broker mediates between multiple  real-time  applications  and  the
       facilities of an underlying real-time operating system: it implements a
       configurable CPU management policy, monitors resource use, and  adjusts
       allocations  over  time.   This  architecture  helps to ensure that the
       quality of service (QoS) needs of "important" applications  are  satis-
       fied,  especially  when the relative importance of applications changes
       dynamically.  An additional and important benefit is  that  the  broker
       can  automatically  determine  many of the scheduling parameters of its
       client applications.

       This document provides an overview of the use and implementation of the
       CPU  Broker.  The first section below is a "quick start" example of the
       Broker in action.  The second section summarizes the implementation  of
       the Broker, and introduces important terminology.  The third section is
       a brief tutorial on writing applications that interface  with  the  CPU
       Broker.



QUICK START

       [Note:  This  demo requires most of the software dependencies described
       in the README: TimeSys Linux, ACE/TAO, QuO, gkrellm v2, etc.]

       [Note 2: On TimeSys v4.1 you will need to run these  programs  as  root
       because  creating  and  manipulating  reservations  is now a privileged
       operation.]

       If you would like to quickly see the Broker in action, you can run  the
       following  commands  in  separate terminals to start the Broker, a syn-
       thetic CORBA server, and its client:

       [nemo@anemone /tmp] broker_allup_default
       Done!

       [nemo@anemone /tmp] rktimes -n rt1 -p 8080 -- rt_server -n rt1
       CPU frequency: 851.947Mhz
       RT server ready, waiting for connection from rt_client...

       [nemo@anemone /tmp] rt_client

       You can then monitor the CPU usage and reservation for rt_server  using
       gkrellm version two (available from http://www.gkrellm.net).

       [nemo@anemone /tmp] gkrellm -s localhost -P 8080

       The  gkrellm window should display two "CPU" graphs: "CPU 0" is the CPU
       usage of the rt_server program, and "CPU 1" is the reservation made  by
       the  broker  for that application.  (Note that the labels are below the
       graphs, not above.)  Furthermore, the "CPU 1" graph  shows  the  unused
       portion  of  the  reservation in green and overlays the used portion in
       cyan.  Both graphs should show square waves after  a  while,  with  the
       reservation  being  slower to drop off.  The important thing to note is
       that the reservation changes over time to meet the needs of the  appli-
       cation.   This  is  different from an ordinary reservation-based system
       without brokering, in which the application's  reservation  would  nor-
       mally remain constant from the beginning to the end of execution.

       Going   through   each   step   in  detail,  the  first  program,  bro-
       ker_allup_default, starts a process to house most of the broker objects
       and  connect  the  objects in a default configuration.  Next we start a
       program, rktimes, that monitors rt_server and sends updates to  gkrellm
       though  TCP  port 8080.  The rt_server is a synthetic CORBA server that
       will consume 250 or 500 milliseconds  of  CPU  time  every  second  and
       reports  this  usage to the Broker.  Finally, we start rt_client, which
       connects to rt_server and periodically does  an  RPC  that  causes  the
       server to actually consume CPU time.

       A  primary  feature  of  the  Broker  is its ability to manage multiple
       applications and arbitrate between them when there is  CPU  contention.
       Continuing  with the previous (and running) example, the following com-
       mands will demonstrate the Broker's ability to  deal  with  contention.
       First,  we  create some (artificial) contention by decreasing the total
       amount of CPU the Broker will reserve for applications from 75% to 60%:

       [nemo@anemone /tmp] cbhey file://strict_policy.ior set max-cpu to 0.60

       The  cbhey  command  is an extremely flexible tool that can communicate
       with different parts of the Broker; we will be using it extensively  in
       later  examples.   Next,  we  add a new application by starting another
       rt_server, rt_client, and gkrellm.

       [nemo@anemone /tmp] rktimes -n rt2 -p  8081  --  rt_server  -n  rt2  -o
       rtserver2.ior 200ms


       [nemo@anemone /tmp] rt_client -s file://rtserver2.ior


       [nemo@anemone /tmp] gkrellm -s localhost -P 8081

       This new application consumes 200 milliseconds of CPU time every period
       (1 second) and its reservation should also be around  200  milliseconds
       per  second.   The  contention  created  by  this  new  application  is
       reflected in the gkrellm graph for "rt1" as it  is  receiving  a  lower
       reservation  than  before.   This occurs because both applications have
       the same level of importance and the Broker cannot decide which one  to
       favor, so it arbitrarily chooses "rt2".  You can change this by setting
       the priority of one of the  applications,  referred  to  as  "rt1"  and
       "rt2".  For example, to bump the priority of "rt1" to one:

       [nemo@anemone /tmp] cbhey file://strict_policy.ior set priority of task
       rt1 to 1

       The graph for "rt1" should return to normal and  the  graph  for  "rt2"
       should  show the Broker automatically dropping the reservation when the
       reservation for "rt1" increases.  You might try raising the priority of
       "rt2" above "rt1" to see what happens.



IMPLEMENTATION

       The  CPU  Broker  is  implemented as a CORBA-based server that receives
       resource requests from soft real-time applications  and  which  "trans-
       lates" those into real reservations.  An application that uses the Bro-
       ker generates a request at the end of (some or all of) its  task  peri-
       ods,  when it knows of how much CPU it will need in future periods.  An
       application can be modified so that it communicates  with  the  Broker;
       alternatively, an unmodified application can be managed through the use
       of a wrapper program, like proc_advocate(1), albeit  at  some  loss  in
       precision.   An application's request, along with any external data, is
       fed into the Broker's adaptation strategies.  Adaptation strategies for
       individual  applications  pass  their  advice to a global strategy that
       resolves contention for the CPU.  Finally, when all of the  adaptations
       have  been computed, the applications' CPU reservations are adjusted by
       the Broker.

       Communication between applications (or their wrappers) and the CPU Bro-
       ker is done through remote method invocations on CORBA objects.  At the
       moment there are three main classes of Broker objects:

       Manager
              A Manager is a central object used to track  other  objects  and
              route  messages  between them.  Typically there is only a single
              manager per machine.

       Task/Advocate
              A Task object is the glue between the applications, the  Broker,
              and  the real-time operating system's reservation scheme.  Advo-
              cates wrap Task objects and perform application-specific adapta-
              tions.

       Policy A Policy object makes global decisions about how to resolve con-
              tention for the CPU.  These objects take advice from  the  Advo-
              cates and make the final decisions about how much CPU time every
              managed process actually receives.

       The standard implementations of these classes and  their  support  code
       are contained in several shared libraries:

       libcpu-broker
              Contains  both a straightforward implementation of a Manager and
              the base Task object, called RealTimeTaskImpl.

       libtask_factories/libdelegates
              Contains a variety of reusable Advocates.

       libpolicies
              Contains the StrictPolicy object,  which  implements  a  "strict
              priority"  contention policy that favors high-priority processes
              when making reservations.  For example, if all of  the  CPU  has
              been  allocated  and a high-priority process requests more time,
              the StrictPolicy will reduce the reservations of lower  priority
              processes.   In  contrast,  a low-priority process will have its
              request lowered (and continually readjusted) to use whatever CPU
              time is currently available.

       The libraries contain the bulk of the CPU Broker's code and are used by
       the tools listed below:

       broker_allup
              A "super server"  that  houses  the  shared  libraries  and  the
              objects that they create.

       cbhey  A  "super client" that can be used to communicate with the vari-
              ous Broker objects.

       proc_advocate
              A wrapper that allows unmodified applications to be  managed  by
              the Broker.

       rktimes
              A generic utility for TimeSys Linux that monitors a resource set
              and performs some of the jobs of rkexec(1).

       Learning to wield these tools, especially cbhey(1), is key to using and
       configuring  the Broker to behave as desired in a variety of scenarios.
       Consult the man pages for more information about these tools, and  read
       the  Doxygen  generated documentation in the install directory for more
       details on the Broker's classes.

       In addition to using the existing tools, effective use of the CPU  Bro-
       ker  may require modifying applications to communicate with the Broker,
       or writing new Broker objects that implement application-specific adap-
       tation  policies.   The  remainder of this man page is a brief tutorial
       about writing applications that interface with the CPU Broker.



TUTORIAL

       This tutorial gives you a detailed example of creating Broker  objects,
       creating  your  own  Broker  classes, and modifying an application.  To
       fully complete this tutorial, you will need the following  software  in
       addition  to  the  Broker: TimeSys Linux, ACE/TAO, QuO, OmniORB, python
       v2, gkrellm v2, etc...

       First, start the broker_allup_default  process  again,  if  it  is  not
       already  running.  This executable is just a shell script that uses the
       Broker's tools to construct a default  configuration  consisting  of  a
       single  Manager  object  that uses a StrictPolicy.  Feel free to peruse
       the script to see how it works.

       The application we will be using is the rt_server, like  in  the  QUICK
       START  section.   However,  this time we will be providing our own Task
       object.  Creating the object is simply a matter of  asking  the  shared
       library  referenced  by  "cb.ior"  and  contained  in  the broker_allup
       process:

       [nemo@anemone /tmp] cbhey file://cb.ior create real-time-task with name
       rt1 > rt1-rtt.ior

       This  command  will  construct an RealTimeTaskImpl object with the name
       "rt1" and print its IOR on standard out.  The name is  used  to  easily
       identify  our  application  and will be the name of the RK resource set
       that holds the application's processes.  Next, we  create  an  Advocate
       object  that  does  no adaptation, but gives us a place to insert other
       advocates:

       [nemo@anemone /tmp] cbhey file://delegates.ior  create  real-time-task-
       delegate > rt1.ior

       This  new  object acts as a delegate that passes all method calls on to
       another object.  Initially, it has nothing to delegate to,  so  you  we
       need  to  set  the  "remote-object"  attribute  to  point  at the Real-
       TimeTaskImpl we just created.

       [nemo@anemone   /tmp]   cbhey    -c    IDL:BrokerDelegates/Delegate:1.0
       file://rt1.ior set remote-object to file://rt1-rtt.ior

       This  command  line  introduces  some  more cbhey syntax so lets take a
       moment to examine the command in more detail.  The purpose of cbhey  is
       to  provide  high-level access to the Broker's objects through a common
       command line syntax.  This approach  makes  it  easier  to  avoid  hard
       wiring object configurations in applications since a user can construct
       arbitrary graphs of objects just prior to execution time.   These  user
       defined  configurations  can then be wrapped in a shell script for ease
       of use, like broker_allup_default. The command line syntax  is  derived
       from  AppleScript's "tell" syntax, wherein a script communicates with a
       particular "property" inside of a server.  In our case, the server is a
       CORBA  object referenced by its IOR and the properties are the object's
       methods and attributes.  For example, in the line above we are  setting
       the  "remote-object"  attribute  of "./rt1.ior" to an object reference.
       The  reverse  is  also  possible,  we  can  "get"  the  "remote-object"
       attribute  and  have  the IOR we just set printed to standard out.  The
       only departure from the English-like syntax is the "-c" option which is
       used  to  perform  a "cast" to another IDL type.  This cast is required
       because there are cases  where  cbhey  is  unable  to  figure  out  the
       object's  type  and  needs  a hint from the user.  See the man page for
       more information and examples.

       With our Advocate and Task object configuration now constructed we  can
       start our server, client, and gkrellm.  Notice that we pass a different
       option, "-t", to the rt_server.  This option informs  the  server  that
       use an externally created Advocate instead of creating its own.

       [nemo@anemone   /tmp]   rktimes   -n   rt1  -p  8080  --  rt_server  -t
       file://rt1.ior

       [nemo@anemone /tmp] rt_client

       [nemo@anemone /tmp] gkrellm -s localhost -P 8080

       At this point, you may notice that the reservation being  made  by  the
       Broker  is  not the same as in the QUICK START section.  The difference
       is due to the fact that neither the RealTimeTaskImpl nor  the  advocate
       are  performing  any  adaptations.   Instead, they are just passing the
       value requested by the application.  To get the same effect  as  before
       we need to create a MaxDecayTaskAdvocate:

       [nemo@anemone  /tmp]  cbhey file://task_factories.ior create max-decay-
       task-advocate > rt1-mdta.ior

       This advocate's strategy is to advise the Broker to set the reservation
       to  the  highest  value seen in the previous five periods.  This policy
       works well in general since it can quickly adapt  to  spikes  in  usage
       without  over-provisioning for long periods of time.  Just like before,
       we need to connect the object before using it:

       [nemo@anemone   /tmp]   cbhey    -c    IDL:BrokerDelegates/Delegate:1.0
       file://rt1-mdta.ior set remote-object to file://rt1-rtt.ior

       We  can  then  insert  our  new  advocate and observe the change in the
       reservation (there is no need to stop/start the application):

       [nemo@anemone   /tmp]   cbhey    -c    IDL:BrokerDelegates/Delegate:1.0
       file://rt1.ior set remote-object to file://rt1-mdta.ior

       Creating  your  own delegate is a relatively simple process, you simply
       overload a single method, adjust one of its parameters,  and  call  the
       same method on the wrapped object.  You can make a simple advocate that
       doubles the application's original  request  by  saving  the  following
       python code in the file "MyDelegate.py":

              import sys
              from omniORB import CORBA, PortableServer

              from RealTimeTaskDelegateImpl import *

              class MyDelegate(RealTimeTaskDelegateImpl):
                  def PassCPU(self, rtt, status, advice, krp):
                      # Call the wrapped object
                      advice.Compute = advice.Compute * 2
                      return self.myRemoteObject.PassCPU(rtt,
                                                         status,
                                                         advice,
                                                         krp)
                  pass

              orb = CORBA.ORB_init(sys.argv, CORBA.ORB_ID)
              poa = orb.resolve_initial_references("RootPOA")

              di = MyDelegate()
              do = di._this()

              file = open(sys.argv[1], "w")
              file.write(orb.object_to_string(do))
              file.close()

              poaManager = poa._get_the_POAManager()
              poaManager.activate()

              orb.run()

       This  program  creates a subclass of a Broker-provided class, instanti-
       ates an object of this subclass, and prints its IOR  on  standard  out.
       You can run it like so:

       [nemo@anemone /tmp] /usr/local/bin/python MyDelegate.py rt1-md.ior

       Again, we connect everything and watch the reservation behavior change:

       [nemo@anemone   /tmp]   cbhey    -c    IDL:BrokerDelegates/Delegate:1.0
       file://rt1-md.ior set remote-object to file://rt1-rtt.ior

       [nemo@anemone    /tmp]    cbhey   -c   IDL:BrokerDelegates/Delegate:1.0
       file://rt1.ior set remote-object to file://rt1-md.ior

       At this point we will create our own application that is managed by the
       Broker, so stop the rt_server and rt_client we started above.  This new
       application will also be synthetic, like rt_server, but it should  give
       you  a  good  idea  of what is required of an application that needs to
       communicate with the Broker.  Read and copy the following  python  code
       into the file "brapp.py":

              import os
              import sys

              from resource import *
              from signal import *
              from omniORB import CORBA, PortableServer

              import Broker

              orb = CORBA.ORB_init(sys.argv, CORBA.ORB_ID)
              obj = orb.string_to_object(sys.argv[1])
              # Get a reference to the Manager.
              man = obj._narrow(Broker.Manager)
              obj = orb.string_to_object(sys.argv[2])
              # Get a reference to the Advocate we should use.
              adv = obj._narrow(Broker.RealTimeTask)

              # Get objects required to make CPU reports.
              cr = Broker.CPUReserve(0, 0)
              krp = []

              def noop(sig, fr):
                  return

              signal(SIGALRM, noop)

              # Get our current resource usage for comparison
              # later on.
              last_usage = getrusage(RUSAGE_SELF)

              # Setup our scheduling parameters
              sp = [
                  # Period of one second.
                  Broker.NamedValue("period",
                                    CORBA.Any(CORBA.TC_string,
                                              "1s")),
                  # The process ID to manage.
                  Broker.NamedValue("pid",
                                    CORBA.Any(CORBA.TC_long,
                                              os.getpid()))]

              # Add ourselves to the manager.
              man.AddTask(adv, sp)
              try:
                  alarm(1.0)
                  while 1:
                      pause() # Wait for a signal
                      alarm(1.0) # Setup a periodic signal

                      # Do something...
                      print "Hello, World!"
                      for x in range(0, 1000):
                          for y in range(0, 50):
                              pass
                          pass

                      # Get our current usage,
                      usage = getrusage(RUSAGE_SELF)

                      # ... subtract our previous usage, and
                      total = ((usage[0] + usage[1]) -
                               (last_usage[0] + last_usage[1]))

                      # ... report it to the broker (in
                      # microseconds).
                      cr.Period = 1000000
                      cr.Compute = int(total * 1000000)
                      adv.ReportCPU(cr, cr, krp)
                      last_usage = usage
                      pass
                  pass
              finally:
                  # Always be sure to remove ourselves
                  # from the manager.
                  man.RemoveTask(adv)
                  pass

       Start your new application by running:

       [nemo@anemone  /tmp]  /usr/local/bin/python  brapp.py `cat manager.ior`
       `cat rt1.ior`

       If everything worked you should see "Hello, World!" being  printed  out
       every second.

       Every  application  that  uses the Broker follows the same basic steps:
       (1) construct the scheduling parameters, (2) add the  Advocate  to  the
       Manager using AddTask(), (3) periodically call ReportCPU() on the advo-
       cate, and (4) remove the Advocate from the  Manager  with  RemoveTask()
       when finished.  The first step, constructing the scheduling parameters,
       depends on the timing requirements of your application.  For example, a
       video  viewer  might have a period of 1/30th of a second.  Applications
       that do not have a natural period, like servers, should be set  to  the
       smallest  period  of its clients, or some value that results in accept-
       able timing behavior.  The process ID that is specified  need  only  be
       one  that  is  in  the resource set to be managed.  Thus, if you have a
       multi-threaded application, you can ensure the other threads are  added
       to  the  set  by creating them after the advocate is added.  If this is
       not possible, start the application already in  a  resource  set  using
       rkexec(1)  or  rktimes(1).   The  broker  will  then  use  the existing
       resource set instead of creating one.  The last part of  initialization
       is  passing  the  advocate and the scheduling parameters to the manager
       using AddTask().

       Once the application is added to the Broker, it can begin its  periodic
       processing  and  call ReportCPU().  In the above example, we figure out
       the CPU usage of our process using  getrusage(2)  and  then  send  that
       value,  in  microseconds,  to the advocate.  Notice that this only gets
       the resource usage for the current thread and not  the  others  in  the
       process.   To do this properly, we would have to call a Broker provided
       utility function, rk_resource_set_get_usage(3), that computes the usage
       for  all  of  the  processes/threads in a resource set.  Unfortunately,
       there is no python binding for this C function, so we have  to  compro-
       mise and use getrusage.

       Finally,  when  the periodic processing of an application terminates we
       have to make sure to remove our advocate  from  the  manager  so  other
       tasks  can use the newly freed resources.  In case something goes cata-
       strophically wrong and the application cannot perform the cleanup,  the
       Broker  should be able to repair itself, at least partially.  Any other
       inconsistencies left by such an occurrence  can  be  repaired  manually
       using cbhey and rkdestroyRS.

       This concludes the tutorial, to learn more about the commands and func-
       tions used, see the man pages listed below.



SEE ALSO

       broker_allup(1), broker_allup_default(1),  cbhey(1),  proc_advocate(1),
       rkexec(1),        rktimes(1),        rt_server(1),        rt_client(1),
       rk_resource_set_get_by_name(3),           rk_resource_set_get_usage(3),
       gkrellm(1)


AUTHOR



Man(1) output converted with man2html