Hourglass is a synthetic real-time application that can be used to
learn how CPU scheduling in a general-purpose operating system works
at microsecond and millisecond granularities. Its distinguishing
It creates very detailed map of when each Hourglass thread has access
to the CPU
It supports multiple thread execution models; e.g. periodic and
It acts as an abstraction layer for threading, timing, and CPU
scheduling functionality on Unix- and Win32-based systems
about Hourglass was presented at the FREENIX track of the USENIX
Technical Conference in Monterey, CA in June 2002. The paper
contains fairly detailed information about how Hourglass works, and
how to use it.
Doug Locke mentioned Hourglass in an
article at Linuxdevices.
Hourglass is distributed under a BSD-style license:
If you find Hourglass to be useful, please drop me a line
letting me know. Better yet, also let me know who you are, where
you work, and what you are using it for. I'm very busy doing
professor types of things and feedback like this is the only thing
that will motivate me to produce new releases.
This release should compile on any reasonably recent Linux or
Cygwin installation. On FreeBSD, Linuxthreads are required.
Hourglass has been tested on MacOS X, Fedora Core 6, various versions
of Redhat Linux,
FreeBSD-4.5, and Windows
2000 and XP. Pentium-class Intel and AMD chips are supported, as are
Old releases of Hourglass are available here, but I do not expect them to be of much
As part of the Bossa
project, Julia Lawall
has modified Hourglass to work with Bossa. The modified version can
be downloaded from their site.
paper contains fairly detailed information about how to use
Hourglass. Also, running Hourglass without any options causes it to
print some usage information. A real user manual would be nice, but
who has time to write one?
This figure shows 25ms segments of three execution traces taken by
Hourglass on a Linux box while (1) it receives full-bandwidth data
over 100 Mbps Ethernet, and (2) a real-time thread attempts to receive
4ms of CPU time during each 5ms period. This models a demanding
application such as professional quality real-time audio processing.
On plain Linux, receive processing interferes with the thread to a
large extent, and it misses lots of deadlines. On Linux patched with
Robert Love's preemptible kernel and lock breaking patches, the 5ms
periodic structure of the application is more apparent, and although
there is still a lot of interference the application meets most of its
deadlines. The TimeSys Linux/GPL kernel almost completely respects
the user's choice to run a thread in preference to kernel activity,
and therefore in this case all deadlines are met. This example and
lots of others are covered in the Hourglass FREENIX paper.
Q: Why does Hourglass print "oops: no trace records left -
A: Unfortunately, there are a number of reasons why this can
occur. The first possibility is that Hourglass is legitimately
running out of trace buffers -- this can happen on long Hourglass
runs, or on runs taken where many gaps are detected. The number of
trace records can be increased using the -e command line
The second possibility is that something is going wrong, causing
Hourglass to eat up many more trace records than it should. First,
make sure that Hourglass is correctly estimating the frequency of the
timestamp counter on your processor -- look at the reported MHz value
near the top of the Hourglass output. Second, on slow computers
and/or when Hourglass is compiled without optimizations, it is
possible that Hourglass is incorrectly detecting gaps between
successive iterations of the execution trace loop -- this will very
quickly eat up the trace record buffer. To fix this, first compile
Hourglass with "-O3" to gain the benefits of inlining in the critical
loop of the execution trace algorithm, and second, consider increasing
the gap threshold value using the "-g" command line option. This
threshold determines the smallest gap between successive iterations of
the execution trace loop that will be logged to the trace buffer. Its
default value of 5us seems to be a good choice on modern machines that
run at a few hundred MHz or more. The short document CALIBRATING in
the Hourglass root directory contains some more detail about
Q: Why is Hourglass not correctly determining the frequency of
the timestamp counter on my machine?
A: Some processors dynamically change their clock speed, and
unfortunately this sometimes affects the timestamp counter as well.
Laptops are particularly problematic. There's probably no portable
solution to this problem, so consider using the "-M" command line
option to manually set the frequency of the timestamp counter. In
general this manual fix is only useful if your processor actually has
a constant-rate timestamp counter but Hourglass somehow fails to
measure it accurately. If your CPU varies the frequency of the
timestamp counter during an Hourglass run, then Hourglass will
silently return incorrect results.
Q: I'm trying to create a graphical execution trace but jgraph
doesn't seem to be working. What's up?
A: Try recompiling jgraph with -DLCC on the cc command
Q: On RedHat Linux 7.3 why do threads that are assigned
real-time priority not seem to be running in preference to threads in
the time-sharing class?
A: The kernel used by RedHat 7.3 (2.4.18-4) is broken with
respect to SCHED_FIFO and SCHED_RR. The function sched_setscheduler()
silently fails to have any effect on this kernel. I suggest
downloading and building a fresh 2.4.18 kernel. Also, it would not
hurt to send a complaint to RedHat -- silent failure of real-time
features is a bad thing!