Time-travel in Closed Distributed Systems
Mike Hibler and
University of Utah
This project has several parts, and will provide several features in
the resulting software. Most software features will primarily benefit
Emulab users, but one (branching storage) is useful and functional
outside of Emulab.
For this project we build on the XEN VM operating system,
consistent checkpointing, Linux LVM, extensions to our "Frisbee" disk
saving/imaging system, and of course extensions to Emulab.
- Stateful swapout of Emulab experiments: In other words, when you leave
for the night, Emulab will be able to save the full running state of your entire
experiment, including memory, processor, local disk, and consistent network state.
Then just swap it all back in when you start working again!
This is a step along the way to full time travel. Almost all the
major parts are done; they just need to be knit together and deployed.
Yes, performance will be a challenge.
- Time travel in closed distributed systems:
The ability to replay an execution is a powerful aid in analyzing and
debugging long-running distributed applications. Our goal is to
time-travel a network of hundreds or thousands of virtual machines spread
across multiple physical hosts in the Emulab testbed environment. Our
time-travel system will provide the ability to navigate through the
original execution history, replay it from an arbitrary point, and--we
believe--even change its state without proscribing further replay.
Many of the pieces of this are working, too-- although good
performance will of course be a challenge. The primary usage scenario
is debugging your system in Emulab. Run it until it goes bad, asking
Emulab to take periodic checkpoints along the way. Back up, move
around in time, until you zero in on the problem. Then mutate the
state of the system just as you would in a normal debugger, and
continue to observe it.
- Branching storage:
Both time travel and another Emulab-related project, the
require support for
storing versioned data in the form of trees. A storage system with
builtin support for "branching" most naturally provides the tree
semantics. We have extended the Linux LVM block storage system
to do this. Code will be distributed soon.