Time-travel in Closed Distributed Systems

Anton Burtsev, Prashanth Radhakrishnan, Mike Hibler and Jay Lepreau

University of Utah

November 2006

Timetravel Project Logo

This project has several parts, and will provide several features in the resulting software. Most software features will primarily benefit Emulab users, but one (branching storage) is useful and functional outside of Emulab.

Stateful swapout of Emulab experiments: In other words, when you leave for the night, Emulab will be able to save the full running state of your entire experiment, including memory, processor, local disk, and consistent network state. Then just swap it all back in when you start working again! This is a step along the way to full time travel. Almost all the major parts are done; they just need to be knit together and deployed. Yes, performance will be a challenge.
Time travel in closed distributed systems: The ability to replay an execution is a powerful aid in analyzing and debugging long-running distributed applications. Our goal is to time-travel a network of hundreds or thousands of virtual machines spread across multiple physical hosts in the Emulab testbed environment. Our time-travel system will provide the ability to navigate through the original execution history, replay it from an arbitrary point, and--we believe--even change its state without proscribing further replay.
Many of the pieces of this are working, too-- although good performance will of course be a challenge. The primary usage scenario is debugging your system in Emulab. Run it until it goes bad, asking Emulab to take periodic checkpoints along the way. Back up, move around in time, until you zero in on the problem. Then mutate the state of the system just as you would in a normal debugger, and continue to observe it.
Branching storage: Both time travel and another Emulab-related project, the Experimentation Workbench, require support for storing versioned data in the form of trees. A storage system with builtin support for "branching" most naturally provides the tree semantics. We have extended the Linux LVM block storage system to do this. Code will be distributed soon.

For this project we build on the XEN VM operating system, consistent checkpointing, Linux LVM, extensions to our "Frisbee" disk saving/imaging system, and of course extensions to Emulab.

Documents: