Preparing your VMs

GRUB Config

A couple of things which are relevant to deterministic time-trave:

  • It's always useful to keep three boot configurations around: i) TT Xen + TT Linux; ii) TT Xen + non-TT Linux; iii) Plain non-Xen Linux. This becomes useful when you break the boot of your dom0, or even Xen.
  • I use a couple of unusual Xen boot params. watchdog=1 enables a watchdog timer in Xen, this is helpful to debug your system when everything but NMIs gets stuck. sync_console=1 forces synchronous console behavior -- this is useful to make sure that all console messages appear on the screen before the crash. Finally, I limit the number of dom0's CPUs to 1, and the size of its memory to 500MB.

Your grub config should look similar to the one below:

default=0
timeout=5
serial --unit=0 --speed=115200
terminal --timeout=0 serial console
title Xen 3.0-unstable (deterministic time-travel) Fedora (2.6.18-xen0)
root (hd0,1)
kernel /boot/xen-3.0-unstable.gz com1=115200,8n1 console=com1,vga dom0_max_vcpus=1 dom0_mem=524288 watchdog=1 sync_console=1
module /boot/vmlinuz-2.6.18-xen0 ro root=LABEL=/ console=tty1 console=ttyS0,115200 selinux=0
module /boot/initrd-2.6.18-xen0.img
title Xen 3.1.4 Fedora (2.6.18.8-xen0)
root (hd0,1)
kernel /boot/xen-3.1.4.gz com1=115200,8n1 console=com1,vga
module /boot/vmlinuz-2.6.18.8-xen0 ro root=LABEL=/ console=tty1 console=ttyS0,115200 selinux=0
module /boot/initrd-2.6.18.8-xen0.img
title Fedora (2.6.25.14-69.emulab1.fc8)
root (hd0,1)
kernel /boot/vmlinuz-2.6.25.14-69.emulab1.fc8 ro boot=UUID=aa0b389c-4831-4ad7-b93b-66bbb1d4384b console=ttyS0,115200 selinux=0
initrd /boot/initrd-2.6.25.14-69.emulab1.fc8.img

Guest VM Configuration

Several things are specific to configuring and running time-traveling VMs:

  • At the moment, we have to keep the amount of memory in a guest VM small. The reason for that is a performance issue, which severely affects the boot time of a guest VM, which freezes everything for a noticeable period of time.
  • Disk configuration has several parameters which our codebase inherits from out previous project (xenvg,/dev/sdb/lvm_1,3G). Similar, time_travel option is required only for backward compatibility.
  • Deterministic time-travel relies on shadow memory to virtualize machine memory for the time-traveling VM. The goal here is to make sure that the time-traveling VM sees the same set of pages across original and replay runs. To enable it we need the auto_translated_physmap feature.

The following is an example of a domU configuration file (this is clientA.conf, which we use in our time-travel invocation examples):

kernel = "/boot/vmlinuz-2.6.18-xenU"
memory = 96
name = "clientA"
disk = ['phy:/dev/xenvg/lvm_1,sda1,w,xenvg,/dev/sdb,lvm_1,3G']
vif = [ 'ip=10.0.0.1' ]
root = "/dev/sda1 ro"
ramdisk = "/tmp/sda4/projects/initrd/initrd-for-tiny-hdd.img"
extra = "ro selinux=0 3 initcall_debug"
time_travel = ['ttc_flag=1', 'tt_dir=/tmp/sda4/images/client_snapshots', 'tt_barrier_name=barrier-1', 'tt_barrier_master=true', 'tt_barrier_nodes=0', 'ttd_flag=0']
features = "auto_translated_physmap"

Preparing VMs Images

We create guest VM images with a simple copy technique. Essentially, we copy content of a dom0 file system into the guest's file system. This can be done with the following steps:

Create a 3GB file for the guest file system:

dd if=/dev/zero of=hd.img bs=1M count=1 seek=3072

Set up a loop device for the file we've created:

losetup /dev/loop1 hd.img

Create a file system:

mkfs.ext3 /dev/loop1

Mount guest and host file systems, and copy files from host to guest. At the end create all necessary directories:

cp -ax /mnt/host/{root,dev,var,etc,usr,bin,sbin,lib} /mnt/guest
mkdir /mnt/guest/{proc,sys,home,tmp}

Make sure you edit guest's /etc/fstab to have a proper root mount:

/dev/sda1 / ext3 defaults 1 1

Simple RAM-based image

If you want to experiment with time-travel, the best way is probably start with a RAM-based Linux -- you can run networking experiments, but at the same time you do not have to wait for a slow disk-based distribution to boot. You can download this minimal busybox-based RAM Linux, and boot it with the VM configuration file below (use root/root to log in):

initrd-busybox-based-ram-linux.img
initrd-busybox-based-ram-linux.conf

The config file expects that you put the initrd file in /boot/initrd-busybox-based-ram-linux.img, and that you enable NAT networking as described below.

Simple Disk-based image

Creating Custom initrd Images

Unpack an existing initrd:

gunzip < initrd-2.6.16-xen.img > initrd-2.6.16-xen.img.ungzip
mkdir new-initrd; cd new-initrd
cpio -i --make-directories < ../initrd-2.6.16-xen.img.ungzip

Repack (from initrd folder do):

cd new-initrd
find . | cpio -oc | gzip - >../new-initrd.img

Disk snapshots with LVM

Create a new LVM volume:

sudo lvcreate -L 600M -n lvm_2 xenvg

Before starting a time-traveling VM, create an LVM snapshot. Of course, make sure that your VM configuration file is edited to start off the snapshot (e.g. /dev/xenvg/snap1):

sudo lvcreate -l 500 -s -n snap1 /dev/xenvg/lvm_1

After finishing the original run, delete the snapshot:

sudo lvremove /dev/xenvg/snap1

And of course create a snapshot again with the command above before starting the replay run.

Networking

Xen takes care of most of network configuration issues. You have several options which you are described here:

wiki.kartbuilding.net/index.php/Xen_Networking

We typically use bridged, or NAT.

  • NAT:
    • Enable NAT in the /etc/xen/xend-config.sxp. Uncomment the following lines (make sure you disable the bridged networking):
    (network-script network-nat)
    (vif-script vif-nat)
    • Add the following line to your guest VM config:
    vif = [ 'ip=10.0.0.1' ]
  • Bridged networking:
    • Enable the bridged networking in the /etc/xen/xend-config.sxp, basically add the following lines (make sure you disable NAT):
    (network-script network-bridge)
    (vif-script vif-bridge)
    • Add the following line to your guest VM config: