Build and Setup Instructions

For a high-level overview of the LCD build subsystem you may want to go visit build overview.

We test and develop LCDs on Dell r710 servers (d710 machines in Emulab) and 64bit Ubuntu 12 LTS and 14 LTS systems.

Step 1: Get these tools

The build requires the GNU build system (gcc, GNU make, autoconf, etc.). In addition, you will need:

  • libncurses (sudo apt-get install ncurses-dev)
    • (for make menuconfig)
  • docbook utils (sudo apt-get install docbook-utils)
    • (for building module init tools manuals)
  • GNU Awk (gawk) (on Emulab this is important, mawk is the default)
    • sudo apt-get install gawk
    • This is needed for building libcap, among other things.

Step 2: Get the source code

You can grab everything via

        git clone --recursive

(note the "recursive" part -- this pulls in a handful of submodules we use) if you are a member of our team. For more details about the source code layout, see here.

Step 3: Configure the Linux kernel

For Emulab d710 specifics beyond what is in this section, see here.

Enter the root of the repository you cloned in Step 2 (this is the root of the 3.10.14 kernel source), and invoke

       make menuconfig     

Make sure you have these turned off:

  • Virtualization (main menu-->Virtualization); this should turn off Linux paravirtualization extensions. Otherwise, LCDs will be compiled with and will try to access pv_ functions, which will not be implemented inside the domain, and will cause an exit.
  • Paravirtualization (main menu-->Processor type and features-->Linux guest support). Turn off the whole thing.
  • Stack protector (main menu-->Processor type and features-->Enable -fstack-protector). This is required for building isolated code. (If we don't turn it off, gcc will try to use stack canaries, which rely on setting up %fs or %gs inside the LCD container, mapping some memory for the canary, and so on. Not worth the effort right now.)
  • Tracers (main menu-->Kernel Hacking-->Tracers). (If you don't turn this off, you may get page faults inside an LCD because it's trying to use the function tracers and so on.)

Make sure you have these turned on:

  • Debug Spinlocks (main menu-->Kernel Hacking-->Spinlock and rw-lock debugging). (The reason is kind of dumb, and I hope to fix it soon.)

Exit and save your configuration.

Step 4a: Dell r710 servers kernel install setup

Due to the use of Broadcom network interfaces by the d710 PC nodes, you will need to place Broadcom firmware in the correct location before installation. Save this as an executable Bash script ( onto your machine and go on to the next step.


        echo "Downloading broadcom firmware"


        echo "Moving files to /lib/firmware/bnx2"

        mv bnx2-mips-06-6.2.3.fw /lib/firmware/bnx2
        mv bnx2-mips-09-6.2.1b.fw /lib/firmware/bnx2

Step 4b: Build and install the kernel, then reboot

In the root of the kernel source:

        make  # (use make -j8 if e.g. you have 8 cores, will go faster)
        (sudo) make && make modules_install && ./ && make install   # order is important; this updates grub
        (sudo) reboot

Select the 3.10.14 kernel from the grub menu. (You may need to enter into the "Advanced Ubuntu Options" or something similar in order to find it.)

If you plan to reboot into LCDs frequently, we recommend you update the grub to choose the LCDs entry by default.

        (sudo) vim /etc/default/grub

Change the GRUB_DEFAULT entry to look like:

        GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 3.10.14"

Run grub update:

        (sudo) update-grub

Step 5: Build and install patched insmod


        cd lcd-domains/module_init_tools
        aclocal -I m4 && automake --add-missing --copy && autoconf
        ./configure --prefix=/ --program-prefix=lcd-
        (sudo) make install

This will install the patched /sbin/lcd-insmod, as well as the other init tools that were left untouched.

The make will fail to build documentation unless you have docbook-utils package. And the man pages won't install on emulab (since /share is read only). You can specify a different man dir via configure if you wish, but this isn't important.

Note: if the compilation failed and generated output like: Makefile:1174: recipe for target 'depmod.conf.5' failed, you can comment out lines where the targets MAN5 and MAN8 are defined. Specifically they are line 419, 420, and 422 in lcd-domains/module_init_tools/build/Makefile.

Step 6: Build LCD microkernel and test modules


        cd lcd-domains

There is no need to install the kernel modules that are built.

Step 7: Try out an example

Load the LCD microkernel module:


Note: If you build the LCD in your virtual machine, you might encounter an error like:

        insmod: ERROR: could not insert module build/microkernel_build/lcd_domains.ko: Input/output error

The reason of that probably lies in the disability of Intel VT-x in your virtual machine. In VMware Workstation 12.0, you must shut down your virtual machine and then enable Intel VT-x by setting:

        Virtual Machine -> Settings -> Processors -> Virtualization Engine

After that, the loading procedure ought to work well.

Run an example:

        scripts/runex ipc1

This script insmod's the boot module for the ipc1 example. The boot module will initialize the LCD and run the example. The script then rmmod's the boot module.

Note: If you want to see all messages from LCDs on the serial line you can do:

        echo 8 > /proc/sys/kernel/printk

Otherwise messages will be logged to syslog.

From here

You can start digging into the main documentation now if you wish.