DNARD Cluster Setup

These are some collected notes about setting up the DNARD cluster for the Flux project.

Other DNARD resources

For more information about the SHARKs and NetBSD-arm32, please see the some of the following references (culled from my bookmarks file; may not be up to date):

Our setup

We run all of our DNARDs with completely shared filesystems. We mount a ramdisk on them for /var/run and for /tmp. We use YP for the password file, and 'amd' for mounting home directories. To save memory, we removed many of the standard daemons, but still run a pretty stock unix system on them.

Firmware

See the DNARD FAQ for more information. For booting NetBSD 1.4, you'll want to make sure your firmware is up to Feb 25, 1998 or later. The 1.3 that DEC shipped worked around a firmware bug detecting the ethernet device, but 1.4 doesn't (and shouldn't). Upgrade. It's really easy.

DHCP

We run our DNARDs off of a single DHCP server. Things to ensure on the DHCP server:

I like having the DNARDs each have a unique IP address, so I can tell which of them is down if one bombs out. To accomplish this, I've set up the DHCP mappings on a per-hardware-address basis. Typing in all of the hardware addresses can be a pain; this perl script will open the serial port, and read the DNARD boot prompt off of it. Just plug your dnard in, and it'll snarf it out. Be sure to change the $ddir settings, and if you use a different serial port, change the open line too. It puts entries like: 

015     08:00:2b:81:61:ea               February 25, 1998 17:19:25
in the "hardware" file. It assigns the numbers in a monotonically increasing fashion. Handy when you've got a labeling machine, too.

Once the hardware file is created, I use another script to parse it out and create the remainder of the dhcpd.conf file. I concatenate the output of this with our dhcpd.conf.head file to create the full dhcpd.conf file. Note: Don't use our dhcpd.conf file, it won't work for you! :) I've provided it here as an (abbreviated) example of how to set things up.

Host filesystem setup

   /z/dnard/
            dhcp/  - Holds the dhcpd, and conf files.
            root/  - Holds the entire NetBSD filesystem
            swap/  - Holds the per-machine swapfiles

I create a swap file for each of the DNARDs in /z/dnard/swap.
I call them 'swap.dnard#', but that's up to you.  The changes to
/etc/rc mentioned below assume this convention.


Dnard filesystem setup


Because we can't tell the DNARDs their swap device until after they
boot, and they don't have independent /etc directories,
I modified /etc/rc to mount their particular swapfile, and
then create the MFS which holds their /tmp and /var/run.
The modified /etc/rc can be found here.
The changes are the section that says:



echo "doing the funky DNARD swap thing"

mount moab.cs.utah.edu:/z/dnard/swap/swap.`hostname` /swap
chmod 700 /swap
/sbin/swapon /swap
echo -n " mfs-tmp"
mount_mfs -s 65535 -b 8192 -f 1024 -m 1 swap /tmp
mount_mfs -s 2048  -b 8192 -f 1024 -m 1 swap /var/run
echo "155.99.214.88" > /var/yp/binding/cs.utah.edu.ypservers



Change the paths and such to your own servers, of course.  The 
change to ypservers was simply a quick hack so I could have
independent binding dirs.  Take it or leave it. :-)


DNARD /etc

We modified /etc/syslog.conf on the DNARDs to log to a central host
with a real backing store, so we can have saved output and not
need to make /var/log a symlink.  In our case, we simply log to
one of the DNARDs, so all logfiles are accessible in /var/log.



We run ssh on the DNARDs.  I've simply set them up with a single,
shared set of ssh* files in /etc, and they haven't seemed to
complain.  Your mileage may vary - I haven't explored the effects
of allowing all of the hosts to share a single ssh_random_seed
file.  If it becomes a problem for you, let me know.


We don't set the machine identity in their /etc, instead allowing the
machines to gather this information via DHCP.  It'a automatic,
so just leave them unset.  Things like /etc/mygate should be
common for all of the machines in your cluster anyway, so set them
as you normally would.  Note that we do NOT set the machines up as
DHCP clients (dhclient=YES).  If you'd like, you can see
how we set up the rc.conf file.


DNARD directory setup

If you use YP, 
Remove /var/yp/binding and replace it with a symlink to ../run/
(the mfs).



Create a /swap directory to hold the mounted swap partition.


Problems?

If there are inaccuracies or goofs in this, please mail me.  If
you have generic NetBSD-arm32/dnard questions, check out the
resources I mentioned above;  they'll likely give you better
assistance.


    

    
David G Andersen



Last modified: Fri May 21 15:40:57 MDT 1999