baggzy said: Hi all! I'm attempting to install kerrighed 3.0.0 via NFSBOOT. There have been some changes to kerrighed and debian which aren't reflected in the online docs, so I thought I'd share my experiences in case it assists others. In this post I'll be covering the NFSBOOT part of the process. I'll compile and run kerrighed later. I'm booting off a machine running Ubuntu 10.10, for anyone who's interested.
The basic process is described in _the official guide_ and also _this contributed doc_ but I had to change some of the details. Follow the official guide but refer to the points below as you go.
Hi baggzy,
First of all thanks for sharing your notes. If you don't mind, I'm going to comment some of the items, and with your permission I could also improve the official Kerrighed wiki :)
baggzy said: 1) The format for [FONT="Courier New"]/etc/default/tftpd-hpa[/FONT] has changed. Using [FONT="Courier New"]RUN_DAEMON = "yes"[/FONT] and [FONT="Courier New"]OPTIONS = "-l -s /srv/tftp"[/FONT] doesn't seem to work any more. Instead I used what's shown below. The default file has [FONT="Courier New"]TFTP_OPTIONS="--secure"[/FONT] but I couldn't get that to work.
According to tfptd-hpa's documentation, when using [FONT="Courier New"]--secure[/FONT], option [FONT="Courier New"]filename[/FONT] in the DHCP server configuration should not contain any path (eg. [FONT="Courier New"]pxelinux.0[/FONT], not [FONT="Courier New"]/srv/tftp/pxelinux.0[/FONT]), and the file should of course live exactly in the daemon's directory (ie [FONT="Courier New"]/srv/tftp[/FONT]). Have you tried setting up tftpd-hpa+dhcpd.conf this way?
baggzy said: tftpd-hpa runs as a daemon by default now, so no need for the -l flag. (And no need for xinetd either.)
# /etc/default/tftpd-hpa
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS=""
Good candidate for improving the wiki.
baggzy said: 2) The official guide doesn't mention which file to edit to "Configure the DHCP server". The file in question is [FONT="Courier New"]/etc/dhcp3/dhcp.conf[/FONT].
I guess that you meant [FONT="Courier New"]/etc/dhcp3/dhcpd.conf[/FONT], right? Note that on newer distros (eg Debian Squeeze), it has even moved to [FONT="Courier New"]/etc/dhcp/dhcpd.conf[/FONT]. The package has also been renamed isc-dhcp-server.
baggzy said: The machine I'm booting off (the "server") has IP address 192.168.0.1 on network card eth0, but is connected to the cluster via a second network card with IP 192.168.1.1 called eth1. The nodes will have IP's starting at 192.168.1.101 and be called node101, node102, etc. My [FONT="Courier New"]dhcp.conf[/FONT] therefore looks like this:
#
# Sample configuration file for ISC dhcpd for Debian
#
# Attention: If /etc/ltsp/dhcpd.conf exists, that will be used as
# configuration file instead of this file.
#
# $Id: dhcpd.conf,v 1.1.1.1 2002/05/21 00:07:44 peloy Exp $
#
# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed. We default to the
# behavior of the version 2 packages ('none', since DHCP v2 didn't
# have support for DDNS.)
ddns-update-style none;
# option definitions common to all supported networks...
#option domain-name "example.org";
#option domain-name-servers ns1.example.org, ns2.example.org;
default-lease-time 600;
max-lease-time 7200;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
#authoritative;
# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;
# No service will be given on this subnet, but declaring it helps the
# DHCP server to understand the network topology.
#subnet 10.152.187.0 netmask 255.255.255.0 {
#}
# This is a very basic subnet declaration.
#subnet 10.254.239.0 netmask 255.255.255.224 {
# range 10.254.239.10 10.254.239.20;
# option routers rtr-239-0-1.example.org, rtr-239-0-2.example.org;
#}
# This declaration allows BOOTP clients to get dynamic addresses,
# which we don't really recommend.
#subnet 10.254.239.32 netmask 255.255.255.224 {
# range dynamic-bootp 10.254.239.40 10.254.239.60;
# option broadcast-address 10.254.239.31;
# option routers rtr-239-32-1.example.org;
#}
# A slightly different configuration for an internal subnet.
#subnet 10.5.5.0 netmask 255.255.255.224 {
# range 10.5.5.26 10.5.5.30;
# option domain-name-servers ns1.internal.example.org;
# option domain-name "internal.example.org";
# option routers 10.5.5.1;
# option broadcast-address 10.5.5.31;
# default-lease-time 600;
# max-lease-time 7200;
#}
# Hosts which require special configuration options can be listed in
# host statements. If no address is specified, the address will be
# allocated dynamically (if possible), but the host-specific information
# will still come from the host declaration.
#host passacaglia {
# hardware ethernet 0:0:c0:5d:bd:95;
# filename "vmunix.passacaglia";
# server-name "toccata.fugue.com";
#}
# Fixed IP addresses can also be specified for hosts. These addresses
# should not also be listed as being available for dynamic assignment.
# Hosts for which fixed IP addresses have been specified can boot using
# BOOTP or DHCP. Hosts for which no fixed address is specified can only
# be booted with DHCP, unless there is an address range on the subnet
# to which a BOOTP client is connected which has the dynamic-bootp flag
# set.
#host fantasia {
# hardware ethernet 08:00:07:26:c0:a5;
# fixed-address fantasia.fugue.com;
#}
# You can declare a class of clients and then do address allocation
# based on that. The example below shows a case where all clients
# in a certain class get addresses on the 10.17.224/24 subnet, and all
# other clients get addresses on the 10.0.29/24 subnet.
#class "foo" {
# match if substring (option vendor-class-identifier, 0, 4) = "SUNW";
#}
#shared-network 224-29 {
# subnet 10.17.224.0 netmask 255.255.255.0 {
# option routers rtr-224.example.org;
# }
# subnet 10.0.29.0 netmask 255.255.255.0 {
# option routers rtr-29.example.org;
# }
# pool {
# allow members of "foo";
# range 10.17.224.10 10.17.224.250;
# }
# pool {
# deny members of "foo";
# range 10.0.29.10 10.0.29.230;
# }
#}
### PART 1
# General options
option dhcp-max-message-size 2048;
use-host-decl-names on;
deny bootp;
### PART 2
option domain-name "kerrighed";
option domain-name-servers 192.168.1.1;
option ntp-servers ntp.network.net;
### PART 3
subnet 192.168.1.0 netmask 255.255.255.0 {
option routers 192.168.1.1;
option broadcast-address 192.168.1.255;
# Define the first and last IP address to be authorized
range 192.168.1.101 192.168.1.199;
# This set up the node name to « nodeXX » with XX the id of the node (ip-address based).
send host-name = concat("node", binary-to-ascii(10, 8, ".", substring(leased-address, 3, 1)));
# If you want to limit which boxes are using kerrighed, use something like this :
# host ssi1 { fixed-address 192.168.0.101; hardware ethernet xx:xx:xx:xx:xx:xx;
filename "/srv/tftp/pxelinux.0";
}
### PART 4
group {
server-name "server";
option root-path "192.168.1.1:/NFSROOT/kerrighed";
}
3) As far as I recall, I didn't change anything in the standard file, I just added the stuff at the bottom. But I left all the default gumph in there just in case. A few things to note:
i) Since the server is on 192.168.1.1, starting the range at 192.168.1.2 would mean that the node names start with node2... So I started the range at 101 so the first node is node101. I suppose I could reconfigure eth1 on the server to be 192.168.1.254 or something, but I didn't.
ii) I moved the [FONT="Courier New"]filename "/srv/tftp/pxelinux.0";[/FONT] from the group block to the subnet block. It didn't seem to work in the group block - I'd get a the following error on the node:
PXE-E53: No boot filename received
I have to admit that I don't understand the reason for this group declaration, since it contains no host declarations. I was told that it was working as it is written in the wiki, but I'm very much inclined to believe you :) To me the contents of this group declaration should be simply put in the subnet statement above. People needing more complex DHCP configurations will probably know how to create appropriate group or class declarations. Do you believe that there is some interest for this group declaration?
baggzy said: 4) I added the following to [FONT="Courier New"]/etc/default/dhcp3-server[/FONT]:
INTERFACES="eth1"
Good candidate for the wiki too.
baggzy said: 5) I added the following to [FONT="Courier New"]/etc/network/interfaces[/FONT]:
auto eth1
iface eth1 inet static
address 192.168.1.1
netmask 255.255.255.0
Good candidate for the wiki.
baggzy said: 6) I added the following to [FONT="Courier New"]/etc/hosts.allow[/FONT] because without it I'd get a TFTP timeout error:
ALL: 192.168.1.*
Good catch! I don't remember anybody mentioning it (which does not say that nobody ever told me about it, I just don't remember:)). As far as I can see, you must have something in /etc/hosts.deny that match your nodes and tftp. Otherwise, leaving /etc/hosts.allow empty should be enough to have tftpd-hpa working.
baggzy said: 7) If you make changes to the dhcp or tftp config files you need to restart various services. As Steve Kelly suggesting in his guide, I set up a script to do this, called "restart-dhcp". You'll need to [FONT="Courier New"]chmod 755[/FONT] it, then put it somewhere in your [FONT="Courier New"]$PATH[/FONT] (I put it in [FONT="Courier New"]~/bin[/FONT] and add that to my [FONT="Courier New"]PATH[/FONT] in my [FONT="Courier New"].bashrc[/FONT]). Note that "restart" doesn't seem to work with tftpd-hpa so you have to "stop" it, then "start" it.
#
# tftpd-hpa restart is broken, so don't use it!
#
sudo restart portmap
sudo service dhcp3-server restart
sudo stop tftpd-hpa
sudo start tftpd-hpa
sudo service nfs-kernel-server restart
sudo exportfs -ra
I would not include this in the wiki, but maybe add some link to your post.
baggzy said: 8) The output from this script should look like the code below. If any of the process id's are missing the daemon probably didn't actually start. But a "status" check will say it did. There's obviously a bug somewhere and the only solution I found was to reboot the server.
portmap start/running, process 844
* Stopping DHCP server dhcpd3 [ OK ]
* Starting DHCP server dhcpd3 [ OK ]
tftpd-hpa stop/waiting
tftpd-hpa start/running, process 23714
* Stopping NFS kernel daemon [ OK ]
* Unexporting directories for NFS kernel daemon... [ OK ]
* Exporting directories for NFS kernel daemon... [ OK ]
* Starting NFS kernel daemon [ OK ]
9) The official guide doesn't mention it, but after you've done "[FONT="Courier New"]sudo mkdir /srv/tftp[/FONT]" you have to do the following. The first thing your node will do when it tries to NFSBOOT is look for a file called [FONT="Courier New"]pxelinux.0[/FONT] in [FONT="Courier New"]/srv/tftp[/FONT]. Obviously if it's not there it won't boot. Next it will fire up TFTP and try to transfer the boot files specified in the [FONT="Courier New"]/srv/tftp/pxlinux.cfg/default[/FONT] file. If it fails it's either because TFTP isn't running, isn't set up correctly, doesn't have the required permissions, or your node isn't listed in [FONT="Courier New"]/etc/hosts.allow[/FONT]. The [FONT="Courier New"]chown[/FONT] may not be necessary, but if you have problems at the TFTP stage give it a try.
sudo cp /usr/lib/syslinux/pxelinux.0 /srv/tftp/
sudo chown -R tftp:tftp /srv/tftp
Good candidate for the wiki.
baggzy said: 10) After you've [FONT="Courier New"]debootstrapp[/FONT]ed lenny and [FONT="Courier New"]chroot[/FONT]'d into [FONT="Courier New"]/NFSROOT/kerrighed[/FONT], in addition to mounting [FONT="Courier New"]/proc[/FONT] I also had to do the following to avoid errors while using "[FONT="Courier New"]apt-get install[/FONT]":
mount -t devpts none /dev/pts
I think that the errors you mention are dpkg complaining that it is not able to log. This however does not prevent dpkg from correctly installing packages, so I personally don't mount /dev/pts. I can add this to the wiki though, if people can feel more comfortable this way.
baggzy said: 11) To avoid warnings about your [FONT="Courier New"]LOCALE[/FONT] the first thing you need to install is [FONT="Courier New"]locales[/FONT], then configure it and select "[FONT="Courier New"]en_GB.UTF-8[/FONT]":
apt-get install locales
dpkg-reconfigure locales
Same category as the previous item, IMHO.
baggzy said: 12) After you've installed [FONT="Courier New"]nfs-common[/FONT], to avoid [FONT="Courier New"]statd[/FONT] failing during boot, and holding things up about 5 minutes each time, edit [FONT="Courier New"]/etc/default/nfs-common[/FONT] and change [FONT="Courier New"]NEED_STATD[/FONT] to no:
NEED_STATD=no
I wouldn't recommend disabling statd, especially when you mount NFS trees with file locking support (IIRC this is the default for mounts done from [FONT="Courier New"]/etc/fstab[/FONT]). You might have issues because either [FONT="Courier New"]/var/run[/FONT] or [FONT="Courier New"]/var/lib/nfs[/FONT] is shared (each node must have its own ones, for instance using tmpfs mounts). [FONT="Courier New"]/var/lib/nfs[/FONT] should be added to the [FONT="Courier New"]/etc/fstab[/FONT] sample of the wiki, however it requires a bit of initialization. I don't know if nfsbooted have some helpers for this.
baggzy said: 13) After installing [FONT="Courier New"]initramfs-tools[/FONT] edit [FONT="Courier New"]/etc/initramfs-tools/initramfs.conf[/FONT] and change [FONT="Courier New"]BOOT=local[/FONT] to:
BOOT=nfs
Good candidate too.
baggzy said: 14) If you want to boot your node into debian to see if everything has worked so far...
i) Do the following. (Answer "No" to creating a link, and "No" to aborting.)
apt-get install linux-image-2.6.26-2-amd64
ii) [FONT="Courier New"]exit[/FONT] (i.e. leave chroot)
iii) Copy the boot files:
sudo cp /NFSROOT/kerrighed/boot/vmlinuz-2.6.26-2-amd64 /srv/tftp/
sudo cp /NFSROOT/kerrighed/boot/initrd.img-2.6.26-2-amd64 /srv/tftp/
iv) Create or edit /srv/tftp/pxelinux.cfg/default to:
DEFAULT debian
LABEL debian
kernel vmlinuz-2.6.26-2-amd64
append ip=dhcp root=/dev/nfs initrd=initrd.img-2.6.26-2-amd64 nfsroot=192.168.1.1:/NFSROOT/kerrighed,rw rw
initrd initrd.img-2.6.26-2-amd64
15) That's it! You should be able to NFSBOOT your node(s)...
Thanks again for your notes!
Louis
