特殊:Badtitle/NS100:AutomatedNodeDeployment:修订间差异
小 创建新页面为 '{{From|https://help.ubuntu.com/community/AutomatedNodeDeployment}} {{Languages|UbuntuHelp:AutomatedNodeDeployment}} == Setting up an Ubuntu System to deploy Ubuntu == Installing...' |
小无编辑摘要 |
||
第48行: | 第48行: | ||
# /opt/cluster/config/global.conf: configuration file for node deployment | # /opt/cluster/config/global.conf: configuration file for node deployment | ||
# This file is sourced by files used by the deployment system | # This file is sourced by files used by the deployment system | ||
# Site information | # Site information | ||
DOMAIN_NAME="home.local" | DOMAIN_NAME="home.local" | ||
第73行: | 第72行: | ||
MIRROR_SERVER_NAME="mirror" | MIRROR_SERVER_NAME="mirror" | ||
MIRROR_SERVER_IP="10.10.1.10" | MIRROR_SERVER_IP="10.10.1.10" | ||
# Service information | # Service information | ||
DHCPD_CONFIG_FILE="/etc/dhcp3/dhcpd.conf" | DHCPD_CONFIG_FILE="/etc/dhcp3/dhcpd.conf" | ||
第87行: | 第85行: | ||
TFTP_ROOT="/var/lib/tftpboot" | TFTP_ROOT="/var/lib/tftpboot" | ||
DEFAULT_PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/default" | DEFAULT_PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/default" | ||
# NODE information | # NODE information | ||
HEAD_NODE="headnode" | HEAD_NODE="headnode" | ||
第103行: | 第100行: | ||
NODE_USER="cluster" | NODE_USER="cluster" | ||
NODE_USER_UID=1010 | NODE_USER_UID=1010 | ||
# NFS Root filesystem information | # NFS Root filesystem information | ||
ARCH="amd64" | ARCH="amd64" | ||
第121行: | 第117行: | ||
HOSTS_FILE="$NFS_BUILD_DIR/etc/hosts" | HOSTS_FILE="$NFS_BUILD_DIR/etc/hosts" | ||
CURRENT_PXE_FILES="$HEAD_NODE_CONFIG_DIR/pxefiles.txt" | CURRENT_PXE_FILES="$HEAD_NODE_CONFIG_DIR/pxefiles.txt" | ||
printf "Global configration file loaded\n" | printf "Global configration file loaded\n" | ||
</nowiki></pre> | </nowiki></pre> | ||
The global configuration file can be sourced with either | The global configuration file can be sourced with either | ||
* | * # /opt/cluster/config/global.conf | ||
* source /opt/cluster/config/global.conf | * source /opt/cluster/config/global.conf | ||
As you read on, you will notice most if not all scripts will refer to /opt/cluster/config/global.conf. | As you read on, you will notice most if not all scripts will refer to /opt/cluster/config/global.conf. | ||
第145行: | 第140行: | ||
* Installs syslinux | * Installs syslinux | ||
* Installs ISC DHCP Server | * Installs ISC DHCP Server | ||
* Configures the scope for the DHCP server | ** Configures the scope for the DHCP server | ||
* Installs NTP | * Installs NTP | ||
* Configures NTP | ** Configures NTP | ||
* Installs BIND | * Installs BIND | ||
* Configures the forward look-up zone | ** Configures the forward look-up zone | ||
* Configures the reverse look-up zone | ** Configures the reverse look-up zone | ||
* Installs NFS server | * Installs NFS server | ||
* Configures our exports | ** Configures our exports | ||
* Install the LAMP stack (see Note(2) regarding LAMP located after script) | * Install the LAMP stack (see Note(2) regarding LAMP located after script) | ||
* Makes our Ubuntu mirror available via HTTP | ** Makes our Ubuntu mirror available via HTTP | ||
* Creates our initial node list (blank to begin) | * Creates our initial node list (blank to begin) | ||
* Installs parallel-ssh | * Installs parallel-ssh | ||
第172行: | 第167行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
# Install packages on Head Node | # Install packages on Head Node | ||
# Create the cluster user and prompt for password | # Create the cluster user and prompt for password | ||
第356行: | 第347行: | ||
# Install wake-on-LAN | # Install wake-on-LAN | ||
apt-get -y install wakeonlan | apt-get -y install wakeonlan | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
Note(1) | Note(1) | ||
Apt-mirror is a tool which will allow you to maintain a local mirror of the Ubuntu repositories. You will be able to make the mirror available via HTTP, FTP, NFS, etc. In our will use HTTP which will require the installation of a Web Server. | Apt-mirror is a tool which will allow you to maintain a local mirror of the Ubuntu repositories. You will be able to make the mirror available via HTTP, FTP, NFS, etc. In our scenario, we will use HTTP which will require the installation of a Web Server. | ||
Mirroring the repository (to include main, updates, multiverse, universe, and restricted for the amd64 architecture of a single release will require approximately 24 GB of space. | Mirroring the repository (to include main, updates, multiverse, universe, and restricted for the amd64 architecture of a single release will require approximately 24 GB of space. | ||
The length of time required for the mirror process to complete is dependent upon several factors, one of which is bandwidth. It took me approximately 2.5 days via DSL. | The length of time required for the mirror process to complete is dependent upon several factors, one of which is bandwidth. It took me approximately 2.5 days via DSL. | ||
第369行: | 第358行: | ||
The entire LAMP stack is not needed. Since we will be providing the Ubuntu repositories via HTTP as well as the preseed configuration file, all that is needed is Apache. I prefer the LAMP stack as it provides a multitude of possibilities. Some of which will be appended to the end of this document. | The entire LAMP stack is not needed. Since we will be providing the Ubuntu repositories via HTTP as well as the preseed configuration file, all that is needed is Apache. I prefer the LAMP stack as it provides a multitude of possibilities. Some of which will be appended to the end of this document. | ||
== Bootstrap an Ubuntu Installation == | == Bootstrap an Ubuntu Installation == | ||
Now that our prerequisite software is installed and configured | Now that our prerequisite software is installed and configured we will create a base installation for use by disk-less nodes. | ||
The script to bootstrap an Ubuntu installation: | The script to bootstrap an Ubuntu installation: | ||
* Archives the previous bootstrap installation if one was completed for the same release and architechure | * Archives the previous bootstrap installation if one was completed for the same release and architechure | ||
第399行: | 第388行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
progress () | progress () | ||
{ | { | ||
第425行: | 第410行: | ||
printf "\n" | printf "\n" | ||
} | } | ||
# Check to see if we have used debootstrap for this same release and architecture on a previous ocassion. | # Check to see if we have used debootstrap for this same release and architecture on a previous ocassion. | ||
if [ -d $NFS_BUILD_DIR ]; then | if [ -d $NFS_BUILD_DIR ]; then | ||
printf "Archiving previous installation...\n" | |||
tar --checkpoint=100 --checkpoint-action=exec='printf "."' -czf $HEAD_NODE_WORKING_DIR/nfsroot-$(date +%m-%d-%y).tgz $NFS_BUILD_DIR | |||
printf "\n" | |||
printf "Removing previous installation..."; progress | |||
rm -rf $NFS_BUILD_DIR | |||
else | else | ||
mkdir -p $NFS_BUILD_DIR | |||
fi | fi | ||
# Bootstrap the installation | # Bootstrap the installation | ||
printf "Performing a bootstrap of $RELEASE-$ARCH..."; progress | printf "Performing a bootstrap of $RELEASE-$ARCH..."; progress | ||
debootstrap --arch $ARCH --include=$PRE_INST_PKGS --exclude=$PRE_INST_EXCL_PKGS $RELEASE $NFS_BUILD_DIR $UBUNTU_MIRROR_URL | debootstrap --arch $ARCH --include=$PRE_INST_PKGS --exclude=$PRE_INST_EXCL_PKGS $RELEASE $NFS_BUILD_DIR $UBUNTU_MIRROR_URL | ||
# Reconfigure our locale and time zone | # Reconfigure our locale and time zone | ||
printf "Updating locale and timezone..."; progress | printf "Updating locale and timezone..."; progress | ||
chroot $NFS_BUILD_DIR dpkg-reconfigure locales | chroot $NFS_BUILD_DIR dpkg-reconfigure locales | ||
chroot $NFS_BUILD_DIR dpkg-reconfigure tzdata | chroot $NFS_BUILD_DIR dpkg-reconfigure tzdata | ||
# Update the apt sources | # Update the apt sources | ||
printf "Updating apt sources..."; progress | printf "Updating apt sources..."; progress | ||
第451行: | 第432行: | ||
printf "deb $UBUNTU_MIRROR_URL $RELEASE-updates $REPOSITORY\n" >> $APT_SOURCES_FILE | printf "deb $UBUNTU_MIRROR_URL $RELEASE-updates $REPOSITORY\n" >> $APT_SOURCES_FILE | ||
printf "deb $UBUNTU_MIRROR_URL $RELEASE-security $REPOSITORY\n" >> $APT_SOURCES_FILE | printf "deb $UBUNTU_MIRROR_URL $RELEASE-security $REPOSITORY\n" >> $APT_SOURCES_FILE | ||
# Update fstab | # Update fstab | ||
printf "Updating fstab to include proc..."; progress | printf "Updating fstab to include proc..."; progress | ||
第459行: | 第439行: | ||
printf "Updating fstab to include $NFS_USER\'s NFS home directory..."; progress | printf "Updating fstab to include $NFS_USER\'s NFS home directory..."; progress | ||
printf "$NFS_SERVER_NAME.$DOMAIN_NAME:$NFS_HOME_EXPORT\t/home/cluster\tnfs\trw,auto\t0 0\n" >> $FSTAB_FILE | printf "$NFS_SERVER_NAME.$DOMAIN_NAME:$NFS_HOME_EXPORT\t/home/cluster\tnfs\trw,auto\t0 0\n" >> $FSTAB_FILE | ||
# Update the build environment with a generic host-name | # Update the build environment with a generic host-name | ||
# The hostname defaults to the host-name of the system on which it was built | # The hostname defaults to the host-name of the system on which it was built | ||
printf "Updating build environment with generic host name..."; progress | printf "Updating build environment with generic host name..."; progress | ||
printf "$BASE_NODE_NAME\n" > $HOSTNAME_FILE | printf "$BASE_NODE_NAME\n" > $HOSTNAME_FILE | ||
# Update NTP configuration to sync with local time server | # Update NTP configuration to sync with local time server | ||
printf "Updating NTP configuration..."; progress | printf "Updating NTP configuration..."; progress | ||
sed -i s/ntp.ubuntu.com/$NTP_SERVER_NAME.$DOMAIN_NAME/ $NTP_CONF_FILE | sed -i s/ntp.ubuntu.com/$NTP_SERVER_NAME.$DOMAIN_NAME/ $NTP_CONF_FILE | ||
sed -i "/$NTP_SERVER_NAME.$DOMAIN_NAME/a restrict $NTP_SERVER_NAME.$DOMAIN_NAME mask 255.255.255.255 nomodify notrap noquery" $NTP_CONF_FILE | sed -i "/$NTP_SERVER_NAME.$DOMAIN_NAME/a restrict $NTP_SERVER_NAME.$DOMAIN_NAME mask 255.255.255.255 nomodify notrap noquery" $NTP_CONF_FILE | ||
# Prepare a generic network interfaces config | # Prepare a generic network interfaces config | ||
printf "Updating build environment with generic interface configuration..."; progress | printf "Updating build environment with generic interface configuration..."; progress | ||
第480行: | 第457行: | ||
printf "Updating hosts files to include loopback interface..."; progress | printf "Updating hosts files to include loopback interface..."; progress | ||
printf "127.0.0.1\tlocalhost\n" > $HOSTS_FILE | printf "127.0.0.1\tlocalhost\n" > $HOSTS_FILE | ||
# Mount proc in our build environment and ensure our base system is at the latest versions | # Mount proc in our build environment and ensure our base system is at the latest versions | ||
printf "Updating packages to most recent versions..."; progress | printf "Updating packages to most recent versions..."; progress | ||
第486行: | 第462行: | ||
chroot $NFS_BUILD_DIR apt-get -y update | chroot $NFS_BUILD_DIR apt-get -y update | ||
chroot $NFS_BUILD_DIR apt-get -y dist-upgrade | chroot $NFS_BUILD_DIR apt-get -y dist-upgrade | ||
# Install additional packages | # Install additional packages | ||
printf "Installing additional packages..."; progress | printf "Installing additional packages..."; progress | ||
for PACKAGE in $POST_INST_PKGS; do | for PACKAGE in $POST_INST_PKGS; do | ||
chroot $NFS_BUILD_DIR apt-get -q -y install $PACKAGE | |||
done | done | ||
# Remove packages | # Remove packages | ||
printf "Purging packages..."; progress | printf "Purging packages..."; progress | ||
for PACKAGE in $PKGS_TO_PURGE; do | for PACKAGE in $PKGS_TO_PURGE; do | ||
chroot $NFS_BUILD_DIR apt-get -y remove --purge $PACKAGE | |||
done | done | ||
# Determine the name of the kernel and initrd that was installed | # Determine the name of the kernel and initrd that was installed | ||
BASE_KERNEL_FILE=$(basename `ls $NFS_BUILD_DIR/boot/vmlinuz*`) | BASE_KERNEL_FILE=$(basename `ls $NFS_BUILD_DIR/boot/vmlinuz*`) | ||
第504行: | 第477行: | ||
ORIGINAL_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-original/) | ORIGINAL_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-original/) | ||
NFS_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-nfs/) | NFS_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-nfs/) | ||
# Copy the unmodified initramfs to our TFTP root | # Copy the unmodified initramfs to our TFTP root | ||
printf "Copying unmodifed inittf to TPTP server..."; progress | printf "Copying unmodifed inittf to TPTP server..."; progress | ||
第510行: | 第482行: | ||
printf "base-initrd,$ORIGINAL_INITRD_FILE\n" > $CURRENT_PXE_FILES | printf "base-initrd,$ORIGINAL_INITRD_FILE\n" > $CURRENT_PXE_FILES | ||
cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$ORIGINAL_INITRD_FILE | cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$ORIGINAL_INITRD_FILE | ||
# Update the initramfs to reflect a NFS root | # Update the initramfs to reflect a NFS root | ||
# We can do this before the kernel is installed in the chroot, howerver, we want to | # We can do this before the kernel is installed in the chroot, howerver, we want to | ||
第517行: | 第488行: | ||
sed -i.orig s/BOOT=local/BOOT=nfs/ $NFS_BUILD_DIR/etc/initramfs-tools/initramfs.conf | sed -i.orig s/BOOT=local/BOOT=nfs/ $NFS_BUILD_DIR/etc/initramfs-tools/initramfs.conf | ||
chroot $NFS_BUILD_DIR update-initramfs -u | chroot $NFS_BUILD_DIR update-initramfs -u | ||
# Install ubuntu-server | # Install ubuntu-server | ||
printf "Installing ubuntu-server..."; progress | printf "Installing ubuntu-server..."; progress | ||
chroot $NFS_BUILD_DIR tasksel install server | chroot $NFS_BUILD_DIR tasksel install server | ||
# Clean up the package cache | # Clean up the package cache | ||
printf "Cleaning package cache..."; progress | printf "Cleaning package cache..."; progress | ||
chroot $NFS_BUILD_DIR apt-get clean | chroot $NFS_BUILD_DIR apt-get clean | ||
# Remove the /etc/rcS.d init script which parses /etc/fstab and replace it | # Remove the /etc/rcS.d init script which parses /etc/fstab and replace it | ||
# with the one located in /etc/network/if-up.d | # with the one located in /etc/network/if-up.d | ||
第532行: | 第500行: | ||
chroot $NFS_BUILD_DIR unlink /etc/rcS.d/S45mountnfs.sh | chroot $NFS_BUILD_DIR unlink /etc/rcS.d/S45mountnfs.sh | ||
chroot $NFS_BUILD_DIR ln -s /etc/network/if-up.d/mountnfs /etc/rcS.d/S45mountnfs | chroot $NFS_BUILD_DIR ln -s /etc/network/if-up.d/mountnfs /etc/rcS.d/S45mountnfs | ||
# set a root password for | # set a root password for | ||
printf "Setting password for root..."; progress | printf "Setting password for root..."; progress | ||
chroot $NFS_BUILD_DIR passwd | chroot $NFS_BUILD_DIR passwd | ||
# create the node user and ask us for password | # create the node user and ask us for password | ||
printf "Setting password for $NODE_USER..."; progress | printf "Setting password for $NODE_USER..."; progress | ||
chroot $NFS_BUILD_DIR adduser --uid $NODE_USER_UID --gecos $NODE_USER $NODE_USER | chroot $NFS_BUILD_DIR adduser --uid $NODE_USER_UID --gecos $NODE_USER $NODE_USER | ||
# Unmount proc from our build environment | # Unmount proc from our build environment | ||
umount $NFS_BUILD_DIR/proc | umount $NFS_BUILD_DIR/proc | ||
# Copy the kernel to our TFTP root | # Copy the kernel to our TFTP root | ||
printf "Copying kernel to TFTP server..."; progress | printf "Copying kernel to TFTP server..."; progress | ||
printf "base-kernel,$BASE_KERNEL_FILE\n" >> $CURRENT_PXE_FILES | printf "base-kernel,$BASE_KERNEL_FILE\n" >> $CURRENT_PXE_FILES | ||
cp $NFS_BUILD_DIR/boot/$BASE_KERNEL_FILE $NODE_PXE/$BASE_KERNEL_FILE | cp $NFS_BUILD_DIR/boot/$BASE_KERNEL_FILE $NODE_PXE/$BASE_KERNEL_FILE | ||
# Copy the NFS aware initrd to our TFTP root | # Copy the NFS aware initrd to our TFTP root | ||
printf "Copying NFS root aware initramfs to TFTP server..."; progress | printf "Copying NFS root aware initramfs to TFTP server..."; progress | ||
printf "nfs-initrd,$NFS_INITRD_FILE\n" >> $CURRENT_PXE_FILES | printf "nfs-initrd,$NFS_INITRD_FILE\n" >> $CURRENT_PXE_FILES | ||
cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$NFS_INITRD_FILE | cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$NFS_INITRD_FILE | ||
# Copy the netboot initrd and kernel into out TFPT root. This can be gathered from | # Copy the netboot initrd and kernel into out TFPT root. This can be gathered from | ||
# the server CD or online | # the server CD or online | ||
if [ ! -f $NODE_PXE/installer-linux ]; then | if [ ! -f $NODE_PXE/installer-linux ]; then | ||
printf "Collecting the netboot installer kernel and initrd from us.archive.ubuntu.com"; progress | |||
wget -nv -O $NODE_PXE/installer-linux http://us.archive.ubuntu.com/ubuntu/dists/$RELEASE/main/installer-$ARCH/current/images/netboot/ubuntu-installer/$ARCH/linux | |||
wget -nv -O $NODE_PXE/installer-initrd.gz http://us.archive.ubuntu.com/ubuntu/dists/$RELEASE/main/installer-$ARCH/current/images/netboot/ubuntu-installer/$ARCH/initrd.gz | |||
fi | fi | ||
printf "Process complete!\n" | printf "Process complete!\n" | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第597行: | 第556行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
INITRD_TMP_DIR="/tmp/initrd-tmp" | INITRD_TMP_DIR="/tmp/initrd-tmp" | ||
# Create a directory to store the build environment | # Create a directory to store the build environment | ||
mkdir -p $INITRD_TMP_DIR | mkdir -p $INITRD_TMP_DIR | ||
# Set our working directory to our temp directory for the initrd | # Set our working directory to our temp directory for the initrd | ||
pushd $INITRD_TMP_DIR | pushd $INITRD_TMP_DIR | ||
# Uncompress the initrd | # Uncompress the initrd | ||
gzip -dc $NODE_PXE/$(grep base-initrd $CURRENT_PXE_FILES | cut -d, -f2) | cpio -id | gzip -dc $NODE_PXE/$(grep base-initrd $CURRENT_PXE_FILES | cut -d, -f2) | cpio -id | ||
# We will Use netcat as the client to talk to the head node | # We will Use netcat as the client to talk to the head node | ||
cp $NFS_BUILD_DIR/bin/nc ./bin | cp $NFS_BUILD_DIR/bin/nc ./bin | ||
#We will use fdisk to check if a local disk exists | #We will use fdisk to check if a local disk exists | ||
# | # | ||
cp $NFS_BUILD_DIR/sbin/fdisk ./sbin | cp $NFS_BUILD_DIR/sbin/fdisk ./sbin | ||
# Modify the init so it will: | # Modify the init so it will: | ||
# 1. Start udev | # 1. Start udev | ||
第640行: | 第589行: | ||
# | # | ||
sed -i '/^load_modules/a /sbin/udevd --daemon\n/sbin/udevadm trigger\n/sbin/udevadm settle\nifconfig eth0 up\n/bin/ipconfig eth0 > /tmp/ipinfo\nmac=\$(cat /tmp/ipinfo \| grep hardware \| cut -d" " -f5)\nif [ -b /dev/sda ] || [ -b /dev/hda ] || [ -b /dev/vda ]; then\n\tboot_method="local"\nelse\n\tboot_method="nfs"\nfi\nprintf "$boot_method,$mac" \| /bin/nc -q2 '"$HEAD_NODE_IP"' 3001\npoweroff' init | sed -i '/^load_modules/a /sbin/udevd --daemon\n/sbin/udevadm trigger\n/sbin/udevadm settle\nifconfig eth0 up\n/bin/ipconfig eth0 > /tmp/ipinfo\nmac=\$(cat /tmp/ipinfo \| grep hardware \| cut -d" " -f5)\nif [ -b /dev/sda ] || [ -b /dev/hda ] || [ -b /dev/vda ]; then\n\tboot_method="local"\nelse\n\tboot_method="nfs"\nfi\nprintf "$boot_method,$mac" \| /bin/nc -q2 '"$HEAD_NODE_IP"' 3001\npoweroff' init | ||
# Compress the initrd | # Compress the initrd | ||
# | # | ||
find . | cpio --quiet --dereference -o -H newc | gzip -9 > $NODE_PXE/discovery-initrd.img | find . | cpio --quiet --dereference -o -H newc | gzip -9 > $NODE_PXE/discovery-initrd.img | ||
popd | popd | ||
rm -rf $INITRD_TMP_DIR | rm -rf $INITRD_TMP_DIR | ||
# Generate the default PXELINUX configuration file | # Generate the default PXELINUX configuration file | ||
printf "PROMPT 0\n" > $DEFAULT_PXE_CONFIG_FILE | printf "PROMPT 0\n" > $DEFAULT_PXE_CONFIG_FILE | ||
第654行: | 第600行: | ||
printf "KERNEL nodes/$RELEASE/$ARCH/$(grep base-kernel $CURRENT_PXE_FILES | cut -d, -f2)\n" >> $DEFAULT_PXE_CONFIG_FILE | printf "KERNEL nodes/$RELEASE/$ARCH/$(grep base-kernel $CURRENT_PXE_FILES | cut -d, -f2)\n" >> $DEFAULT_PXE_CONFIG_FILE | ||
printf "APPEND initrd=nodes/$RELEASE/$ARCH/discovery-initrd.img ip=dhcp\n" >> $DEFAULT_PXE_CONFIG_FILE | printf "APPEND initrd=nodes/$RELEASE/$ARCH/discovery-initrd.img ip=dhcp\n" >> $DEFAULT_PXE_CONFIG_FILE | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第664行: | 第609行: | ||
* Runs netcat indefinitely waiting for node information | * Runs netcat indefinitely waiting for node information | ||
* Registers: | * Registers: | ||
* Node name | ** Node name | ||
* Architecture | ** Architecture | ||
* Ubuntu release | ** Ubuntu release | ||
* Boot method (local disk or NFS) | ** Boot method (local disk or NFS) | ||
* MAC address | ** MAC address | ||
* IP address | ** IP address | ||
Create getMACs and make it executable: | Create getMACs and make it executable: | ||
<pre><nowiki> | <pre><nowiki> | ||
第682行: | 第627行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
# Check to see what our first node number will be. | # Check to see what our first node number will be. | ||
# | # | ||
TEMP_NUMBER=$(tail -n1 $MASTER_NODE_LIST | cut -d, -f1 | sed s/$BASE_NODE_NAME//) | TEMP_NUMBER=$(tail -n1 $MASTER_NODE_LIST | cut -d, -f1 | sed s/$BASE_NODE_NAME//) | ||
if [ ! $TEMP_NUMBER = "" ]; then | if [ ! $TEMP_NUMBER = "" ]; then | ||
第708行: | 第647行: | ||
((NODE_NUMBER++)) | ((NODE_NUMBER++)) | ||
fi | fi | ||
# Infinitely run netcat listening for new nodes | # Infinitely run netcat listening for new nodes | ||
# Update the node list with: | # Update the node list with: | ||
第756行: | 第694行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
updateDHCP () | updateDHCP () | ||
{ | { | ||
case $BOOT_METHOD in | |||
local) | |||
printf "host $NODE_NAME {\n" >> $DHCPD_CONFIG_FILE | |||
printf "\thardware ethernet $MAC_ADDRESS;\n" >> $DHCPD_CONFIG_FILE | |||
printf "\tfixed-address $IP_ADDRESS;\n" >> $DHCPD_CONFIG_FILE | |||
printf "\toption host-name \"$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE | |||
printf "}\n\n" >> $DHCPD_CONFIG_FILE | |||
# | |||
# Create the PXELINUX configuration for the node | |||
# | |||
PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/01-$(printf $MAC_ADDRESS | tr : -)" | |||
printf "PROMPT 0\n" > $PXE_CONFIG_FILE | |||
printf "DEFAULT linux\n\n" >> $PXE_CONFIG_FILE | |||
printf "LABEL linux\n" >> $PXE_CONFIG_FILE | |||
printf "KERNEL nodes/$RELEASE/$ARCH/installer-linux\n" >> $PXE_CONFIG_FILE | |||
printf "APPEND initrd=nodes/$RELEASE/$ARCH/installer-initrd.gz ip=dhcp preseed/url=http://www.home.local/preseed.cfg auto-install/enable debconf/priority=critical locale=en_US console-setup/ask_detect=false console-setup/modelcode=pc105 console-setup/layoutcode=us hw-detect/start_pcmcia=false\n" >> $PXE_CONFIG_FILE | |||
;; | |||
nfs) | |||
printf "host $NODE_NAME {\n" >> $DHCPD_CONFIG_FILE | |||
printf "\thardware ethernet $MAC_ADDRESS;\n" >> $DHCPD_CONFIG_FILE | |||
printf "\tfixed-address $IP_ADDRESS;\n" >> $DHCPD_CONFIG_FILE | |||
printf "\toption host-name \"$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE | |||
printf "\toption root-path \"$NFS_ROOT_EXPORT/$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE | |||
printf "}\n\n" >> $DHCPD_CONFIG_FILE | |||
# | |||
# Create the PXELINUX configuration for the node | |||
# | |||
PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/01-$(printf $MAC_ADDRESS | tr : -)" | |||
printf "PROMPT 0\n" > $PXE_CONFIG_FILE | |||
printf "DEFAULT linux\n\n" >> $PXE_CONFIG_FILE | |||
printf "LABEL linux\n" >> $PXE_CONFIG_FILE | |||
printf "KERNEL nodes/$RELEASE/$ARCH/$(grep base-kernel $CURRENT_PXE_FILES | cut -d, -f2)\n" >> $PXE_CONFIG_FILE | |||
printf "APPEND root=/dev/nfs initrd=nodes/$RELEASE/$ARCH/$(grep nfs-initrd $CURRENT_PXE_FILES | cut -d, -f2) ip=dhcp\n" >> $PXE_CONFIG_FILE | |||
;; | |||
esac | |||
} | } | ||
updateDNS () | updateDNS () | ||
{ | { | ||
printf "$NODE_NAME\t\tIN\tA\t$IP_ADDRESS\n" >> $DNS_FORWARD_CONFIG | |||
# reverse zone | # reverse zone | ||
IP_OCTET=$(printf $IP_ADDRESS | cut -d. -f4) | IP_OCTET=$(printf $IP_ADDRESS | cut -d. -f4) | ||
printf "$IP_OCTET\t\tIN\tPTR\t$NODE_NAME.$DOMAIN_NAME.\n" >> $DNS_REVERSE_CONFIG | printf "$IP_OCTET\t\tIN\tPTR\t$NODE_NAME.$DOMAIN_NAME.\n" >> $DNS_REVERSE_CONFIG | ||
} | } | ||
updateNFSROOT () | updateNFSROOT () | ||
{ | { | ||
# Create the directory filesystem for the node | # Create the directory filesystem for the node | ||
# | # | ||
if [ ! -d $NFS_ROOT_EXPORT/$NODE_NAME ]; then | |||
mkdir -p $NFS_ROOT_EXPORT/$NODE_NAME | |||
rsync -a --numeric-ids $NFS_BUILD_DIR/ $NFS_ROOT_EXPORT/$NODE_NAME | |||
# | |||
# Make filesystem node specific | |||
# | |||
printf "$NODE_NAME\n" > $NFS_ROOT_EXPORT/$NODE_NAME/etc/hostname | |||
printf "$IP_ADDRESS\t$NODE_NAME.$DOMAIN_NAME\t$NODE_NAME\n" >> $NFS_ROOT_EXPORT/$NODE_NAME/etc/hosts | |||
printf "$NODE_NAME... file-system created\n" | |||
else | |||
printf "$NODE_NAME... file-system already exists; not created\n" | |||
fi | |||
} | } | ||
while read LINE ; do | while read LINE ; do | ||
NODE_NAME=$(printf $LINE | cut -d, -f1) | |||
BOOT_METHOD=$(printf $LINE | cut -d, -f4) | |||
MAC_ADDRESS=$(printf $LINE | cut -d, -f5) | |||
IP_ADDRESS=$(printf $LINE | cut -d, -f6) | |||
updateDHCP | |||
printf "$NODE_NAME... DHCP updated\n" | |||
updateDNS | |||
printf "$NODE_NAME... DNS updated\n" | |||
if [ $BOOT_METHOD = "nfs" ]; then | |||
printf "$NODE_NAME... creating file-system\n" | |||
updateNFSROOT | |||
else | |||
printf "$NODE_NAME... local boot; NFS root file-system not created\n" | |||
fi | |||
printf "\n" | |||
done < $NEW_NODES_LIST | done < $NEW_NODES_LIST | ||
# Append the new node list to the master node list | # Append the new node list to the master node list | ||
cat $NEW_NODES_LIST >> $MASTER_NODE_LIST | cat $NEW_NODES_LIST >> $MASTER_NODE_LIST | ||
#clear the new node list | #clear the new node list | ||
> $NEW_NODES_LIST | > $NEW_NODES_LIST | ||
# Restart the DHCP server | # Restart the DHCP server | ||
/etc/init.d/dhcp3-server restart | /etc/init.d/dhcp3-server restart | ||
# Reread DNS zones with RNDC | # Reread DNS zones with RNDC | ||
/usr/sbin/rndc reload | /usr/sbin/rndc reload | ||
# Update the hosts.txt file for parallel-ssh | # Update the hosts.txt file for parallel-ssh | ||
cat $MASTER_NODE_LIST | cut -d, -f1 > $PSSH_HOST_FILE | cat $MASTER_NODE_LIST | cut -d, -f1 > $PSSH_HOST_FILE | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第936行: | 第859行: | ||
xserver-xorg xserver-xorg/autodetect_monitor boolean true | xserver-xorg xserver-xorg/autodetect_monitor boolean true | ||
xserver-xorg xserver-xorg/config/monitor/selection-method \ | xserver-xorg xserver-xorg/config/monitor/selection-method \ | ||
select medium | |||
xserver-xorg xserver-xorg/config/monitor/mode-list \ | xserver-xorg xserver-xorg/config/monitor/mode-list \ | ||
select 1024x768 @ 60 Hz | |||
d-i preseed/late_command string wget -q -O - http://headnode.home.local/preseed_late_command.sh | chroot /target /bin/bash | d-i preseed/late_command string wget -q -O - http://headnode.home.local/preseed_late_command.sh | chroot /target /bin/bash | ||
</nowiki></pre> | </nowiki></pre> | ||
第952行: | 第875行: | ||
Our preseed late command will: | Our preseed late command will: | ||
* Add our NFS mounted home directory to /etc/fstab | * Add our NFS mounted home directory to /etc/fstab | ||
* Modify /etc/ntp.conf to | * Modify /etc/ntp.conf to use our head node as the time server | ||
* Update /etc/apt/sources.list | * Update /etc/apt/sources.list | ||
* Modify the symlink for /bin/sh | * Modify the symlink for /bin/sh | ||
The preseed/late command | The preseed/late command allows us to make changes as if we were logged in to the system. | ||
Example preseed_late_command.sh: | Example preseed_late_command.sh: | ||
<pre><nowiki> | <pre><nowiki> | ||
#/bin/bash | #/bin/bash | ||
# /var/www/preseed_late_command.sh: A script to make configuration changes at the end of installation | # /var/www/preseed_late_command.sh: A script to make configuration changes at the end of installation | ||
# Add our NFS mount to fstab | # Add our NFS mount to fstab | ||
printf "nfs.home.local:/srv/cluster\t/home/cluster\tnfs\trw,auto\t0 0\n" >> /etc/fstab | printf "nfs.home.local:/srv/cluster\t/home/cluster\tnfs\trw,auto\t0 0\n" >> /etc/fstab | ||
# change /etc/hosts file to remove 127.0.1.1 and use the IP address of the node | # change /etc/hosts file to remove 127.0.1.1 and use the IP address of the node | ||
NODE_NAME=$(hostname) | NODE_NAME=$(hostname) | ||
NODE_IP=$(host $NODE_NAME | cut -d" " -f4) | NODE_IP=$(host $NODE_NAME | cut -d" " -f4) | ||
sed -i s/127.0.1.1/$NODE_IP/ /etc/hosts | sed -i s/127.0.1.1/$NODE_IP/ /etc/hosts | ||
# change /etc/ntp.conf to use our NTP server | # change /etc/ntp.conf to use our NTP server | ||
sed -i s/ntp.ubuntu.com/ntp.home.local/ /etc/ntp.conf | sed -i s/ntp.ubuntu.com/ntp.home.local/ /etc/ntp.conf | ||
sed -i '/ntp.home.local/a restrict ntp.home.local mask 255.255.255.255 nomodify notrap noquery' /etc/ntp.conf | sed -i '/ntp.home.local/a restrict ntp.home.local mask 255.255.255.255 nomodify notrap noquery' /etc/ntp.conf | ||
# Clean up apt sources | # Clean up apt sources | ||
# /etc/apt/sources.list | # /etc/apt/sources.list | ||
第978行: | 第897行: | ||
sed -i /#/d /etc/apt/sources.list | sed -i /#/d /etc/apt/sources.list | ||
sed -i /^$/d /etc/apt/sources.list | sed -i /^$/d /etc/apt/sources.list | ||
# Ubuntu symlinks /bin/sh to /bin/dash. Change it to /bin/bash | # Ubuntu symlinks /bin/sh to /bin/dash. Change it to /bin/bash | ||
ln -sf /bin/bash /bin/sh | ln -sf /bin/bash /bin/sh | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第1,008行: | 第925行: | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
printf "We need a password to communicate with disk-ful systems:\n" | printf "We need a password to communicate with disk-ful systems:\n" | ||
第1,034行: | 第951行: | ||
fi | fi | ||
done < $MASTER_NODE_LIST | done < $MASTER_NODE_LIST | ||
# The local copy of ssh known_hosts should now contain host keys for all nodes | # The local copy of ssh known_hosts should now contain host keys for all nodes | ||
# Loop through the nodes once more and copy our .ssh to to each node - This ensure all nodes | # Loop through the nodes once more and copy our .ssh to to each node - This ensure all nodes | ||
第1,055行: | 第971行: | ||
# /opt/cluster/copykeys.exp: A script to automate the password on SSH | # /opt/cluster/copykeys.exp: A script to automate the password on SSH | ||
# so we can copy the keys for passwordless logins | # so we can copy the keys for passwordless logins | ||
set force_conservative 0 ;# set to 1 to force conservative mode even if | set force_conservative 0 ;# set to 1 to force conservative mode even if | ||
;# script wasn't run conservatively originally | |||
if {$force_conservative} { | if {$force_conservative} { | ||
set send_slow {1 .1} | |||
proc send {ignore arg} { | |||
sleep .1 | |||
exp_send -s -- $arg | |||
} | |||
} | } | ||
set NODE_NAME "[lindex $argv 0]" | set NODE_NAME "[lindex $argv 0]" | ||
set SSH_PASSWORD "[lindex $argv 1]" | set SSH_PASSWORD "[lindex $argv 1]" | ||
第1,098行: | 第1,012行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
cut -d, -f1 $MASTER_NODE_LIST > $HEAD_NODE_CONFIG_DIR/hosts.txt | cut -d, -f1 $MASTER_NODE_LIST > $HEAD_NODE_CONFIG_DIR/hosts.txt | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第1,125行: | 第1,034行: | ||
sudo pssh -i -h hosts.txt uptime | sudo pssh -i -h hosts.txt uptime | ||
[1] 00:48:32 [SUCCESS] node81 | [1] 00:48:32 [SUCCESS] node81 | ||
00:48:32 up 4 min, 0 users, load average: 0.05, 0.03, 0.00 | |||
[2] 00:48:33 [SUCCESS] node71 | [2] 00:48:33 [SUCCESS] node71 | ||
00:48:32 up 7 min, 0 users, load average: 0.00, 0.02, 0.01 | |||
[3] 00:48:33 [SUCCESS] node75 | [3] 00:48:33 [SUCCESS] node75 | ||
00:48:32 up 0 min, 0 users, load average: 0.53, 0.17, 0.06 | |||
[4] 00:48:33 [SUCCESS] node73 | [4] 00:48:33 [SUCCESS] node73 | ||
00:48:32 up 6 min, 0 users, load average: 0.00, 0.03, 0.01 | |||
[5] 00:48:33 [SUCCESS] node77 | [5] 00:48:33 [SUCCESS] node77 | ||
00:48:32 up 4 min, 0 users, load average: 0.13, 0.08, 0.03 | |||
[6] 00:48:33 [SUCCESS] node72 | [6] 00:48:33 [SUCCESS] node72 | ||
00:48:32 up 7 min, 0 users, load average: 0.00, 0.02, 0.00 | |||
[7] 00:48:33 [SUCCESS] node79 | [7] 00:48:33 [SUCCESS] node79 | ||
00:48:32 up 3 min, 0 users, load average: 0.04, 0.14, 0.07 | |||
[8] 00:48:33 [SUCCESS] node74 | [8] 00:48:33 [SUCCESS] node74 | ||
00:48:32 up 6 min, 0 users, load average: 0.00, 0.04, 0.02 | |||
[9] 00:48:33 [SUCCESS] node80 | [9] 00:48:33 [SUCCESS] node80 | ||
00:48:32 up 3 min, 0 users, load average: 0.03, 0.07, 0.03 | |||
[10] 00:48:33 [SUCCESS] node76 | [10] 00:48:33 [SUCCESS] node76 | ||
00:48:32 up 5 min, 0 users, load average: 0.08, 0.11, 0.07 | |||
[11] 00:48:33 [SUCCESS] node78 | [11] 00:48:33 [SUCCESS] node78 | ||
00:48:32 up 4 min, 0 users, load average: 0.08, 0.14, 0.08 | |||
# Example installation of a package | # Example installation of a package | ||
sudo pssh -i -h hosts.txt apt-get -y -qq -s install ethtool | sudo pssh -i -h hosts.txt apt-get -y -qq -s install ethtool | ||
第1,194行: | 第1,102行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
printf "Enter name of node to replace:\n" | printf "Enter name of node to replace:\n" | ||
read -p "(e.g. node71): " NODE_TO_REPLACE | read -p "(e.g. node71): " NODE_TO_REPLACE | ||
printf "Enter MAC address of new system:\n" | printf "Enter MAC address of new system:\n" | ||
read -p "(e.g. 00:aa:01:bb:02:cc): " REPLACEMENT_NODE_MAC | read -p "(e.g. 00:aa:01:bb:02:cc): " REPLACEMENT_NODE_MAC | ||
# Convert the MAC address to lower case in case CAPS lock was on | # Convert the MAC address to lower case in case CAPS lock was on | ||
NEW_MAC=$(printf $REPLACEMENT_NODE_MAC | tr '[:upper | NEW_MAC=$(printf $REPLACEMENT_NODE_MAC | tr '[[UbuntuHelp:upper|]]' '[[UbuntuHelp:lower|]]') | ||
LINE=$(grep $NODE_TO_REPLACE $MASTER_NODE_LIST) | LINE=$(grep $NODE_TO_REPLACE $MASTER_NODE_LIST) | ||
if [ ! -z $LINE ]; then | if [ ! -z $LINE ]; then | ||
NODE_NAME=$(printf $LINE | cut -d, -f1) | |||
BOOT_METHOD=$(printf $LINE | cut -d, -f4) | |||
MAC_ADDRESS=$(printf $LINE | cut -d, -f5) | |||
IP_ADDRESS=$(printf $LINE | cut -d, -f6) | |||
case $BOOT_METHOD in | |||
local) | |||
# Update the node list to reflect the new MAC | |||
sed -i s/$MAC_ADDRESS/$NEW_MAC/ $MASTER_NODE_LIST | |||
# Update DHCP to reflect the new MAC | |||
sed -i s/$MAC_ADDRESS/$NEW_MAC/ $DHCPD_CONFIG_FILE | |||
# UPDATE PXELINUX to reflect the new MAC | |||
mv $TFTP_ROOT/pxelinux.cfg/$(printf 01-$MAC_ADDRESS | tr : -) $TFTP_ROOT/pxelinux.cfg/$(printf 01-$NEW_MAC | tr : - ) | |||
# Update the node NFSROOT udev configuration to reflect the new MAC | |||
printf "$NODE_NAME MAC changed from $MAC_ADDRESS to $NEW_MAC\n" | |||
# Changes made. Restart DHCP server | |||
/etc/init.d/dhcp3-server restart | |||
;; | |||
nfs) | |||
# Update the node list to reflect the new MAC | |||
sed -i s/$MAC_ADDRESS/$NEW_MAC/ $MASTER_NODE_LIST | |||
# Update DHCP to reflect the new MAC | |||
sed -i s/$MAC_ADDRESS/$NEW_MAC/ $DHCPD_CONFIG_FILE | |||
# UPDATE PXELINUX to reflect the new MAC | |||
mv $TFTP_ROOT/pxelinux.cfg/$(printf 01-$MAC_ADDRESS | tr : -) $TFTP_ROOT/pxelinux.cfg/$(printf 01-$NEW_MAC | tr : - ) | |||
# Update the node NFSROOT udev configuration to reflect the new MAC | |||
sed -i s/$MAC_ADDRESS/$NEW_MAC/ $NFS_ROOT_EXPORT/$NODE_NAME/etc/udev/rules.d/70-persistent-net.rules | |||
printf "$NODE_NAME MAC changed from $MAC_ADDRESS to $NEW_MAC\n" | |||
# Changes made. Restart DHCP server | |||
/etc/init.d/dhcp3-server restart | |||
;; | |||
esac | |||
else | else | ||
printf "The node name you entered is invalid\n" | printf "The node name you entered is invalid\n" | ||
exit 192 | exit 192 | ||
fi | fi | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第1,273行: | 第1,174行: | ||
# Version 0.1 | # Version 0.1 | ||
# Author: geekshlby | # Author: geekshlby | ||
CONFIG_DIR="/opt/cluster/config" | CONFIG_DIR="/opt/cluster/config" | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
if [ -f $CONFIG_DIR/global.conf ]; then | if [ -f $CONFIG_DIR/global.conf ]; then | ||
source $CONFIG_DIR/global.conf | |||
else | else | ||
printf "Unable to locate the global configuration file.\n" | |||
printf "This script looks for configuration files in:\n" | |||
printf "$CONFIG_DIR\n" | |||
exit 192 | |||
fi | fi | ||
if [ ! $(whoami) = "root" ]; then | if [ ! $(whoami) = "root" ]; then | ||
printf "This script must run with root access.\n" | |||
exit 192 | |||
fi | fi | ||
helpMe () | helpMe () | ||
{ | { | ||
第1,303行: | 第1,199行: | ||
printf "\t\tMultiple nodes names should be separated by white space.\n" | printf "\t\tMultiple nodes names should be separated by white space.\n" | ||
printf "\t\t\te.g. node_on.sh node1 node2\n" | printf "\t\t\te.g. node_on.sh node1 node2\n" | ||
} | |||
if [ -z $1 ]; then | if [ -z $1 ]; then | ||
helpMe | |||
exit 0 | |||
fi | fi | ||
case $1 in | case $1 in | ||
all) | all) | ||
第1,327行: | 第1,221行: | ||
*) | *) | ||
until [ -z $1 ]; do | until [ -z $1 ]; do | ||
NODE_NAME=$1 | |||
MAC_ADDRESS=$(grep -w $NODE_NAME $MASTER_NODE_LIST | cut -d, -f5) | MAC_ADDRESS=$(grep -w $NODE_NAME $MASTER_NODE_LIST | cut -d, -f5) | ||
if [ ! -z $MAC_ADDRESS ]; then | |||
printf "Sending Wake-On-LAN packet to $NODE_NAME at $MAC_ADDRESS\n" | printf "Sending Wake-On-LAN packet to $NODE_NAME at $MAC_ADDRESS\n" | ||
# Send more than one WOL packet to the node | |||
I=0; until [ $I -gt 5 ]; do | I=0; until [ $I -gt 5 ]; do | ||
wakeonlan $MAC_ADDRESS >/dev/null | |||
((I++)); done | ((I++)); done | ||
else | else | ||
第1,342行: | 第1,236行: | ||
;; | ;; | ||
esac | esac | ||
exit 0 | exit 0 | ||
</nowiki></pre> | </nowiki></pre> | ||
第1,348行: | 第1,241行: | ||
<pre><nowiki> | <pre><nowiki> | ||
sh /opt/cluster/nodeOn node1 | sh /opt/cluster/nodeOn node1 | ||
</nowiki></pre> | </nowiki></pre> | ||
To turn on multiple nodes: | To turn on multiple nodes: | ||
第1,364行: | 第1,256行: | ||
* Use a multiple headnode model | * Use a multiple headnode model | ||
* Daemonize some of the processes | * Daemonize some of the processes | ||
* getMacs | ** getMacs | ||
* makeNodes | ** makeNodes | ||
* makePSSH | ** makePSSH | ||
An addition to the above method to deploy Ubuntu, could be a method to monitor the status of the nodes. This can be done with bash of course. Since part of creating the head node was installing the LAMP stack, the following PHP code will parse the master node list and determine if the nodes are online. This too can be improved by using SNMP to communicate status information. | An addition to the above method to deploy Ubuntu, could be a method to monitor the status of the nodes. This can be done with bash of course. Since part of creating the head node was installing the LAMP stack, the following PHP code will parse the master node list and determine if the nodes are online. This too can be improved by using SNMP to communicate status information. | ||
Currently, the following ping.php simply pings the nodes to determine if they are online, and displays a web page with their status: | Currently, the following ping.php simply pings the nodes to determine if they are online, and displays a web page with their status: | ||
第1,375行: | 第1,267行: | ||
<?php | <?php | ||
function ping($host) { | function ping($host) { | ||
exec(sprintf('ping -c 1 -w1 -W1 %s', escapeshellarg($host)), $res, $rval); | |||
return $rval === 0; | |||
} | |||
$green = "/images/status/green.png"; | $green = "/images/status/green.png"; | ||
$red = "/images/status/red.png"; | $red = "/images/status/red.png"; | ||
第1,384行: | 第1,275行: | ||
$hostfile = "status.conf"; | $hostfile = "status.conf"; | ||
$nodefile = "/opt/cluster/config/nodes.txt"; | $nodefile = "/opt/cluster/config/nodes.txt"; | ||
$nodefile_handle = fopen($nodefile, "r"); | $nodefile_handle = fopen($nodefile, "r"); | ||
$counter = 1; | $counter = 1; | ||
第1,392行: | 第1,282行: | ||
echo "<tr>"; | echo "<tr>"; | ||
while (!feof($nodefile_handle)) { | while (!feof($nodefile_handle)) { | ||
$line_of_text = fgets($nodefile_handle); | |||
if (!feof($nodefile_handle)): | |||
$nodeinfo = explode(',', $line_of_text); | |||
$up = ping($nodeinfo[0]); | |||
if ($counter > 2): | |||
echo "<tr>"; | |||
endif; | |||
if ($up == 1): | if ($up == 1): | ||
echo "<td><img border=0 height=49 width=58 alt=Up title=Up src=$green></td><td>$nodeinfo[0] <br> $nodeinfo[1]</td>"; | echo "<td><img border=0 height=49 width=58 alt=Up title=Up src=$green></td><td>$nodeinfo[0] <br> $nodeinfo[1]</td>"; | ||
第1,406行: | 第1,296行: | ||
$counter = $counter + 1; | $counter = $counter + 1; | ||
endif; | endif; | ||
if ($counter > 2): | |||
echo "</tr>"; | |||
$counter = 1; | |||
endif; | endif; | ||
endif; | endif; | ||
第1,421行: | 第1,311行: | ||
The output of the above page will list nodes in 2 columns. | The output of the above page will list nodes in 2 columns. | ||
{|border="1" cellspacing="0" | {|border="1" cellspacing="0" | ||
|https://help.ubuntu.com/community/AutomatedNodeDeployment?action=AttachFile&do=get&target=green.png | |{{https://help.ubuntu.com/community/AutomatedNodeDeployment?action=AttachFile&do=get&target=green.png%7D%7D%7C%7Cnode71.home.local <<BR>>Arch:amd64 Release:intrepid <<BR>>Boot method:local <<BR>>MAC address:08:00:27:0c:09:27 <<BR>>IP address:10.10.1.71||{{https://help.ubuntu.com/community/AutomatedNodeDeployment?action=AttachFile&do=get&target=red.png%7D%7D%7C%7Cnode72.home.local <<BR>>Arch:amd64 Release:intrepid <<BR>>Boot method:nfs <<BR>>MAC address:08:00:27:3f:4e:2c <<BR>>IP address:10.10.1.72 | ||
|} | |} | ||
[[category:UbuntuHelp]] | [[category:UbuntuHelp]] |
2010年5月19日 (三) 16:50的版本
文章出处: |
{{#if: | {{{2}}} | https://help.ubuntu.com/community/AutomatedNodeDeployment }} |
点击翻译: |
English {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/af | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Afrikaans| [[::AutomatedNodeDeployment/af|Afrikaans]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ar | • {{#if: UbuntuHelp:AutomatedNodeDeployment|العربية| [[::AutomatedNodeDeployment/ar|العربية]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/az | • {{#if: UbuntuHelp:AutomatedNodeDeployment|azərbaycanca| [[::AutomatedNodeDeployment/az|azərbaycanca]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/bcc | • {{#if: UbuntuHelp:AutomatedNodeDeployment|جهلسری بلوچی| [[::AutomatedNodeDeployment/bcc|جهلسری بلوچی]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/bg | • {{#if: UbuntuHelp:AutomatedNodeDeployment|български| [[::AutomatedNodeDeployment/bg|български]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/br | • {{#if: UbuntuHelp:AutomatedNodeDeployment|brezhoneg| [[::AutomatedNodeDeployment/br|brezhoneg]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ca | • {{#if: UbuntuHelp:AutomatedNodeDeployment|català| [[::AutomatedNodeDeployment/ca|català]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/cs | • {{#if: UbuntuHelp:AutomatedNodeDeployment|čeština| [[::AutomatedNodeDeployment/cs|čeština]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/de | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Deutsch| [[::AutomatedNodeDeployment/de|Deutsch]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/el | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Ελληνικά| [[::AutomatedNodeDeployment/el|Ελληνικά]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/es | • {{#if: UbuntuHelp:AutomatedNodeDeployment|español| [[::AutomatedNodeDeployment/es|español]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/fa | • {{#if: UbuntuHelp:AutomatedNodeDeployment|فارسی| [[::AutomatedNodeDeployment/fa|فارسی]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/fi | • {{#if: UbuntuHelp:AutomatedNodeDeployment|suomi| [[::AutomatedNodeDeployment/fi|suomi]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/fr | • {{#if: UbuntuHelp:AutomatedNodeDeployment|français| [[::AutomatedNodeDeployment/fr|français]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/gu | • {{#if: UbuntuHelp:AutomatedNodeDeployment|ગુજરાતી| [[::AutomatedNodeDeployment/gu|ગુજરાતી]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/he | • {{#if: UbuntuHelp:AutomatedNodeDeployment|עברית| [[::AutomatedNodeDeployment/he|עברית]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/hu | • {{#if: UbuntuHelp:AutomatedNodeDeployment|magyar| [[::AutomatedNodeDeployment/hu|magyar]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/id | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Bahasa Indonesia| [[::AutomatedNodeDeployment/id|Bahasa Indonesia]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/it | • {{#if: UbuntuHelp:AutomatedNodeDeployment|italiano| [[::AutomatedNodeDeployment/it|italiano]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ja | • {{#if: UbuntuHelp:AutomatedNodeDeployment|日本語| [[::AutomatedNodeDeployment/ja|日本語]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ko | • {{#if: UbuntuHelp:AutomatedNodeDeployment|한국어| [[::AutomatedNodeDeployment/ko|한국어]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ksh | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Ripoarisch| [[::AutomatedNodeDeployment/ksh|Ripoarisch]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/mr | • {{#if: UbuntuHelp:AutomatedNodeDeployment|मराठी| [[::AutomatedNodeDeployment/mr|मराठी]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ms | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Bahasa Melayu| [[::AutomatedNodeDeployment/ms|Bahasa Melayu]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/nl | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Nederlands| [[::AutomatedNodeDeployment/nl|Nederlands]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/no | • {{#if: UbuntuHelp:AutomatedNodeDeployment|norsk| [[::AutomatedNodeDeployment/no|norsk]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/oc | • {{#if: UbuntuHelp:AutomatedNodeDeployment|occitan| [[::AutomatedNodeDeployment/oc|occitan]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/pl | • {{#if: UbuntuHelp:AutomatedNodeDeployment|polski| [[::AutomatedNodeDeployment/pl|polski]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/pt | • {{#if: UbuntuHelp:AutomatedNodeDeployment|português| [[::AutomatedNodeDeployment/pt|português]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ro | • {{#if: UbuntuHelp:AutomatedNodeDeployment|română| [[::AutomatedNodeDeployment/ro|română]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/ru | • {{#if: UbuntuHelp:AutomatedNodeDeployment|русский| [[::AutomatedNodeDeployment/ru|русский]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/si | • {{#if: UbuntuHelp:AutomatedNodeDeployment|සිංහල| [[::AutomatedNodeDeployment/si|සිංහල]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/sq | • {{#if: UbuntuHelp:AutomatedNodeDeployment|shqip| [[::AutomatedNodeDeployment/sq|shqip]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/sr | • {{#if: UbuntuHelp:AutomatedNodeDeployment|српски / srpski| [[::AutomatedNodeDeployment/sr|српски / srpski]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/sv | • {{#if: UbuntuHelp:AutomatedNodeDeployment|svenska| [[::AutomatedNodeDeployment/sv|svenska]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/th | • {{#if: UbuntuHelp:AutomatedNodeDeployment|ไทย| [[::AutomatedNodeDeployment/th|ไทย]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/tr | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Türkçe| [[::AutomatedNodeDeployment/tr|Türkçe]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/vi | • {{#if: UbuntuHelp:AutomatedNodeDeployment|Tiếng Việt| [[::AutomatedNodeDeployment/vi|Tiếng Việt]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/yue | • {{#if: UbuntuHelp:AutomatedNodeDeployment|粵語| [[::AutomatedNodeDeployment/yue|粵語]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/zh | • {{#if: UbuntuHelp:AutomatedNodeDeployment|中文| [[::AutomatedNodeDeployment/zh|中文]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/zh-hans | • {{#if: UbuntuHelp:AutomatedNodeDeployment|中文(简体)| [[::AutomatedNodeDeployment/zh-hans|中文(简体)]]}}|}} {{#ifexist: {{#if: UbuntuHelp:AutomatedNodeDeployment | UbuntuHelp:AutomatedNodeDeployment | {{#if: | :}}AutomatedNodeDeployment}}/zh-hant | • {{#if: UbuntuHelp:AutomatedNodeDeployment|中文(繁體)| [[::AutomatedNodeDeployment/zh-hant|中文(繁體)]]}}|}} |
{{#ifeq:UbuntuHelp:AutomatedNodeDeployment|:AutomatedNodeDeployment|请不要直接编辑翻译本页,本页将定期与来源同步。}} |
{{#ifexist: :AutomatedNodeDeployment/zh | | {{#ifexist: AutomatedNodeDeployment/zh | | {{#ifeq: {{#titleparts:AutomatedNodeDeployment|1|-1|}} | zh | | }} }} }} {{#ifeq: {{#titleparts:AutomatedNodeDeployment|1|-1|}} | zh | | }}
Setting up an Ubuntu System to deploy Ubuntu
Installing an operating system on multiple systems can be a complicated and daunting task. A check-list may be employed to ensure all systems have similar if not identical configurations. We will employ tools to automate the process not only for disk-full systems, but will include disk-less systems as well. Many "how to" documents exist on automating the installation of the Linux operating system to include making hard drive images and scripted installations. Many how to documents exist on how to set up various services in Linux as well. This document was created with the intent of being a comprehensive document that will allow the easy deployment of an Ubuntu HPC cluster. We will not concentrate on parallel computing, only on the deployment of the nodes to support a HPC Cluster. Bash will be our main tool for the automation. I will explain the purpose of each script as you encounter them on this page.
Prerequisites
Required
- TFTP Server
- Syslinux
- DHCP Server
- NFS Server
- Debootstrap
- PXE capable NICs
Optional
- Wake-on-LAN
- Apache
- Apt-mirror
- BIND
- Whois (for the mkpassword command)
- Expect
I highly recommend the optional requirements as they will ease the installation of the operating system.
Getting Started
Install Ubuntu 9.04 (Jaunty Jackalope) on a system that you will use as the deployment server. I prefer a minimal install and then install required/optional packages as needed. I maintain all the scripts in /opt/cluster, however, you can modify them to be placed in any location. All bash scripts make reference to a global configuration file. Having common variables in a single location eases the coding involved in other scripts, however, may cause confusion when encountering unknown variables when you are making changes to the code.
- /opt/cluster will contain bash scripts
- /opt/cluster/config will contain configuration information
Let's create the directory that will contain our scripts and various configuration files:
mkdir -p /opt/cluster/config
Global Configuration
The global configuration file /opt/cluster/config/global.conf contains information related to:
- Network
- Server identification
- File locations
- Operating system
- Packages
- Nodes
Create the common configuration file that is sourced by all other scripts. - Please note: a file that is sourced does not need to start with a shebang. The shebang i.e #!/bin/bash may be missing from the other scripts due to the wiki mark-up Create the global configuration:
touch /opt/cluster/config/global.conf
# /opt/cluster/config/global.conf: configuration file for node deployment # This file is sourced by files used by the deployment system # Site information DOMAIN_NAME="home.local" DOMAIN_ADMIN="root" NETWORK="10.10.1.0" SUBNET_MASK="255.255.255.0" BROADCAST="10.10.1.255" ROUTER_NAME="router" ROUTER_IP="10.10.1.1" NAME_SERVER_NAME="dns" NAME_SERVER_IP="10.10.1.10" NTP_SERVER_NAME="ntp" NTP_SERVER_IP="10.10.1.10" DHCP_SERVER_NAME="dhcp" DHCP_SERVER_IP="10.10.1.10" HTTP_SERVER_NAME="www" HTTP_SERVER_IP="10.10.1.10" PROXY_SERVER_NAME="proxy" PROXY_SERVER_IP="10.10.1.10" TFTP_SERVER_NAME="tftp" TFTP_SERVER_IP="10.10.1.10" NFS_SERVER_NAME="nfs" NFS_SERVER_IP="10.10.1.10" MIRROR_SERVER_NAME="mirror" MIRROR_SERVER_IP="10.10.1.10" # Service information DHCPD_CONFIG_FILE="/etc/dhcp3/dhcpd.conf" DNS_CONFIG_FILE="/etc/bind/named.conf.local" DNS_FORWARD_CONFIG="/etc/bind/db.$DOMAIN_NAME" DNS_REVERSE_CONFIG="/etc/bind/db.10.10.1" UBUNTU_MIRROR_URL="http://$MIRROR_SERVER_NAME.$DOMAIN_NAME/ubuntu" NFS_CONFIG_FILE="/etc/exports" NFS_ROOT_EXPORT="/srv/nfsroot" NFS_HOME_EXPORT="/srv/cluster" OFFICIAL_MIRROR="us.archive.ubuntu.com/ubuntu" MIRROR_LIST_FILE="/etc/apt/mirror.list" TFTP_ROOT="/var/lib/tftpboot" DEFAULT_PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/default" # NODE information HEAD_NODE="headnode" HEAD_NODE_IP="10.10.1.10" HEAD_NODE_WORKING_DIR="/root" HEAD_NODE_CONFIG_DIR="/opt/cluster/config" BASE_NODE_NAME="node" NODE_NUMBER=71 MASTER_NODE_LIST="$HEAD_NODE_CONFIG_DIR/nodes.txt" NEW_NODES_LIST="$HEAD_NODE_CONFIG_DIR/new_nodes.txt" PSSH_HOST_FILE="$HEAD_NODE_CONFIG_DIR/hosts.txt" # Undiscovered node DHCP Range DHCP_RANGE_START="10.10.1.100" DHCP_RANGE_STOP="10.10.1.200" NODE_USER="cluster" NODE_USER_UID=1010 # NFS Root filesystem information ARCH="amd64" RELEASE="jaunty" PRE_INST_PKGS="language-pack-en,language-pack-en-base,vim,wget,openssh-server,ntp,nfs-common" PRE_INST_EXCL_PKGS="ubuntu-minimal" # separated by a comma POST_INST_PKGS="linux-image-server" #separated by a space PKGS_TO_PURGE="" # separated by a space NODE_PXE="$TFTP_ROOT/nodes/$RELEASE/$ARCH" REPOSITORY="main restricted universe multiverse" NFS_BUILD_DIR="$HEAD_NODE_WORKING_DIR/nfsroot/$RELEASE/$ARCH" APT_SOURCES_FILE="$NFS_BUILD_DIR/etc/apt/sources.list" FSTAB_FILE="$NFS_BUILD_DIR/etc/fstab" HOSTNAME_FILE="$NFS_BUILD_DIR/etc/hostname" NTP_CONF_FILE="$NFS_BUILD_DIR/etc/ntp.conf" INTERFACE_FILE="$NFS_BUILD_DIR/etc/network/interfaces" HOSTS_FILE="$NFS_BUILD_DIR/etc/hosts" CURRENT_PXE_FILES="$HEAD_NODE_CONFIG_DIR/pxefiles.txt" printf "Global configration file loaded\n"
The global configuration file can be sourced with either
- # /opt/cluster/config/global.conf
- source /opt/cluster/config/global.conf
As you read on, you will notice most if not all scripts will refer to /opt/cluster/config/global.conf. global.conf sections:
- Site information – identifies how our site is configured
- Service information – identifies various services and their related configuration files
- Node information – identifies node information. Please take node of the variable NODE as it identifies the name of the first node in the cluster.
- NFS Root file-system information – identifies information related to creating our NFS root for disk-less systems.
You will notice that Site information uses the same IP address for multiple servers. This is from dedicating a “head node” to systems deployment. You can substitute pre-existing servers to perform some of the same roles.
Head Node Creation
Now that we have defined our global configuration, we will now install the necessary software to make our system the head node. If we can automate the installation of nodes, we should be able to automate the installation and configuration of our deployment server. The script to create the “head node”:
- Creates our cluster user with a specific ID and home directory
- Installs openssh
- Installs debootstrap
- Installs apt-mirror (see Note(1) regarding apt-mirror located after script)
- Configures /etc/apt/mirror.list
- Installs TFTP
- Installs syslinux
- Installs ISC DHCP Server
- Configures the scope for the DHCP server
- Installs NTP
- Configures NTP
- Installs BIND
- Configures the forward look-up zone
- Configures the reverse look-up zone
- Installs NFS server
- Configures our exports
- Install the LAMP stack (see Note(2) regarding LAMP located after script)
- Makes our Ubuntu mirror available via HTTP
- Creates our initial node list (blank to begin)
- Installs parallel-ssh
- Installs expect
- Installs wakeonlan
Create makeHeadNode make it executable:
touch /opt/cluster/makeHeadNode chmod u+x /opt/cluster/makeHeadNode
The following bash script will install the necessary software to allow our head node to be the system used to deploy others, and configure the services provided by each package as it is installed:
#!/bin/bash # # /opt/cluster/makeHeadNode: A script to install necessary software to become the head node # used for node deployment. # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi # Install packages on Head Node # Create the cluster user and prompt for password adduser --uid $NODE_USER_UID --home $NFS_HOME_EXPORT --gecos $NODE_USER $NODE_USER # SSH server tasksel install openssh-server # Debootstrap apt-get -y install debootstrap # Apt-mirror apt-get -y install apt-mirror # Since we don't know what release the apt-mirror package defaults to, # create the mirror list from scratch e.g. installing on intrepid the mirror list # was configured for hardy (this was later fixed in an updated intrepid package) # after we have mirrored the repositories, we can point /etc/apt/sources.list to our local repository printf "set nthreads\t\n" > $MIRROR_LIST_FILE printf "set _tilde 0\n\n" >> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE main restricted universe multiverse\n" >> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE-updates main restricted universe multiverse\n" >> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE-security main restricted universe multiverse\n" >> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE main/debian-installer restricted/debian-installer universe/debian-installer multiverse/debian-installer" >> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE-updates main/debian-installer universe/debian-installer\n">> $MIRROR_LIST_FILE printf "deb http://$OFFICIAL_MIRROR $RELEASE-security main/debian-installer\n\n" >> $MIRROR_LIST_FILE printf "clean http://$OFFICIAL_MIRROR\n" >> $MIRROR_LIST_FILE # TFTP Server apt-get -y install tftpd-hpa /etc/init.d/openbsd-inetd stop update-rc.d -f openbsd-inetd remove sed -i s/no/yes/ /etc/default/tftpd-hpa /etc/init.d/tftpd-hpa start # Syslinux apt-get -y install syslinux cp /usr/lib/syslinux/pxelinux.0 $TFTP_ROOT mkdir -p $TFTP_ROOT/pxelinux.cfg touch $DEFAULT_PXE_CONFIG_FILE # DHCP Server apt-get -y install dhcp3-server # Global options for DHCP that will apply to all client DDNS_STYLE="none" DEFAULT_LEASE_TIME="86400" MAX_LEASE_TIME="604800" TIME_OFFSET="-18000" AUTHORITATIVE="authoritative" LOG_FACILITY="local7" ALLOW_BOOT="booting" ALLOW_BOOTP="bootp" FILENAME="pxelinux.0" # Undiscovered nodes GET_LEASE_NAMES="on" USE_HOST_DECL_NAME="on" # Generate a new base dhcpd.conf printf "#Global options\n\n" > $DHCPD_CONFIG_FILE printf "ddns-update-style $DDNS_STYLE;\n" >> $DHCPD_CONFIG_FILE printf "option domain-name \"$DOMAIN_NAME\";\n" >> $DHCPD_CONFIG_FILE printf "option domain-name-servers $NAME_SERVER_IP;\n" >> $DHCPD_CONFIG_FILE printf "option ntp-servers $NTP_SERVER_IP;\n" >> $DHCPD_CONFIG_FILE printf "option routers $ROUTER_IP;\n" >> $DHCPD_CONFIG_FILE printf "option subnet-mask $SUBNET_MASK;\n" >> $DHCPD_CONFIG_FILE printf "option broadcast-address $BROADCAST;\n" >> $DHCPD_CONFIG_FILE printf "default-lease-time $DEFAULT_LEASE_TIME;\n" >> $DHCPD_CONFIG_FILE printf "max-lease-time $MAX_LEASE_TIME;\n" >> $DHCPD_CONFIG_FILE printf "option time-offset $TIME_OFFSET;\n" >> $DHCPD_CONFIG_FILE printf "$AUTHORITATIVE;\n" >> $DHCPD_CONFIG_FILE printf "log-facility $LOG_FACILITY;\n" >> $DHCPD_CONFIG_FILE printf "allow $ALLOW_BOOT;\n" >> $DHCPD_CONFIG_FILE printf "allow $ALLOW_BOOTP;\n" >> $DHCPD_CONFIG_FILE printf "filename \"$FILENAME\";\n" >> $DHCPD_CONFIG_FILE printf "next-server $TFTP_SERVER_IP;\n\n" >> $DHCPD_CONFIG_FILE printf "#Undiscovered nodes\n\n" >> $DHCPD_CONFIG_FILE printf "subnet $NETWORK netmask $SUBNET_MASK {\n" >> $DHCPD_CONFIG_FILE printf "\tget-lease-hostnames $GET_LEASE_NAMES;\n" >> $DHCPD_CONFIG_FILE printf "\tuse-host-decl-names $USE_HOST_DECL_NAME;\n" >> $DHCPD_CONFIG_FILE printf "\trange $DHCP_RANGE_START $DHCP_RANGE_STOP;\n" >> $DHCPD_CONFIG_FILE printf "}\n\n" >> $DHCPD_CONFIG_FILE printf "#Begin reservations for discovered nodes\n\n" >> $DHCPD_CONFIG_FILE /etc/init.d/dhcp3-server start # NTP Server apt-get -y install ntp sed -i s/server\ ntp.ubuntu.com/server\ us.pool.ntp.org/ /etc/ntp.conf sed -i '/us.pool.ntp.org/a server ntp.ubuntu.com\nrestrict us.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery\nrestrict ntp.ubuntu.com mask 255.255.255.255 nomodify notrap noquery\nrestrict 10.10.1.0 mask 255.255.255.0 nomodify notrap' /etc/ntp.conf /etc/init.d/ntp restart # DNS Server apt-get -y install bind9 printf "zone \"$DOMAIN_NAME\" {\n" > $DNS_CONFIG_FILE printf "\ttype master;\n" >> $DNS_CONFIG_FILE printf "\tnotify no;\n" >> $DNS_CONFIG_FILE printf "\tfile \"$DNS_FORWARD_CONFIG\";\n" >> $DNS_CONFIG_FILE printf "};\n\n" >> $DNS_CONFIG_FILE printf "zone \"1.10.10.in-addr.arpa\" {\n" >> $DNS_CONFIG_FILE printf "\ttype master;\n" >> $DNS_CONFIG_FILE printf "\tnotify no;\n" >> $DNS_CONFIG_FILE printf "\tfile \"$DNS_REVERSE_CONFIG\";\n" >> $DNS_CONFIG_FILE printf "};\n" >> $DNS_CONFIG_FILE # Config for forward zone printf ";\n; BIND data file for $DOMAIN_NAME domain\n;\n" > $DNS_FORWARD_CONFIG printf "\$TTL\t604800\n" >> $DNS_FORWARD_CONFIG printf "@\t\tIN\tSOA\t$HEAD_NODE.$DOMAIN_NAME.\t$DOMAIN_ADMIN.$DOMAIN_NAME. (\n" >> $DNS_FORWARD_CONFIG printf "\t\t\t\t1\t\t; Serial\n" >> $DNS_FORWARD_CONFIG printf "\t\t\t\t604800\t\t; Refresh\n" >> $DNS_FORWARD_CONFIG printf "\t\t\t\t86400\t\t; Retry\n" >> $DNS_FORWARD_CONFIG printf "\t\t\t\t2419200\t\t; Expire\n" >> $DNS_FORWARD_CONFIG printf "\t\t\t\t604800 )\t; Negative Cache TTL\n" >> $DNS_FORWARD_CONFIG printf ";\n" >> $DNS_FORWARD_CONFIG printf "\t\t\tNS\t$NAME_SERVER_NAME.$DOMAIN_NAME.\n" >> $DNS_FORWARD_CONFIG printf "\t\t\tMX\t10 $HEAD_NODE.$DOMAIN_NAME.\n" >> $DNS_FORWARD_CONFIG printf "@\t\tIN\tA\t$HEAD_NODE_IP\n" >> $DNS_FORWARD_CONFIG printf "$ROUTER_NAME\t\tIN\tA\t$ROUTER_IP\n" >> $DNS_FORWARD_CONFIG printf "$HEAD_NODE\t\tIN\tA\t$HEAD_NODE_IP\n" >> $DNS_FORWARD_CONFIG printf "$NAME_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$DHCP_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$NTP_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$NFS_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$MIRROR_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$NTP_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$PROXY_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$HTTP_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n" >> $DNS_FORWARD_CONFIG printf "$TFTP_SERVER_NAME\t\tIN\tCNAME\t$HEAD_NODE\n\n" >> $DNS_FORWARD_CONFIG # Generate names for undiscovered nodes printf ";Automatic name generation for undiscovered nodes\n" >> $DNS_FORWARD_CONFIG printf "\$GENERATE $(printf $DHCP_RANGE_START | cut -d. -f4)-$(printf $DHCP_RANGE_STOP | cut -d. -f4) dhcp-\$\tIN\tA\t$(printf $NETWORK | sed 's\0$\\')\$\n\n" >> $DNS_FORWARD_CONFIG printf ";Discovered nodes\n" >> $DNS_FORWARD_CONFIG # Config for reverse zone printf ";\n; BIND reverse data file for $DOMAIN_NAME domain\n;\n" > $DNS_REVERSE_CONFIG printf "\$TTL\t604800\n" >> $DNS_REVERSE_CONFIG printf "@\t\tIN\tSOA\t$NAME_SERVER_NAME.$DOMAIN_NAME.\t$DOMAIN_ADMIN.$DOMAIN_NAME. (\n" >> $DNS_REVERSE_CONFIG printf "\t\t\t\t1\t\t; Serial\n" >> $DNS_REVERSE_CONFIG printf "\t\t\t\t604800\t\t; Refresh\n" >> $DNS_REVERSE_CONFIG printf "\t\t\t\t86400\t\t; Retry\n" >> $DNS_REVERSE_CONFIG printf "\t\t\t\t2419200\t\t; Expire\n" >> $DNS_REVERSE_CONFIG printf "\t\t\t\t604800 )\t; Negative Cache TTL\n" >> $DNS_REVERSE_CONFIG printf ";\n" >> $DNS_REVERSE_CONFIG printf "\t\tIN\tNS\t$NAME_SERVER_NAME.$DOMAIN_NAME.\n" >> $DNS_REVERSE_CONFIG printf "$(printf $ROUTER_IP | cut -d. -f4)\t\tIN\tPTR\t$ROUTER_NAME.$DOMAIN_NAME.\n" >> $DNS_REVERSE_CONFIG printf "$(printf $HEAD_NODE_IP | cut -d. -f4)\t\tIN\tPTR\t$HEAD_NODE.$DOMAIN_NAME.\n\n" >> $DNS_REVERSE_CONFIG # Generate reverse names for undiscovered nodes printf ";Automatic name generation for undiscovered nodes\n" >> $DNS_REVERSE_CONFIG printf "\$GENERATE $(printf $DHCP_RANGE_START | cut -d. -f4)-$(printf $DHCP_RANGE_STOP | cut -d. -f4) \$\t\tIN\tPTR\tdhcp-\$.$DOMAIN_NAME.\n\n" >> $DNS_REVERSE_CONFIG printf ";Discovered nodes\n" >> $DNS_REVERSE_CONFIG /etc/init.d/bind9 restart #Make ourself the resolver printf "domain $DOMAIN_NAME\n" > /etc/resolv.conf printf "search $DOMAIN_NAME\n" >> /etc/resolv.conf printf "nameserver $NAME_SERVER_IP\n" >> /etc/resolv.conf # NFS Server apt-get -y install nfs-kernel-server mkdir -p $NFS_ROOT_EXPORT printf "$NFS_ROOT_EXPORT\t\t$NETWORK/24(rw,sync,no_root_squash,no_subtree_check)\n" > $NFS_CONFIG_FILE printf "$NFS_HOME_EXPORT\t\t$NETWORK/24(rw,sync,no_root_squash,no_subtree_check)\n" >> $NFS_CONFIG_FILE exportfs -a # Web Server tasksel install lamp-server ln -s /var/spool/apt-mirror/mirror/us.archive.ubuntu.com/ubuntu /var/www/ubuntu # Create an empty node list > $MASTER_NODE_LIST > $NEW_NODES_LIST # Install parallel-ssh apt-get -y install pssh for PUSHTOPLACE in /root /srv/cluster; do pushd $PUSHTOPLACE printf "alias /usr/bin/parallel-ssh='pssh -h /opt/cluster/config/hosts.txt'\n" >> $PUSHTOPLACE/.bashrc printf "alias /usr/bin/parallel-scp='pscp -h /opt/cluster/config/hosts.txt'\n" >> $PUSHTOPLACE/.bashrc printf "alias /usr/bin/parallel-rsync='prsync -h /opt/cluster/config/hosts.txt'\n" >> $PUSHTOPLACE/.bashrc printf "alias /usr/bin/parallel-nuke='pnuke -h /opt/cluster/config/hosts.txt'\n" >> $PUSHTOPLACE/.bashrc printf "alias /usr/bin/parallel-slurp='pslurp -h /opt/cluster/config/hosts.txt'\n" >> $PUSHTOPLACE/.bashrc popd done # Install expect apt-get -y install expect # Install wake-on-LAN apt-get -y install wakeonlan exit 0
Note(1) Apt-mirror is a tool which will allow you to maintain a local mirror of the Ubuntu repositories. You will be able to make the mirror available via HTTP, FTP, NFS, etc. In our scenario, we will use HTTP which will require the installation of a Web Server. Mirroring the repository (to include main, updates, multiverse, universe, and restricted for the amd64 architecture of a single release will require approximately 24 GB of space. The length of time required for the mirror process to complete is dependent upon several factors, one of which is bandwidth. It took me approximately 2.5 days via DSL. Apt-mirror is not a requirement for this exercise, however, greatly reduces the length of time to deploy an Ubuntu system. I recommend apt-mirror to be installed. If you choose to use apt-mirror, you can update /etc/apt/sources.list to point to the local repository. This can be included in the makeHeadNode script. Note(2) The entire LAMP stack is not needed. Since we will be providing the Ubuntu repositories via HTTP as well as the preseed configuration file, all that is needed is Apache. I prefer the LAMP stack as it provides a multitude of possibilities. Some of which will be appended to the end of this document.
Bootstrap an Ubuntu Installation
Now that our prerequisite software is installed and configured we will create a base installation for use by disk-less nodes. The script to bootstrap an Ubuntu installation:
- Archives the previous bootstrap installation if one was completed for the same release and architechure
- Performs a bootstrap installation based upon variables sourced from /opt/cluster/global.conf
- Updates the bootstrap installation:
- Updates /etc/apt/sources.list to use our local repository
- Updates /etc/fstab to mount our cluster user home directory
- Updates the bootstrap build environment with a generic host name
- Updates /etc/ntp.conf to use our own NTP server
- Updates /etc/apt/network/interfaces to be a generic configuration
- Chroots into out build environment and installs additional software based upon /opt/cluster/global.conf
- Chroots into our build environment and removes software based upon /opt/cluster/global.conf
- Updates the build environment's initramfs to be NFS aware and copies it to out TFTP server root directory
- Prompts for a root password
- Creates our cluster user with a specific UID and home directory
- Determines what kernel and initramfs was installed
- Copies the build environment kernel and initramfs to our TFTP server root directory
- Downloads the Ubuntu installer from an official repository to out TFTP server root directory (see Note(3) following the script)
Create buildBase and make it executable:
touch /opt/cluster/buildBase chmod u+x /opt/cluster/buildBase
The following script will create our bootstrapped system to be used as our NFS root file-system:
#!/bin/bash # # /opt/cluster/buildBase: A script to boostrap an Ubuntu system # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi progress () { DOT=0 until [ $DOT -gt 30 ]; do printf "." ((DOT++)) done printf "\n" } # Check to see if we have used debootstrap for this same release and architecture on a previous ocassion. if [ -d $NFS_BUILD_DIR ]; then printf "Archiving previous installation...\n" tar --checkpoint=100 --checkpoint-action=exec='printf "."' -czf $HEAD_NODE_WORKING_DIR/nfsroot-$(date +%m-%d-%y).tgz $NFS_BUILD_DIR printf "\n" printf "Removing previous installation..."; progress rm -rf $NFS_BUILD_DIR else mkdir -p $NFS_BUILD_DIR fi # Bootstrap the installation printf "Performing a bootstrap of $RELEASE-$ARCH..."; progress debootstrap --arch $ARCH --include=$PRE_INST_PKGS --exclude=$PRE_INST_EXCL_PKGS $RELEASE $NFS_BUILD_DIR $UBUNTU_MIRROR_URL # Reconfigure our locale and time zone printf "Updating locale and timezone..."; progress chroot $NFS_BUILD_DIR dpkg-reconfigure locales chroot $NFS_BUILD_DIR dpkg-reconfigure tzdata # Update the apt sources printf "Updating apt sources..."; progress printf "deb $UBUNTU_MIRROR_URL $RELEASE $REPOSITORY\n" > $APT_SOURCES_FILE printf "deb $UBUNTU_MIRROR_URL $RELEASE-updates $REPOSITORY\n" >> $APT_SOURCES_FILE printf "deb $UBUNTU_MIRROR_URL $RELEASE-security $REPOSITORY\n" >> $APT_SOURCES_FILE # Update fstab printf "Updating fstab to include proc..."; progress printf "proc\t\t/proc\t\tproc\tdefaults\t0\t0\n" >> $FSTAB_FILE printf "Updating fstab to include CD-ROM..."; progress printf "/dev/scd0\t/media/cdrom\tudf,iso9660\tuser,noauto,exec,utf8\t0\t0\n" >> $FSTAB_FILE printf "Updating fstab to include $NFS_USER\'s NFS home directory..."; progress printf "$NFS_SERVER_NAME.$DOMAIN_NAME:$NFS_HOME_EXPORT\t/home/cluster\tnfs\trw,auto\t0 0\n" >> $FSTAB_FILE # Update the build environment with a generic host-name # The hostname defaults to the host-name of the system on which it was built printf "Updating build environment with generic host name..."; progress printf "$BASE_NODE_NAME\n" > $HOSTNAME_FILE # Update NTP configuration to sync with local time server printf "Updating NTP configuration..."; progress sed -i s/ntp.ubuntu.com/$NTP_SERVER_NAME.$DOMAIN_NAME/ $NTP_CONF_FILE sed -i "/$NTP_SERVER_NAME.$DOMAIN_NAME/a restrict $NTP_SERVER_NAME.$DOMAIN_NAME mask 255.255.255.255 nomodify notrap noquery" $NTP_CONF_FILE # Prepare a generic network interfaces config printf "Updating build environment with generic interface configuration..."; progress printf "#The loopback network interface\n" >> $INTERFACE_FILE printf "auto lo\n" >> $INTERFACE_FILE printf "iface lo inet loopback\n\n" >> $INTERFACE_FILE printf "#The primary network interface\n" >> $INTERFACE_FILE printf "auto eth0\n" >> $INTERFACE_FILE printf "iface eth0 inet manual\n" >> $INTERFACE_FILE printf "Updating hosts files to include loopback interface..."; progress printf "127.0.0.1\tlocalhost\n" > $HOSTS_FILE # Mount proc in our build environment and ensure our base system is at the latest versions printf "Updating packages to most recent versions..."; progress mount proc $NFS_BUILD_DIR/proc -t proc chroot $NFS_BUILD_DIR apt-get -y update chroot $NFS_BUILD_DIR apt-get -y dist-upgrade # Install additional packages printf "Installing additional packages..."; progress for PACKAGE in $POST_INST_PKGS; do chroot $NFS_BUILD_DIR apt-get -q -y install $PACKAGE done # Remove packages printf "Purging packages..."; progress for PACKAGE in $PKGS_TO_PURGE; do chroot $NFS_BUILD_DIR apt-get -y remove --purge $PACKAGE done # Determine the name of the kernel and initrd that was installed BASE_KERNEL_FILE=$(basename `ls $NFS_BUILD_DIR/boot/vmlinuz*`) BASE_INITRD_FILE=$(basename `ls $NFS_BUILD_DIR/boot/initrd*`) ORIGINAL_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-original/) NFS_INITRD_FILE=$(printf $BASE_INITRD_FILE | sed s/img/img-nfs/) # Copy the unmodified initramfs to our TFTP root printf "Copying unmodifed inittf to TPTP server..."; progress mkdir -p $NODE_PXE printf "base-initrd,$ORIGINAL_INITRD_FILE\n" > $CURRENT_PXE_FILES cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$ORIGINAL_INITRD_FILE # Update the initramfs to reflect a NFS root # We can do this before the kernel is installed in the chroot, howerver, we want to # have both disk-less and disk-ful node support printf "Updating initramfs to reflect an NFS root..."; progress sed -i.orig s/BOOT=local/BOOT=nfs/ $NFS_BUILD_DIR/etc/initramfs-tools/initramfs.conf chroot $NFS_BUILD_DIR update-initramfs -u # Install ubuntu-server printf "Installing ubuntu-server..."; progress chroot $NFS_BUILD_DIR tasksel install server # Clean up the package cache printf "Cleaning package cache..."; progress chroot $NFS_BUILD_DIR apt-get clean # Remove the /etc/rcS.d init script which parses /etc/fstab and replace it # with the one located in /etc/network/if-up.d # See https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bugs/275451 # for details chroot $NFS_BUILD_DIR unlink /etc/rcS.d/S45mountnfs.sh chroot $NFS_BUILD_DIR ln -s /etc/network/if-up.d/mountnfs /etc/rcS.d/S45mountnfs # set a root password for printf "Setting password for root..."; progress chroot $NFS_BUILD_DIR passwd # create the node user and ask us for password printf "Setting password for $NODE_USER..."; progress chroot $NFS_BUILD_DIR adduser --uid $NODE_USER_UID --gecos $NODE_USER $NODE_USER # Unmount proc from our build environment umount $NFS_BUILD_DIR/proc # Copy the kernel to our TFTP root printf "Copying kernel to TFTP server..."; progress printf "base-kernel,$BASE_KERNEL_FILE\n" >> $CURRENT_PXE_FILES cp $NFS_BUILD_DIR/boot/$BASE_KERNEL_FILE $NODE_PXE/$BASE_KERNEL_FILE # Copy the NFS aware initrd to our TFTP root printf "Copying NFS root aware initramfs to TFTP server..."; progress printf "nfs-initrd,$NFS_INITRD_FILE\n" >> $CURRENT_PXE_FILES cp $NFS_BUILD_DIR/boot/$BASE_INITRD_FILE $NODE_PXE/$NFS_INITRD_FILE # Copy the netboot initrd and kernel into out TFPT root. This can be gathered from # the server CD or online if [ ! -f $NODE_PXE/installer-linux ]; then printf "Collecting the netboot installer kernel and initrd from us.archive.ubuntu.com"; progress wget -nv -O $NODE_PXE/installer-linux http://us.archive.ubuntu.com/ubuntu/dists/$RELEASE/main/installer-$ARCH/current/images/netboot/ubuntu-installer/$ARCH/linux wget -nv -O $NODE_PXE/installer-initrd.gz http://us.archive.ubuntu.com/ubuntu/dists/$RELEASE/main/installer-$ARCH/current/images/netboot/ubuntu-installer/$ARCH/initrd.gz fi printf "Process complete!\n" exit 0
Note(3) The Ubuntu installer is downloaded to support the installation of Ubuntu on disk-full nodes. At this point, we can simple copy the Ubuntu bootstrap installation to our NFS export and update our PXELINUX configuration however, this document is intended to mass produce both disk-full and disk-less systems.
Create a method to discover new nodes
We can either use a menu based PXE install and boot each node one at a time to either a NFS root or to the Ubuntu installation routine, or we can allow the nodes to communicate with the head node and register if they have a local disk installed along with their MAC address. The head node will in-turn maintain a list of known nodes and configure DHCP and DNS to support them along with a NFS root if the nodes do not have local disks. To support the nodes communicating with the head node, we will boot the nodes via PXE to a minimal environment using the kernel and initramfs we initially copied to the root of our TFTP server. The initramfs needs to be modified to provide a method of informing the head node of its hardware details. We can either uncompress the initial initramfs manually, or we can create a script to do it for us. The script to modify the initramfs should:
- Create a temporary working location
- Uncompress the initramfs
- Copy netcat into the initramfs working location
- Modify init so it will:
- Start udev
- Start networking
- Check if a local disk exists
- Communicate with the head node
- Power off
- Compress our modified initramfs and copy it to our TFTP server
Create buildDiscoveryInitrd and make it executable:
touch /opt/cluster/ buildDiscoveryInitrd chmod u+x /opt/cluster/ buildDiscoveryInitrd
The following bash script will add netcat to our initramfs and modify init:
#!/bin/bash # # /opt/cluster/buildDiscoveryInitrd: A scipt to generate a new initrd used for node discovery # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi INITRD_TMP_DIR="/tmp/initrd-tmp" # Create a directory to store the build environment mkdir -p $INITRD_TMP_DIR # Set our working directory to our temp directory for the initrd pushd $INITRD_TMP_DIR # Uncompress the initrd gzip -dc $NODE_PXE/$(grep base-initrd $CURRENT_PXE_FILES | cut -d, -f2) | cpio -id # We will Use netcat as the client to talk to the head node cp $NFS_BUILD_DIR/bin/nc ./bin #We will use fdisk to check if a local disk exists # cp $NFS_BUILD_DIR/sbin/fdisk ./sbin # Modify the init so it will: # 1. Start udev # 2. Start networking # 3. Check if a local disk exists # 3. Communicate with head node # 4. Poweroff before booting to a full system # sed -i '/^load_modules/a /sbin/udevd --daemon\n/sbin/udevadm trigger\n/sbin/udevadm settle\nifconfig eth0 up\n/bin/ipconfig eth0 > /tmp/ipinfo\nmac=\$(cat /tmp/ipinfo \| grep hardware \| cut -d" " -f5)\nif [ -b /dev/sda ] || [ -b /dev/hda ] || [ -b /dev/vda ]; then\n\tboot_method="local"\nelse\n\tboot_method="nfs"\nfi\nprintf "$boot_method,$mac" \| /bin/nc -q2 '"$HEAD_NODE_IP"' 3001\npoweroff' init # Compress the initrd # find . | cpio --quiet --dereference -o -H newc | gzip -9 > $NODE_PXE/discovery-initrd.img popd rm -rf $INITRD_TMP_DIR # Generate the default PXELINUX configuration file printf "PROMPT 0\n" > $DEFAULT_PXE_CONFIG_FILE printf "DEFAULT linux\n\n" >> $DEFAULT_PXE_CONFIG_FILE printf "LABEL linux\n" >> $DEFAULT_PXE_CONFIG_FILE printf "KERNEL nodes/$RELEASE/$ARCH/$(grep base-kernel $CURRENT_PXE_FILES | cut -d, -f2)\n" >> $DEFAULT_PXE_CONFIG_FILE printf "APPEND initrd=nodes/$RELEASE/$ARCH/discovery-initrd.img ip=dhcp\n" >> $DEFAULT_PXE_CONFIG_FILE exit 0
At this point, when an undiscovered node boots via PXE it will attempt to inform the head node what the node's MAC address is and if it has a local hard drive.
Head Node Listens for Connections
We will need netcat on the head node to listen for connections from undiscovered nodes and register them to await Ubuntu deployment. The script to get the MACs from the nodes:
- Determines what the node name will be
- Runs netcat indefinitely waiting for node information
- Registers:
- Node name
- Architecture
- Ubuntu release
- Boot method (local disk or NFS)
- MAC address
- IP address
Create getMACs and make it executable:
touch /opt/cluster/getMACs chmod u+x /opt/cluster/getMACs
If the getMACs script is not running, no new nodes will be registered for deployment.
#!/bin/bash # # /opt/cluster/getMACs: A script to log the MACs from the nodes # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi # Check to see what our first node number will be. # TEMP_NUMBER=$(tail -n1 $MASTER_NODE_LIST | cut -d, -f1 | sed s/$BASE_NODE_NAME//) if [ ! $TEMP_NUMBER = "" ]; then ((NODE_NUMBER=+TEMP_NUMBER)) ((NODE_NUMBER++)) fi # Infinitely run netcat listening for new nodes # Update the node list with: # Generated node name # MAC from the node # Generated IP address # while true; do NODE_INFO=$(netcat -l -p 3001) printf "$BASE_NODE_NAME$NODE_NUMBER,$ARCH,$RELEASE,$NODE_INFO,$(printf $NETWORK | sed s/0$//)$NODE_NUMBER\n" >> $NEW_NODES_LIST ((NODE_NUMBER++)) done
When an undiscovered node is booted via PXE, it will now boot our discovery initramfs and provide the head node with useful information. The head node will register the nodes into a simple text file /opt/cluster/config/new_nodes.txt. You can watch the process of new nodes registering once /opt/cluster/getMACs is executed by switching to a different console or terminal and executing:
tail -f /opt/cluster/config/new_nodes.txt
Process new nodes
We now need to process the registered nodes. We need to create a NFS root file system for each node that does not have a local disk, and we need to prepare to install Ubuntu on nodes that have a disk. We will use a preseed configuration file to install Ubuntu on the nodes that have local disks. The script to process the newly registered nodes should:
- Determine if it will be a NFS node or a local boot node
- Creates the PXELINUX configuration for the nodes
- Updates DNS with the node's host name and IP address
- Reserves an IP address with DHCP for the nodes
- Create the NFS root filesystem for the node (copies our boostrapped installation to our NFS export for each node)
- Moves the node from the new node registration to our master node list
Create makeNodes and make it executable:
touch /opt/cluster/makeNodes chmod u+x /opt/cluster/makeNodes
The following script will parse newly registered nodes:
#!/bin/bash # # /opt/cluster/makeNodes: A script to update/generate configuration for new nodes. # configuration files for the nodes # This script will: # Update DHCP # Update DNS # Create nfs root filesystems if needed # Create PXELINUX configuration files # Update the master node list # Restart DHCP and reload DNS # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi updateDHCP () { case $BOOT_METHOD in local) printf "host $NODE_NAME {\n" >> $DHCPD_CONFIG_FILE printf "\thardware ethernet $MAC_ADDRESS;\n" >> $DHCPD_CONFIG_FILE printf "\tfixed-address $IP_ADDRESS;\n" >> $DHCPD_CONFIG_FILE printf "\toption host-name \"$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE printf "}\n\n" >> $DHCPD_CONFIG_FILE # # Create the PXELINUX configuration for the node # PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/01-$(printf $MAC_ADDRESS | tr : -)" printf "PROMPT 0\n" > $PXE_CONFIG_FILE printf "DEFAULT linux\n\n" >> $PXE_CONFIG_FILE printf "LABEL linux\n" >> $PXE_CONFIG_FILE printf "KERNEL nodes/$RELEASE/$ARCH/installer-linux\n" >> $PXE_CONFIG_FILE printf "APPEND initrd=nodes/$RELEASE/$ARCH/installer-initrd.gz ip=dhcp preseed/url=http://www.home.local/preseed.cfg auto-install/enable debconf/priority=critical locale=en_US console-setup/ask_detect=false console-setup/modelcode=pc105 console-setup/layoutcode=us hw-detect/start_pcmcia=false\n" >> $PXE_CONFIG_FILE ;; nfs) printf "host $NODE_NAME {\n" >> $DHCPD_CONFIG_FILE printf "\thardware ethernet $MAC_ADDRESS;\n" >> $DHCPD_CONFIG_FILE printf "\tfixed-address $IP_ADDRESS;\n" >> $DHCPD_CONFIG_FILE printf "\toption host-name \"$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE printf "\toption root-path \"$NFS_ROOT_EXPORT/$NODE_NAME\";\n" >> $DHCPD_CONFIG_FILE printf "}\n\n" >> $DHCPD_CONFIG_FILE # # Create the PXELINUX configuration for the node # PXE_CONFIG_FILE="$TFTP_ROOT/pxelinux.cfg/01-$(printf $MAC_ADDRESS | tr : -)" printf "PROMPT 0\n" > $PXE_CONFIG_FILE printf "DEFAULT linux\n\n" >> $PXE_CONFIG_FILE printf "LABEL linux\n" >> $PXE_CONFIG_FILE printf "KERNEL nodes/$RELEASE/$ARCH/$(grep base-kernel $CURRENT_PXE_FILES | cut -d, -f2)\n" >> $PXE_CONFIG_FILE printf "APPEND root=/dev/nfs initrd=nodes/$RELEASE/$ARCH/$(grep nfs-initrd $CURRENT_PXE_FILES | cut -d, -f2) ip=dhcp\n" >> $PXE_CONFIG_FILE ;; esac } updateDNS () { printf "$NODE_NAME\t\tIN\tA\t$IP_ADDRESS\n" >> $DNS_FORWARD_CONFIG # reverse zone IP_OCTET=$(printf $IP_ADDRESS | cut -d. -f4) printf "$IP_OCTET\t\tIN\tPTR\t$NODE_NAME.$DOMAIN_NAME.\n" >> $DNS_REVERSE_CONFIG } updateNFSROOT () { # Create the directory filesystem for the node # if [ ! -d $NFS_ROOT_EXPORT/$NODE_NAME ]; then mkdir -p $NFS_ROOT_EXPORT/$NODE_NAME rsync -a --numeric-ids $NFS_BUILD_DIR/ $NFS_ROOT_EXPORT/$NODE_NAME # # Make filesystem node specific # printf "$NODE_NAME\n" > $NFS_ROOT_EXPORT/$NODE_NAME/etc/hostname printf "$IP_ADDRESS\t$NODE_NAME.$DOMAIN_NAME\t$NODE_NAME\n" >> $NFS_ROOT_EXPORT/$NODE_NAME/etc/hosts printf "$NODE_NAME... file-system created\n" else printf "$NODE_NAME... file-system already exists; not created\n" fi } while read LINE ; do NODE_NAME=$(printf $LINE | cut -d, -f1) BOOT_METHOD=$(printf $LINE | cut -d, -f4) MAC_ADDRESS=$(printf $LINE | cut -d, -f5) IP_ADDRESS=$(printf $LINE | cut -d, -f6) updateDHCP printf "$NODE_NAME... DHCP updated\n" updateDNS printf "$NODE_NAME... DNS updated\n" if [ $BOOT_METHOD = "nfs" ]; then printf "$NODE_NAME... creating file-system\n" updateNFSROOT else printf "$NODE_NAME... local boot; NFS root file-system not created\n" fi printf "\n" done < $NEW_NODES_LIST # Append the new node list to the master node list cat $NEW_NODES_LIST >> $MASTER_NODE_LIST #clear the new node list > $NEW_NODES_LIST # Restart the DHCP server /etc/init.d/dhcp3-server restart # Reread DNS zones with RNDC /usr/sbin/rndc reload # Update the hosts.txt file for parallel-ssh cat $MASTER_NODE_LIST | cut -d, -f1 > $PSSH_HOST_FILE exit 0
When a discovered disk-less system is now booted via PXE, the node will boot the NFS aware initramfs and mount their root directory over NFS. The nodes with local disks will boot the Ubuntu installer kernel and initramfs and will attempt to download a preseed configuration file from the head node. Using the preseed configuration file will allow us to have a no questions asked installation of Ubuntu. The installer will attempt to locate the preseed file from the root document directory of our Apache server. Create /var/www/preseed.cfg
touch /var/www/preseed.cfg
The following is the pressed.cfg file I am using to perform a hands-off installation of Ubuntu:
d-i debian-installer/locale string en_US d-i console-setup/ask_detect boolean false d-i console-setup/modelcode string pc105 d-i console-setup/layoutcode string us d-i netcfg/choose_interface select eth0 d-i netcfg/get_hostname string unassigned-hostname d-i netcfg/get_domain string unassigned-domain d-i mirror/country string manual d-i mirror/http/hostname string mirror.home.local d-i mirror/http/directory string /ubuntu d-i mirror/http/proxy http://proxy.home.local:3128/ d-i mirror/suite string jaunty d-i mirror/udeb/suite string jaunty d-i mirror/udeb/components multiselect main, restricted d-i clock-setup/utc boolean false d-i time/zone string US/Eastern d-i clock-setup/ntp boolean false d-i partman-auto/method string lvm d-i partman-lvm/device_remove_lvm boolean true d-i partman-lvm/confirm boolean true d-i partman-auto/choose_recipe select atomic d-i partman/confirm_write_new_label boolean true d-i partman/choose_partition select finish d-i partman/confirm boolean true d-i base-installer/kernel/image string linux-server d-i passwd/root-login boolean true d-i passwd/make-user boolean true d-i passwd/root-password-crypted password $1$lQ7iu8aE$9YeFJJsCCVd9hgWD48VG11 d-i passwd/user-fullname string cluster d-i passwd/username string cluster d-i passwd/user-password-crypted password $1$lQ7iu8aE$9YeFJJsCCVd9hgWD48VG11 d-i passwd/user-uid string 1010 d-i apt-setup/restricted boolean true d-i apt-setup/universe boolean true d-i apt-setup/services-select multiselect security d-i apt-setup/security_host string mirror.home.local d-i apt-setup/security_path string /ubuntu tasksel tasksel/first multiselect ubuntu-server d-i pkgsel/include string openssh-server nfs-common wget ntp vim d-i pkgsel/language-packs multiselect en d-i pkgsel/update-policy select none popularity-contest popularity-contest/participate boolean false d-i grub-installer/only_debian boolean true d-i grub-installer/with_other_os boolean true d-i finish-install/reboot_in_progress note d-i cdrom-detect/eject boolean false xserver-xorg xserver-xorg/autodetect_monitor boolean true xserver-xorg xserver-xorg/config/monitor/selection-method \ select medium xserver-xorg xserver-xorg/config/monitor/mode-list \ select 1024x768 @ 60 Hz d-i preseed/late_command string wget -q -O - http://headnode.home.local/preseed_late_command.sh | chroot /target /bin/bash
I have removed the comments from the above preseed.cfg file to make the document shorter. A commented example for intrepid can be located at: https://help.ubuntu.com/9.04/installation-guide/example-preseed.txt The encrypted password was generated with mkpasswd. mkpasswd is part of the whois package. The format of the preseed file changes as each new Ubuntu release. Be sure to use an example for your specific version. When our discovered nodes boot via PXE, they will initiate the Ubuntu installer and automatically have their operating system installed. Our preseed example includes the statement:
d-i preseed/late_command string wget -q -O – http://headnode.home.local/preseed_late_command.sh | chroot /target /bin/bash
This will cause the node to download the file preseed_late_command.sh from our head node using wget. Using the preseed/late_command in our preseed will allow us to make any post-installation tasks to the newly installed operating system prior to its reboot. Our preseed late command will:
- Add our NFS mounted home directory to /etc/fstab
- Modify /etc/ntp.conf to use our head node as the time server
- Update /etc/apt/sources.list
- Modify the symlink for /bin/sh
The preseed/late command allows us to make changes as if we were logged in to the system. Example preseed_late_command.sh:
#/bin/bash # /var/www/preseed_late_command.sh: A script to make configuration changes at the end of installation # Add our NFS mount to fstab printf "nfs.home.local:/srv/cluster\t/home/cluster\tnfs\trw,auto\t0 0\n" >> /etc/fstab # change /etc/hosts file to remove 127.0.1.1 and use the IP address of the node NODE_NAME=$(hostname) NODE_IP=$(host $NODE_NAME | cut -d" " -f4) sed -i s/127.0.1.1/$NODE_IP/ /etc/hosts # change /etc/ntp.conf to use our NTP server sed -i s/ntp.ubuntu.com/ntp.home.local/ /etc/ntp.conf sed -i '/ntp.home.local/a restrict ntp.home.local mask 255.255.255.255 nomodify notrap noquery' /etc/ntp.conf # Clean up apt sources # /etc/apt/sources.list sed -i /deb-src/d /etc/apt/sources.list sed -i /#/d /etc/apt/sources.list sed -i /^$/d /etc/apt/sources.list # Ubuntu symlinks /bin/sh to /bin/dash. Change it to /bin/bash ln -sf /bin/bash /bin/sh exit 0
SSH
By this time, our nodes are running their new Ubuntu operating system. In order to communicate with the nodes using SSH, we must first gather the host key for each node, and then copy our SSH public key to each node. This can be accomplished manually, however, would be very time intensive on multiple nodes. We will script this process using both BASH and Expect but first we should generate our SSH keys for our root user and our cluster user. We will not use a password to allow for password-less SSH.
Generate Head Node SSH keys
ssh-keygen -t dsa Enter file in which to save the key (/root/.ssh/id_dsa): <press enter> Enter passphrase (empty for no passphrase): <press enter> Enter same passphrase again: <press enter>
Update Node SSH keys
The following two scripts will gather the SSH host key from each node, and then copy the known hosts along with our public and private keys to every node. A common set of keys will allow us to SSH between nodes without having to copy public keys from node A to every node, then node B to every node, etc. Although the script will prompt us for a password, please consider this insecure as the password can be viewed with ps aux while the script is running. Since the home directory for the cluster user is shared among all the nodes, they will share the same ssh keys. We will however, need to update the cluster users known_hosts file. A similar bash script can be used to accomplish this. The bash script will execute the expect script.
#!/bin/bash # # /opt/cluster/copyKeys: A script to generate the SSH known_hosts file # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi printf "We need a password to communicate with disk-ful systems:\n" read -s -p "Password: " SSH_PASSWORD # Iterate the node list # while read LINE; do NODE_NAME=$(printf $LINE | cut -d, -f1) # If we can ping the host, attempt to ssh to it just to get its host key if /bin/ping -c2 -w2 $NODE_NAME > /dev/null; then printf "Ping of $NODE_NAME succeeded\n" ssh -n -o "StrictHostKeyChecking no" -o "BatchMode yes" $NODE_NAME date > /dev/null else printf "Unable to ping $NODE_NAME\n" printf "$NODE_NAME not added to ssh known hosts\n" fi done < $MASTER_NODE_LIST # The local copy of ssh known_hosts should now contain host keys for all nodes # Loop through the nodes once more and copy our .ssh to to each node - This ensure all nodes # have the same keys while read LINE; do NODE_NAME=$(printf $LINE | cut -d, -f1) if /bin/ping -c2 -w2 $NODE_NAME > /dev/null; then printf "Ping of $NODE_NAME succeeded\n" /usr/bin/expect copyKeys.exp $NODE_NAME $SSH_PASSWORD else printf "Unable to ping $NODE_NAME\n" printf "SSH keys not copied to $NODE_NAME\n" fi done < $MASTER_NODE_LIST exit 0
#!/usr/bin/expect -f # # /opt/cluster/copykeys.exp: A script to automate the password on SSH # so we can copy the keys for passwordless logins set force_conservative 0 ;# set to 1 to force conservative mode even if ;# script wasn't run conservatively originally if {$force_conservative} { set send_slow {1 .1} proc send {ignore arg} { sleep .1 exp_send -s -- $arg } } set NODE_NAME "[lindex $argv 0]" set SSH_PASSWORD "[lindex $argv 1]" set timeout -1 match_max 10000 set timeout 100 spawn /bin/bash -c "scp -r /root/.ssh $NODE_NAME:/root/" expect { -re "password: " {send "$SSH_PASSWORD\r";exp_continue} } exit
Parallel-SSH
Attempting to execute commands via SSH to each node may be a bit cumbersome. We can of course create a script to loop through all the nodes listed in nodes.txt. Parallel-ssh is an alternative method to accomplish this. Parallel-ssh allows execution of commands on the nodes at the same time (i.e. in parallel). The pssh package provides:
- Parallel ssh
- Parallel scp
- Parallel rsync
- Parallel nuke
- Parallel slurp
The parallel-ssh suite of tools requires the list of hosts to be in a format different than master node list (nodes.txt). Create the host file that contains only the host names:
pushd /opt/cluster/config cut -d, -f1 nodes.txt > hosts.txt popd
We of course, can create a script that will parse the master node list and output the host list in a pssh friendly format. The following script will do just that:
#!/bin/bash # # /opt/cluster/makePSSH: A script to update the hosts file for parallel-ssh # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi cut -d, -f1 $MASTER_NODE_LIST > $HEAD_NODE_CONFIG_DIR/hosts.txt exit 0
When we installed the parallel-ssh suite of tools, we also created aliases to shorten their names for both the root and cluster users. Instead of executing parallel-ssh -h /opt/cluster/config/hosts.txt command, we can simply type pssh command. Example output from a pssh session:
# Example uptime from all the nodes sudo pssh -i -h hosts.txt uptime [1] 00:48:32 [SUCCESS] node81 00:48:32 up 4 min, 0 users, load average: 0.05, 0.03, 0.00 [2] 00:48:33 [SUCCESS] node71 00:48:32 up 7 min, 0 users, load average: 0.00, 0.02, 0.01 [3] 00:48:33 [SUCCESS] node75 00:48:32 up 0 min, 0 users, load average: 0.53, 0.17, 0.06 [4] 00:48:33 [SUCCESS] node73 00:48:32 up 6 min, 0 users, load average: 0.00, 0.03, 0.01 [5] 00:48:33 [SUCCESS] node77 00:48:32 up 4 min, 0 users, load average: 0.13, 0.08, 0.03 [6] 00:48:33 [SUCCESS] node72 00:48:32 up 7 min, 0 users, load average: 0.00, 0.02, 0.00 [7] 00:48:33 [SUCCESS] node79 00:48:32 up 3 min, 0 users, load average: 0.04, 0.14, 0.07 [8] 00:48:33 [SUCCESS] node74 00:48:32 up 6 min, 0 users, load average: 0.00, 0.04, 0.02 [9] 00:48:33 [SUCCESS] node80 00:48:32 up 3 min, 0 users, load average: 0.03, 0.07, 0.03 [10] 00:48:33 [SUCCESS] node76 00:48:32 up 5 min, 0 users, load average: 0.08, 0.11, 0.07 [11] 00:48:33 [SUCCESS] node78 00:48:32 up 4 min, 0 users, load average: 0.08, 0.14, 0.08 # Example installation of a package sudo pssh -i -h hosts.txt apt-get -y -qq -s install ethtool [1] 00:53:15 [SUCCESS] node81 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [2] 00:53:20 [SUCCESS] node71 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [3] 00:53:22 [SUCCESS] node72 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [4] 00:53:22 [SUCCESS] node75 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [5] 00:53:22 [SUCCESS] node76 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [6] 00:53:22 [SUCCESS] node74 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [7] 00:53:22 [SUCCESS] node73 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [8] 00:53:22 [SUCCESS] node80 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [9] 00:53:22 [SUCCESS] node78 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [10] 00:53:22 [SUCCESS] node79 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty) [11] 00:53:22 [SUCCESS] node77 Inst ethtool (6+20080227-1 Ubuntu:8.10/jaunty) Conf ethtool (6+20080227-1 Ubuntu:8.10/jaunty)
Node Maintenance
Node Replacement
If you experience a hardware failure on a node, the configuration files pertaining to that node will need to be updated with the MAC of the replacement node. We can manually update the configuration files, or we can have the head node update the files for us. The following script will parse the configuration files and update the failed node MAC with that of the replacement node:
#!/bin/bash # # /opt/cluster/replaceNode: A script to replace a node in case of failure. # will prompt for node name to replace and the MAC of the new node # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi printf "Enter name of node to replace:\n" read -p "(e.g. node71): " NODE_TO_REPLACE printf "Enter MAC address of new system:\n" read -p "(e.g. 00:aa:01:bb:02:cc): " REPLACEMENT_NODE_MAC # Convert the MAC address to lower case in case CAPS lock was on NEW_MAC=$(printf $REPLACEMENT_NODE_MAC | tr '[[UbuntuHelp:upper|]]' '[[UbuntuHelp:lower|]]') LINE=$(grep $NODE_TO_REPLACE $MASTER_NODE_LIST) if [ ! -z $LINE ]; then NODE_NAME=$(printf $LINE | cut -d, -f1) BOOT_METHOD=$(printf $LINE | cut -d, -f4) MAC_ADDRESS=$(printf $LINE | cut -d, -f5) IP_ADDRESS=$(printf $LINE | cut -d, -f6) case $BOOT_METHOD in local) # Update the node list to reflect the new MAC sed -i s/$MAC_ADDRESS/$NEW_MAC/ $MASTER_NODE_LIST # Update DHCP to reflect the new MAC sed -i s/$MAC_ADDRESS/$NEW_MAC/ $DHCPD_CONFIG_FILE # UPDATE PXELINUX to reflect the new MAC mv $TFTP_ROOT/pxelinux.cfg/$(printf 01-$MAC_ADDRESS | tr : -) $TFTP_ROOT/pxelinux.cfg/$(printf 01-$NEW_MAC | tr : - ) # Update the node NFSROOT udev configuration to reflect the new MAC printf "$NODE_NAME MAC changed from $MAC_ADDRESS to $NEW_MAC\n" # Changes made. Restart DHCP server /etc/init.d/dhcp3-server restart ;; nfs) # Update the node list to reflect the new MAC sed -i s/$MAC_ADDRESS/$NEW_MAC/ $MASTER_NODE_LIST # Update DHCP to reflect the new MAC sed -i s/$MAC_ADDRESS/$NEW_MAC/ $DHCPD_CONFIG_FILE # UPDATE PXELINUX to reflect the new MAC mv $TFTP_ROOT/pxelinux.cfg/$(printf 01-$MAC_ADDRESS | tr : -) $TFTP_ROOT/pxelinux.cfg/$(printf 01-$NEW_MAC | tr : - ) # Update the node NFSROOT udev configuration to reflect the new MAC sed -i s/$MAC_ADDRESS/$NEW_MAC/ $NFS_ROOT_EXPORT/$NODE_NAME/etc/udev/rules.d/70-persistent-net.rules printf "$NODE_NAME MAC changed from $MAC_ADDRESS to $NEW_MAC\n" # Changes made. Restart DHCP server /etc/init.d/dhcp3-server restart ;; esac else printf "The node name you entered is invalid\n" exit 192 fi exit 0
Power on with wake-on-LAN
You may find it convenient to power on your nodes using wake-on-LAN. Wake-On-LAN requires the MAC of the node you wish to power on to be known. The nodes.txt file containts the known MACs of installed nodes. Using Wake-on-LAN, we should be able to:
- Turn on a single node by name
- Turn on multiple nodes by name
- Turn on all nodes
The following script will parse nodes.txt and send a wake-On-LAN packed to the nodes you specify:
#!/bin/bash # # /opt/cluster/nodeOn: A scipt to power on nodes # Version 0.1 # Author: geekshlby CONFIG_DIR="/opt/cluster/config" if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi if [ -f $CONFIG_DIR/global.conf ]; then source $CONFIG_DIR/global.conf else printf "Unable to locate the global configuration file.\n" printf "This script looks for configuration files in:\n" printf "$CONFIG_DIR\n" exit 192 fi if [ ! $(whoami) = "root" ]; then printf "This script must run with root access.\n" exit 192 fi helpMe () { printf "usage: $0 [NODE] \n" printf "[NODE]\n" printf "\tall\tpower on all nodes\n" printf "\tnode#\tpower on a single node\n" printf "\t\tMultiple nodes names should be separated by white space.\n" printf "\t\t\te.g. node_on.sh node1 node2\n" } if [ -z $1 ]; then helpMe exit 0 fi case $1 in all) while read LINE; do NODE_NAME=$(printf $LINE | cut -d, -f1) MAC_ADDRESS=$(printf $LINE | cut -d, -f5) printf "Sending Wake-On-LAN packet to $NODE_NAME at $MAC_ADDRESS\n" # Send more than one WOL packet to the node I=0; until [ $I -gt 5 ]; do wakeonlan $MAC_ADDRESS >/dev/null ((I++)); done done < $MASTER_NODE_LIST ;; -h|--help) helpMe ;; *) until [ -z $1 ]; do NODE_NAME=$1 MAC_ADDRESS=$(grep -w $NODE_NAME $MASTER_NODE_LIST | cut -d, -f5) if [ ! -z $MAC_ADDRESS ]; then printf "Sending Wake-On-LAN packet to $NODE_NAME at $MAC_ADDRESS\n" # Send more than one WOL packet to the node I=0; until [ $I -gt 5 ]; do wakeonlan $MAC_ADDRESS >/dev/null ((I++)); done else printf "$NODE_NAME does not have a MAC listed in nodes.txt\n" fi shift done ;; esac exit 0
To turn on node1.home.local:
sh /opt/cluster/nodeOn node1
To turn on multiple nodes:
sh /opt/cluster/nodeOn node1 node23 node43
To turn on all nodes:
sh /opt/cluster/nodeOn all
Summary
This article has descirbed a method to deploy Ubuntu onto both disk-full and disk-less nodes in an automated manner. It is not intended to be the one-for-everyone solution. The possibilities for improvement are endless, and I hope this article has spawned an “oh I can do that” attitude within its readers. My own improvement ideas are:
- Use a database backend
- Use a multiple headnode model
- Daemonize some of the processes
- getMacs
- makeNodes
- makePSSH
An addition to the above method to deploy Ubuntu, could be a method to monitor the status of the nodes. This can be done with bash of course. Since part of creating the head node was installing the LAMP stack, the following PHP code will parse the master node list and determine if the nodes are online. This too can be improved by using SNMP to communicate status information. Currently, the following ping.php simply pings the nodes to determine if they are online, and displays a web page with their status:
<html><head><title>System Status</title></head> <body> <pre> <?php function ping($host) { exec(sprintf('ping -c 1 -w1 -W1 %s', escapeshellarg($host)), $res, $rval); return $rval === 0; } $green = "/images/status/green.png"; $red = "/images/status/red.png"; $counter = 1; $hostfile = "status.conf"; $nodefile = "/opt/cluster/config/nodes.txt"; $nodefile_handle = fopen($nodefile, "r"); $counter = 1; echo "Nodes:<br>"; echo "<table border=0 width=80%>"; echo "<tbody>"; echo "<tr>"; while (!feof($nodefile_handle)) { $line_of_text = fgets($nodefile_handle); if (!feof($nodefile_handle)): $nodeinfo = explode(',', $line_of_text); $up = ping($nodeinfo[0]); if ($counter > 2): echo "<tr>"; endif; if ($up == 1): echo "<td><img border=0 height=49 width=58 alt=Up title=Up src=$green></td><td>$nodeinfo[0] <br> $nodeinfo[1]</td>"; $counter = $counter + 1; else: echo "<td><img border=0 height=49 width=58 alt=Down title=Down src=$red></td><td>$nodeinfo[0].home.local<br>Arch:$nodeinfo[1] Release:$nodeinfo[2]<br>Boot method:$nodeinfo[3]<br>MAC address:$nodeinfo[4]<br>IP address:$nodeinfo[5]</td>"; $counter = $counter + 1; endif; if ($counter > 2): echo "</tr>"; $counter = 1; endif; endif; } fclose($nodefile_handle); echo "</tbody>"; echo "</table>"; ?> </body> </html>
The output of the above page will list nodes in 2 columns.
{{https://help.ubuntu.com/community/AutomatedNodeDeployment?action=AttachFile&do=get&target=green.png%7D%7D%7C%7Cnode71.home.local < >Arch:amd64 Release:intrepid < >Boot method:local < >MAC address:08:00:27:0c:09:27 < >IP address:10.10.1.71 |
{{https://help.ubuntu.com/community/AutomatedNodeDeployment?action=AttachFile&do=get&target=red.png%7D%7D%7C%7Cnode72.home.local < >Arch:amd64 Release:intrepid < >Boot method:nfs < >MAC address:08:00:27:3f:4e:2c < >IP address:10.10.1.72 |