The most important change for me is in the maintenance of the distribution.
In the past I have, in a largely manual process, maintained the lineup of binaries from a list mostly driven by what I need for my applications of the CD. This has been a lot of work, and had become very time-consuming.
I have now enlisted the package manager to obtain the dependencies of files that belong to a given package. In order to keep the size under control, I strip out most of the man pages, examples, and other anxiliary files (usually the files installed into /usr/share).
In this way, I now get a more comprehensive list of files by identifying the top-level packages to include in a more automated fashion. As a result, the size of the project has gone up slightly; I have incresed the ramdisk sizes to account for this.
Along the way, I have consolidated the 64bit and 32bit versions, which I had allowed to drift apart slightly in the last release.
I could need some feedback about the utility of maintaining both 32bit and 64bit versions here. It doubles the workload, and I myself cannot even remember when I used the 32bit version other that to test that it works.
If having the 32bit is critical for your your application, please let me know. This production release still has both versions, but I'm thinking about dropping 32bit support in the distant future.
A user requested a number of "add-ons" which are too rarely used to merit space on the distribution for all. I built sshguard, ntp, busybox, and open-vm-tools. I add the links to them below if you ever need those. I made them only for the 64bit version (but you know how ot contact me for the other version if you are desperate).
Over time, I have upgraded this system in a number of ways.
You will need some way of booting them first to make repairs or download the OS. This is where a boot CD comes in handy - it allows you to boot your system, independent from anything on the disks, and you will be able to make repairs or get your system into a bootable state.
This project has a slightly different purpose. It is meant to be a customizable Swiss army knife that allows you to get done what you need, quickly. This setup enables you to quickly produce your own custom boot CD tailored to your needs. The bare-bones CD typically boots in less than 30 seconds on modern hardware. (I'm always referring to a CD, but the same goes for a USB boot stick, or a PXE (network) boot setup.)
Although I'll provide you with a ready-made CD image that you can burn and use as my personal version of a rescue CD, the real product here is a script and a template setup which you can easily customize to produce your own toolset or application. The script automates the whole process of producing the various images and comes up with the CD ISO image which you can burn.
For example, you could use an older PC that's just sitting there as a disk-less firewall using the CD, or (that's my main use) use it to automate the installation of large Linux clusters. Or you can just keep the CD handy for the day when you need to repair a machine. Or you can produce your own installation CD (probably a DVD). Whatever you need, you can roll your own.
You will see that there is no X window system here - you get a few virtual consoles. Good enough for what we have in mind here. However, if you feel you can't do without X... well, you can put it on.
Note: I provide the ready-made boot iso file mostly as a reference to help you find problems in case you produce a CD which doesn't work as expected. It is fully functional, but if you'd ever want to use a ssh server on that Rescue system, you can't do that out of the box, because the template setup does not ship with ssh host keys, and the ssh server is configured to not honor passwords, only ssh keys (and obviously, your ssh key isn't on it). Before you make a production setup, read the "Using ssh" chapter below.
mkisofs -b isolinux/isolinux.bin -c isolinux/boot.cat \ -o bootcd.iso \ -no-emul-boot -boot-load-size 4 -boot-info-table \ -J -r -T \ -p "firstname.lastname@example.org" \ -A "Rescue Disk" \ cd_top_areaand you have a CD that boots. In version 1 of this project, before isolinux, I had to make a bootable floppy image, which was much more complicated.
Maybe this is a good moment to download and unpack the template to a fresh directory, so you can look at what I'm talking about here. You will find a number of scripts, a Makefile, and (next to a few others) the directories "cdtree", "root_tree32", and "root_tree64". "cdtree" is the area where the ready-made later CD contents will be and where the ISO image will be made from. "root_tree32" and "root_tree64" hold the contents of the later root file systems of the OS, one for a 32bit installation, one for 64bit. Most of the changes and customizations you might want to make will take place in the root_treeXX areas.
Again, quickly the sequence:
If this is confusing, don't worry. Just remember that your normal Linux installation usually has the root file system already sitting on some physical disk, such as /dev/hda1 or /dev/sda5 or something like that. You just need to mount that disk and are ready to go. Here we don't have such a ready-made disk, but we have to prepare it on the fly first. That's what's happening with the initial ramdisk mechanism.
You will have realized that the whole OS that we run after the boot is entirely ramdisk-based. It does not use or touch any physical disk, and even the CD is no longer needed, unless you need it for some special purpose. The whole OS can now function without the CD.
Let's get ourselves a working setup first.
Get yourself a fresh directory, and untar the template there. You need
to define this topmost directory as TOPDIR. For example:
tar xfj /location/of/the/thing/rescuecd_4.16.tar.bz2
I typically have different boot CD projects sitting in various places. Just define TOPDIR to point to the right place you are working with.
All default settings and definitions are kept in a file $TOPDIR/settings.sh. While the default settings are reasonable (I think), this allows you to adapt the project to the environment of your system. There is no need to change any of them if the default works for you.
The build process can take a few minutes. The Makefile takes care that only those components which are truly changed are rebuilt.
The Makefile and assorted scripts assume that they can use (and your system supports) loopback devices. By default, it wants to use the device /dev/loop2 (I found that loop0 and loop1 are occasionally taken by some system processes). If you can't use loop2 for any reason (this is rare), you can change the loop device in the settings.sh file.
Why don't you just type "make" at this point -- that should reproduce the standard bootcd.iso image without any problems.
Those areas are completely independent at this point. Modifications made to, say, the 32 bit system are not reflected in the 64bit system automatically.
You should probably make the same modifications to both areas in order to avoid confusion.
As my personal choice, I like to steer the general behavior of the system by choosing a run level, just like you can boot your standard installations in different run levels (for example single user, no network, just console logins, X windows, etc). My template will show you how to expand on this.
I find that most of the time, all my customizations are in modifying or adding startup scripts, or, on rare occasions, adding binaries. I rarely have to go beyond that (which you can, easily enough, by the way, that's the next chapter). The choice of what scripts run is taken by the run level.
Let's take a look at the file $TOPDIR/cdtree/isolinux/isolinux.cfg, which controls the boot process.
prompt 1 timeout 100 display boot.msg default r2-64-nofb label r1 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 1 vga=791 label r2 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 2 vga=791 label r3 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 3 vga=791 label r1-nofb kernel vm32 append initrd=sdisk32.img root=/dev/ram0 1 label r2-nofb kernel vm32 append initrd=sdisk32.img root=/dev/ram0 2 label r3-nofb kernel vm32 append initrd=sdisk32.img root=/dev/ram0 3 label r1-64 kernel vm64 append initrd=sdisk64.img root=/dev/ram0 1 vga=791 label r2-64 kernel vm64 append initrd=sdisk64.img root=/dev/ram0 2 vga=791 label r3-64 kernel vm64 append initrd=sdisk64.img root=/dev/ram0 3 vga=791 label r1-64-nofb kernel vm64 append initrd=sdisk64.img root=/dev/ram0 1 label r2-64-nofb kernel vm64 append initrd=sdisk64.img root=/dev/ram0 2 label r3-64-nofb kernel vm64 append initrd=sdisk64.img root=/dev/ram0 3
This is set up to give you 3 run levels, in either the 32- or 64bit version, and with a choice of using the framebuffer or not (looks nicer on a console, but there are consoles where this can appear garbled). Also, scrolling under a framebuffer is much slower than without it, so you if just want to look at something really quick, you might be better off without it.
So "r2" boots into run level 2 using the 32bit system and the framebuffer; "r2-nofb" is level2, 32bit, and no framebuffer, r2-64 is level 2 in 64bit, and r2-64-nofb is level2, 64bit, no framebuffer. Of course you can modify the labels if you prefer a different naming scheme.
Isolinux will print the contents of $TOPDIR/cdtree/isolinux/boot.msg to the screen, wait 10 seconds, and then boot by default in run level 2. The text in boot.msg, which you should adapt if you make changes, reads (I have here removed some escape characters that bring out some keywords in color on the console):
Welcome to Martin's Rescue CD You can read about it at http://www.phenix.bnl.gov/~purschke/RescueCD/ Select r
-xx-xx for kernel options ( -64 for 64 bit, -nofb for no framebuffer) e.g. r2 for 32bit, r2-64 for 64bit, r2-nofb, r2-64-nofb (default r2-64-nofb) - Run Level 1 goes straight into a single-user shell -- type r1* - Run Level 2 normal login shells -- type r2* - Run Level 3 normal login shells AND we get on the network -- type r3* So what do you want?
So you get the idea what run levels 1,2,3 are meant to do.
Once the system has booted, the behavior is controlled by its /etc/inittab (remember, this is $TOPDIR/root_treeXX/etc/inittab as we look at it). The essential lines are:
id:2:initdefault: # System initialization. si::sysinit:/etc/sysinit #shutdown l0:0:wait:/etc/shutdown shutdown # normal run levels l1:1:wait:/bin/bash -i -l l2:2:wait:/etc/rc.d/rc 2 l3:3:wait:/etc/rc.d/rc 3 # reboot l6:6:wait:/etc/shutdown reboot # Trap CTRL-ALT-DELETE ca::ctrlaltdel:/etc/shutdown reboot # Run gettys in standard runlevels c1:2345:respawn:/sbin/agetty 38400 tty1 linux c2:2345:respawn:/sbin/agetty 38400 tty2 linux c3:2345:respawn:/sbin/agetty 38400 tty3 linux r2:2:wait:/etc/rc.d/rcend 2 r3:3:wait:/etc/rc.d/rcend 3
#! /bin/sh argv1="$1" if [ $argv1 -eq 1 ] ; then exec init -t1 S fi [ $argv1 -gt 2 ] && /etc/start_network.sh
The handling of run level 1 is for good measure only, since this script is never called with parameter 1; the inittab file specifies this differently. But you see, it does nothing really for run level 2, but remember run level 3?
- Run Level 3 normal login shells AND we get on the network -- type r3*
You can use the run levels for whatever purpose you need, and there is, to my knowledge, no practical limit to the number of such run levels.
You could, for example, add level 4 and make "r4" the default boot level, and say in boot.msg:
- Run Level 4 like run level 3, AND we start the firewall setup
#! /bin/sh argv1="$1" if [ $argv1 -eq 1 ] ; then exec init -t1 S fi [ $argv1 -gt 2 ] && /etc/start_network.sh [ $argv1 -gt 3 ] && /etc/start_firewall.sh
Keep in mind, though, that the rc script is executed before we get to the handling of the logins. You should not put anything in there that takes a long time to complete, or even runs indefinitely. You will probably want the ability to login to the system when something long is going on. That's the reason that after the login handling of the inittab file, we execute yet another script, /etc/rc.d/rcend with the same run level parameter, where we can put things that will take a long time. For example, on my customized install disk, I put the start of the script that will automatically install the new OS on the disks into the rcend script. That will run for some time, 20, 30 minutes, but I still want to be able to login to check on things.
Also keep in mind that you have a full inittab file, so if you need a process that needs to run all the time and needs to get restarted if it ends or gets killed, you can add it to the inittab file. The /sbin/agetty processes you can see above are examples of this.
prompt 1 timeout 100 display boot.msg default r5 label r1 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 1 vga=791 label r2 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 2 vga=791 label r3 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 3 vga=791 label r4 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 4 vga=791 label r5 kernel vm32 append initrd=sdisk32.img root=/dev/ram0 5 vga=791 label r1-nofb kernel vm32 append initrd=sdisk32.img root=/dev/ram0 1 ... .... lines deleted ...
And the boot.msg file:
Welcome to Martin's Rescue CD
You can read about it at
- Run Level 1 goes straight into a
single-user shell -- type r1
- Run Level 2 normal login shells
-- type r2
- Run Level 3 normal login shells AND
we get on the network -- type r3
- Run Level 4 like 3, AND we mount the
distribution area -- type r4
- Run Level 5 like 4, AND automatically
start the installation -- r5 (DEFAULT)
So what do you want?
I added the new run levels 4 and 5 to /etc/inittab:
In the /etc/rc.d/rcend script, I added:
grep -q /cdrom /etc/mtab && eject /cdrom
[ $1 -gt 3 ] && /etc/mount_distribution.sh
[ $1 -gt 4 ] && /etc/just_do_it.sh
This "just do it" script will go ahead and install the OS. I use this
fully unattended install for racks of machines where I don't want to hook
up a console first.
Do you remember that I said that we do no longer need the CD after the boot? Here in my "rcend" script, I eject the CD before the begin of the installation. This is my cue that I can power on the next machine and move the CD there, starting the next install. In this way, I can move through a typical rack with 32 machines in less than half an hour. The machines take much longer to install, but I minimize the time I need to spend in the air-conditioned computer room.
I should mention at this point that the CD boots fine from most USB-connected CD-ROMS. I noticed that some Live-CDs don't. Most of my rack-mounted machines don't have internal CD-ROMs any longer, and in any case, moving a USB cable to the next machine is faster than moving the actual CD from one drive to the next.
At some point you will find that you need some application or binary that is not on the CD, or you will want to update binaries with a newer or different version from your system. That in itself is not a big deal, but the binaries need shared libraries and typically a number of configuration files, which you will have to find and put on the CD as well.
Also keep in mind that your host machine has either a 32- or 64bit system. Binaries present on your system can only go into either root_tree32 or root_tree64 - make sure that you do not mix up the binaries.
Find out what shared libraries the binary you intend to add or replace needs. On your system, the ldd command will tell you. See if those libraries with the same name are present in the respective root_treeXX/lib/ area.
Here is an example that shows what shared libraries the "ls" program needs:
% ldd /bin/ls
librt.so.1 => /lib64/librt.so.1 (0x00007f646a6d7000)
libacl.so.1 => /lib64/libacl.so.1 (0x00007f646a4ce000)
libc.so.6 => /lib64/libc.so.6 (0x00007f646a12f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6469f13000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007f6469d0e000)
(You will find that they are all in the right place).
You should try any additions on a copy of your project, or have a good backup in case this goes wrong.
Another problem is that ldd will not
necessarily find all libraries that we need; some programs have
optional libraries which get loaded dynamically and which will not be
listed by ldd. One important such program is /bin/sh, where ldd lists
a rather small list of shared libraries:
% ldd /bin/sh
libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f39350b0000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3934eac000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3934b0d000)
The fact that it loads libdl.so, the dynamic loader, is a bit of a giveaway;
you should watch out for those binaries. Here is a way, at least for /bin/sh,
to find the additional libraries. In a running shell, use
$ lsof -p $$
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 11324 purschke cwd DIR 8,8 32768 3244033 /home/purschke
sh 11324 purschke rtd DIR 8,1 4096 2 /
sh 11324 purschke txt REG 8,1 856344 65554 /bin/bash
sh 11324 purschke mem REG 8,1 52040 3354 /lib/libnss_files-2.15.so
sh 11324 purschke mem REG 8,1 43472 3332 /lib/libnss_nis-2.15.so
sh 11324 purschke mem REG 8,1 88976 3361 /lib/libnsl-2.15.so
sh 11324 purschke mem REG 8,1 31504 3372 /lib/libnss_compat-2.15.so
sh 11324 purschke mem REG 8,1 1689272 3370 /lib/libc-2.15.so
sh 11324 purschke mem REG 8,1 14616 3362 /lib/libdl-2.15.so
sh 11324 purschke mem REG 8,1 333648 3315 /lib/libncurses.so.5.9
sh 11324 purschke mem REG 8,1 140816 3365 /lib/ld-2.15.so
sh 11324 purschke 0u CHR 136,10 0t0 13 /dev/pts/10
sh 11324 purschke 1u CHR 136,10 0t0 13 /dev/pts/10
sh 11324 purschke 2u CHR 136,10 0t0 13 /dev/pts/10
sh 11324 purschke 255u CHR 136,10 0t0 13 /dev/pts/10
Here you can see that it wants the additional libraries libnss_files-2.15.so, libnss_nis-2.15.so, libnsl-2.15.so, and libnss_compat-2.15.so.
But sooner or later, you may need a different kernel, or some additional driver which is not included, or you need some other functionality, or simply a newer version. I assume that you know your way around configuring and compiling a new kernel. Start with a fresh kernel (if you use your running kernel, be sure to save the config file for your system kernel in a safe place). I included my kernel config files in $TOPDIR/rescue_kernel32.config (and -64). Use this one to initially configure your kernel, then make all the other changes that you need.
Once you have recompiled a new 32bit kernel in /usr/src/linux on your system, copy arch/i386/boot/bzImage to $TOPDIR/cdtree/isolinux/vm32. A new 64-bit kernel goes to $TOPDIR/cdtree/isolinux/vm64.
Then run (for 32bit, replace 64 for 32 else)
INSTALL_MOD_PATH=$TOPDIR/root_tree32 make modules modules_install
You can check the dates of the files in the target directories if you
missed a file.
While we are making new kernels: There is a parameter in the setting.sh file which specifies how large the initial ramdisk file systems are that we build. The current setting of 420MB makes the file system come out to be about 65% full.
If you modify that value in the settings.sh file, you also must build new kernels (both 32 and 64bit) with the initial ramdisk size adjusted. This is the parameter CONFIG_BLK_DEV_RAM_SIZE in the kernel; in the kernel setup, go to "Device Drivers" -> "Block Devices" -> "Default Ramdisk size".
Note that you must do this for both kernels.
If you make a CD that is used for some simple task, such as my CD that installs an OS on a new machine, you could choose to move some of the binaries that I have included in the ramdisk but which are not needed for the standard task over to the cdtree/bin area. In this way you make the ramdisk smaller, and the CD boots faster because there is simply less data to unpack during the preparation of the root file system. On the other hand, you still have all other binaries that you might occasionally need on the CD.
Warning... I assume that you know what you are doing. Remember that you can log into the root account without password, based on the assumption that access to this machine is from its console only. Starting sshd will change that. For this reason, the standard sshd_config files that come with the project will not accept passwords at all, only ssh keys. If you decide to change this and let the ssh daemon accept passwords, you have been warned.
In almost all cases I can imagine, I think it is better if you leave logins with passwords disabled, and add the ssh keys of the users who might need to login via ssh to the root_treeXX/root/.ssh/authorized_keys files, and then use the ssh agent to manage your keys.
Here is a good explanation how this works. You should generate a dedicated key pair for this, and use the ssh-agent to manage the key.
Even more importantly, the project does not come with ssh hosts keys preinstalled, which will prevent you from starting the ssh daemon out of the box. You would need to generate the keys on the fly on the running system, but this is not a good solution, because then they change each time you boot the CD.
I have added a "KEYS" target to the Makefile, which will generate a
set of hosts keys in $TOPDIR/hostkeys/. I recommend that you not only
generate them (and distribute them to the root_treeXX areas as
indicated by the instructions you will get), but that you also
distribute the same keys to all the different RescueCD projects you
might have. In this way you will avoid getting the ominous ssh warning
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
In a running system, starting the ssh daemon is a simple as typing "/sbin/sshd". You could also start it based on a run levels described above.
As I said in the introduction, I have added the ability to produce a bootable USB stick easily, as well as setting up a PXE network boot environment. I carry a bootable stick on my keyring, and it comes in handy at times.
With that said, my main use of this project is to set up a large number of machines quickly. I have a few machines which cannot boot from a stick at all, but for those that can, this typically takes going into the BIOS setup, selecting the boot medium, then boot - this is the opposite of a fast process. With the USB-connected CD-ROM, the BIOS can remain set up to boot off a CD, if any is present, so this is much faster.
And once booted, the USB stick shows up as yet another disk, which, depending on the particular BIOS, may show up as /dev/sda, shifting all other disk names up. Yes, there are ways to not use physical device files through their UUIDs, but it's involved, and it's easier if /dev/sda is the same disk as it will be in the eventual system, in particular if you have software RAID systems set up with mdadm. You don't have this problem with a CD.
But if you want a USB or PXE network boot instead, it's all supported.
And this brings us to...
You must set the boot flag of the designated boot partition (and,
while you are at it, you can change the partition Id to 83). Here is
what one of my sticks looks like:
# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 128 MB, 128974848 bytes
3 heads, 60 sectors/track, 1399 cylinders, total 251904 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x33c708c2
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 251903 124928 83 Linux
In fdisk, you can use the "a" command to toggle the boot flag, and "t" to change
In virtually all cases of a problem booting off a stick that I have
seen, the partition was found not to have the boot flag turned on.
The default behavior, customizable in the $TOPDIR/settings.sh file, is to use partition 1 of the device which you identify with the USBDEV variable. For example, if you set this to
then /dev/sdb1 will be used as the bootable partition. USBDEV needs to point to a device and not a particular partition here because the procedure needs to set up the master boot record of the device.
Be very careful setting this variable. If you mistakenly set this to /dev/sda, and this happens to be your system boot disk, you will end up with a non-booting system. Review this setting once more before you issue the following commands.
You need to format the stick once (most of the time, they are already partioned properly, so we only need to format). In order to avoid disasters, the build system will not format the USB stick for you. You need to do that yourself.
Assuming you are using /dev/sdb1 as in the example above:
mkfs -t ext2 /dev/sdb1
This needs to be done only once. Out of the box, most USB sticks come with a VFAT (Windows) file system. We need an ext2 file system here.
and you will end up with a bootable stick.
Note1: If you already made your stick bootable with "make USB" before, and only wish to update the installation with some changes you made, you can issue
This only updates the ramdisk images, kernels, etc, but will not re-do the steps to make the stick bootable.
Note 2: Building a bootable stick will make temporary changes to both the root_tree32 and root_tree64 areas (which are getting undone after the build). The areas will appear modified to the Makefile, and a full rebuild will be performed the next time you build a CD.
The changes we make to the root areas are adding a mount point for the USB stick to the /etc/fstab file so you can just say "mount /usbstick" (it is not mounted automatically). In addition, we add a file that allows the sysinit script to tell that we were booted from USB so it will not attempt to mount the CD. It will also change the hostname, and with it the later prompt, from "RescueCD" to "RescueUSB".
A PXE setup allows machines which support it (the BIOS needs to support this, but most modern BIOSes do) to boot from the network.
Basically, the machine to be booted asks for a dynamic IP address from the dhcp server. The dhcp servers supplies one, and offers a boot file if the machine asks for it. The machine starts to boot, and uses tftp to download a configuration file with further instructions, and so bootstraps itself.
This is relatively straightforward to set up if you control both the dhcp and a tftp server on your network. This is very often not the case. If not, this becomes a bit more involved. However, you can turn your laptop or another dedicated machine into a standalone boot server with a bit of effort. I need this because I have a few machines at work where the only way to first-time boot them is through PXE. But even outside this application, I have come to appreciate the versatility of such a setup. Being able to quickly start a localized dhcp server in this way allows another machine, such as a Windows Laptop, to easily connect to yours in order to quickly transfer a few files, just by connecting them directly with an ethernet cable. I have been getting a lot of questions how I set this up, so let me describe my setup in a bit more detail on a separate page.
Since in most cases your tftp server will be different from the machine on which you build the boot images, the "PXE" target (make PXE) will give you a tftp_area.tar.gz tarball, which you can transfer to your server and unpack in your tftp area.