http:// www.jms1.net / xen / 32bit-on-64bit.shtml

Running 32-bit xenU sessions on a 64-bit CentOS xen0

For the past few years I've been running Xen on my server, with several virtual child sessions (one of which is serving this web site.) Until a few days ago, the machine was dual 1GHz Pentium 4's with 3.5GB of RAM. The P4 processor doesn't support hardware virtualization, and the 1GHz version isn't the fastest thing on the planet, but for simple email and web hosting, it was actually doing fairly well as a rarely-used dom0 with five domU's. The only problems I had were hardware-related, and the lack of reliable air conditioning at the old location didn't make things any eaiser.

Last week I built a new machine, and I had planned to conduct an orderly transition of my own domU, plus my friends' and clients' domU's, to the new machine. The new machine is quite a bit faster than the old one- it's a 3 GHz Core 2 Duo with 8GB of RAM. It runs 64-bit code, and supports hardware virtualization. Because it has 8GB of RAM, I installed the 64-bit version of CentOS 5 on it.

Without going into detail, I had to move everything to the new hardware much more quickly, and much more suddenly, than I had originally planned. I don't generally like to do things like this without testing first, but in this case I didn't really have any choice in the matter.

When I started up the first domU (my own) I found that the even though the kernel was x86_64, the 32-bit libraries and programs on the domU still seemed to work normally, so long as I didn't try to update any of the packages. It turned out that yum wanted to update every single package on the system to the 64-bit equivalent. Which at first sounds appealing, but it didn't work- apparently the dependencies are different between the 32- and 64-bit packages. It cycled through about seven minutes(!) of checking dependencies and then failed, saying that various glibc and other library packages were "not available".

I figured the solution was to make the children run 32-bit kernels, if I could figure out how to make that happen without having to re-install the machine's OS (and possibly give up 4GB of RAM in the process.) I did some google searching and found that such a thing was possible, however I couldn't find any pages which explained the specifics of how to set it up. I did eventually figure it out, which is why I'm writing this web page- one, to help others who may need to know, and two, for my own reference in case I ever need to do this again.

Overview

The trick is, as I mentioned above, to make the domU run a 32-bit kernel. This is possible because the domU's kernel is controlled by a text file on the dom0. There are three things you need:

A 32-bit CentOS system, to "borrow" for a few minutes. This can be either a non-virtual system, or if you have another xen system with a 32-bit dom0, you can use the dom0 for that machine. (Basically, the machine it needs to NOT be a domU.)
The 32-bit kernel-xen package for the kernel you wish to run on the domU. You can get this via yum, or you can download it from one of the CentOS mirrors. We need the vmlinuz file, and the /lib/modules directory, from this package.
An initrd file with the appropriate drivers to finish booting a domU running that kernel. This can be built on a machine which has that kernel package installed.

Once you have these three things, it's a simple matter of configuring the control file and the domU filesystem to make it work.

Installing the kernel-xen package

The first step is to install the kernel-xen package containing the 32-bit kernel you wish to run. This needs to be done on a 32-bit machine which is not a domU. I used a CentOS 5 machine which is not involved in any kind of virtualization.

The easiest way to install it is to use yum, like so:

# yum install kernel-xen

If the machine is already a dom0, this will either install the latest xen kernel, or tell you that it's already up to date. If it's not a dom0, this will install several other packages along with the kernel. Keep a list of the other packages which it installs, because in a minute we're going to un-install, and you're going to want to remove those packages as well.

Copying the kernel files

Once this package is installed, figure out how you're going to transfer three files from this system to the one with the 64-bit dom0. I used scp, however these instructions will assume you are using a USB memory stick.

Copy the /boot/vmlinuz-version.el5xen file to the USB stick (or to the new system.) This is the kernel itself.
Create a tarball of the kernel modules, and transfer it to the USB stick. This is done using a command like this:

# cd /lib/modules
# tar czf /root/lib-modules-2.6.18-164.el5xen.tar.gz 2.6.18-164.el5xen

This creates /root/lib/modules-version.tar.gz. Copy this file to the USB stick.
Create an initrd (initial ram disk) to go with the kernel. The process looks like the following. Note that this is one single command, I'm showing it with "\" at the end of each line in order to avoid making the web page too wide for a normal browser. You can type it as three separate lines, exactly as shown here, or you can combine the lines into one long command.

# mkinitrd -v --preload xenblk --with=xennet \
> --omit-scsi-modules --omit-raid-modules --omit-lvm-modules \
> /root/initrd-2.6.18-164.el5xenU.img 2.6.18-164.el5xen
Creating initramfs
Looking for deps of module xenblk
...
...
...
Adding module xennet

This will create the file /root/initrd-versionU.img. Copy this file to the USB stick.

If you've been using a USB stick, un-mount it and remove it from the machine.

If the machine is not a xen server, you can remove the kernel-xen package, along with any other packages it added, using "yum erase" or "rpm -e".

Install the kernel files on the dom0

I probably shouldn't say "install" here, because we're not really installing the files in the same sense that we would install a package- we're simply putting the three files into a specific location on the system, so we know where to find them.

On my own machines, I created directories called /etc/xen/32bit and /etc/xen/64bit which hold the kernel files used by the domU's. This allows me to update the dom0 kernel if needed, without affecting the kernel used by the domU's (although I generally update them at the same time.)

Create the directory, and copy the three files from the USB stick into it. When you're done, it should look like this:

# ls -laF /etc/xen/32bit
total 18260 drwx------ 2 root root 4096 Oct 13 12:06 ./ drwx------ 6 root root 4096 Oct 13 12:39 ../ -rw------- 1 root root 2378975 Oct 13 11:47 initrd-2.6.18-164.el5xenU.img -rw------- 1 root root 14047403 Oct 13 11:52 lib-modules-2.6.18-164.el5xen.tar.gz -rw------- 1 root root 2225793 Oct 13 11:47 vmlinuz-2.6.18-164.el5xen

Making an existing domU use the 32-bit kernel

Again, this is the whole reason I started messing with this. My first xen server was 32-bit, and all of the domU's were 32-bit. When I built a new machine with 8GB of RAM, I installed the 64-bit version of CentOS in order to be able to use the RAM without having PAE slow things down. Knowing what I know now, I probably could have just kept the same kernel files I was using before, but at least this way I've upgraded all of the children to the latest kernel, and I know how to do kernel upgrades in the future (I wasn't too sure about it before, so I left things as-is, under the "if it ain't broke, don't fix it" theory.)

The first step is to copy the kernel modules to the /lib/modules directory within the domU filesystem. This can be done using scp if the domU is running and you have access to it, or you can shut the domU down and mount its filesystem directly (which is what I did.)

# xm shutdown name

Wait for it to shut down. Use "xm list" to make sure it's gone.

# xm list
Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 4934 2 r----- 10037.4 name 7 511 1 -b---- 32.2
# xm list
Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 4934 2 r----- 10058.5

Now that we know the child isn't running...

# mount /dev/Disks/name_root /mnt/work
# cd /mnt/work/lib/modules
# tar xzpf /etc/xen/32bit/lib-modules-2.6.18-164.el5xen.tar.gz
# cd
# umount /mnt/work

You should already have an /etc/xen/name file for the domU. Find the "kernel =" and "ramdisk =" lines, and change them so they point to the vmlinuz and initrd files in the /etc/xen/32bit directory (or wherever you stored them.

# cat /etc/xen/name
name = "name" memory = 512 disk = [ 'phy:/dev/Disks/name_root,xvda1,w' , 'phy:/dev/Disks/name_swap,xvda2,w' ] vif = [ 'mac=4A:53:D0:6F:03:ac, bridge=xenbr0' , 'mac=4A:53:C0:A8:FA:ac, bridge=virbr0' ] uuid = "85e4933a-a51c-11dc-a13e-0002a58a9952" kernel = "/etc/xen/32bit/vmlinuz-2.6.18-164.el5xen" ramdisk = "/etc/xen/32bit/initrd-2.6.18-164.el5xenU.img" root = "/dev/xvda1 ro" extra = "3" vcpus = 1 on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'

After editing this file, if the domU is still running, shut it down. Then start it back up, and it should be running the 32-bit kernel.

Watch the output to make sure everything works. It should boot normally, and will normally end with a login prompt (unless you've customized the domU to boot into some other program, of course.)

Creating a new domU which uses the 32-bit kernel

I haven't had time to figure this out yet. Watch this space, I will be updating the page once I figure it out myself.