Jail Security

June 26th, 2004

UMLazi Guests are locked in tightly controlled chroot()ed jails to prevent attackers who take control of a Guest from getting much access to the host system. Here are some of the considerations that went into the design of the jail.

* Directory permissions

It’s a lot more difficult to bust out of a jail and/or attack a machine from a jail if you don’t have the ability to write files (such as exploit scripts) to the jail. By locking down the permissions of the directories in the jail, and setting the ownerships to root, Guest users aren’t able to write files into the jail.

In a couple of places, user write permissions are unavoidable because the UML kernel _must_ write to the directory. These are /tmp (where the UML kernel mmaps its RAM file) and /control (where the UML kernel stores its PID file and mconsole socket) . We mitigate the risk by limiting the size of the files and number of inodes the user is allowed to create to the bare minimum needed to run a Guest. We also mount these directories with “noexec” and “nosuid”, to make things especially difficult.

* UBD devices

A root user inside a Guest might overwrite the beginning of a ubd device with a malicious program, then, chmod it executable, and coerce the UML kernel into executing it. The best defense we can offer here is preventing the Guest from being able to chmod the UBD device. Since a user can’t chmod a file they don’t own, we set the ownership of the UBD devices to “root”, and give the UML user write permissions through group ownership.

* Kernel permissions

The UML kernel in the jail is owned by root and not writable by the user.

* /dev/net/tun

Okay, if I’m pointing out the strengths of the jail, I also need to point out the weaknesses. /dev/net/tun is probably the biggest weaknessl. This device needs to exist in the jail and be writable by the Guest user for networking to function. Unfortunately, any user with write permissions on this device can create as many tun devices as they want on the host, named pretty much whatever they want. I don’t think this can be used to compromise the Host (though it is a window into kernelspace), but it could be used as a DoS.

The tuntap driver _does_ understand users, and has the ability to limit access to tun interfaces to certain users. I’m thinking this could be solved by a patch that limits tun interface creation to root. Any other ideas?

Stable UML Kernels

June 16th, 2004

Just a warning: If you want a nice, stable UML farm, stay away from 2.4.26-1um. “Apparently”:http://sourceforge.net/mailarchive/message.php?msg_id=8570980, 2.4.24-1um is the latest stable patch.

*Update*: Also, some people have reported problems with the 2.6.6 kernel on the download. I’d advise against using it in production until a new one is posted.

I’ve been running 2.4.22-5um since last October. For the most part it’s been pretty stable– I have guests that have been up since the first day I pushed it out– but there were still the occasional odd kernel panics. Over time, as the farm grew from dozens of guests, to several hundred guests, those occassional crashes added up to several a day.

So, I built a new kernel (2.4.26-1um) and after some perfunctory testing (does it boot, can I compile a kernel in it, will it pass a few thousand emails) pushed it out to a batch of about a hundred non-critical guests. Man, what a disaster. I immediately went from one crash for every three or four hours, to three or four crahes for every one hour. Then, when backups kicked off, they all pretty much collapsed at the same time.

Looks like 2.4.26-1m has some problems with high load.

So, now I have a 2.4.26 kernel patched with the 2.4.24-1um patch. It’s pushed out to the same hundred hosts, and so far, four hours later, no crashes. After it runs for a few days, I’ll promote it to “default”, and the rest of the guests will pick it up with their maintenance reboots.

For UMLazi users, the new kernel (”2.4.26-20040615-1um”:/src/linux-2.4.26-20040615-1um.bz2) is available on the download page.

Adding More Loop Devices

June 12th, 2004

When creating a large number of UMLazi Guests at the same time, or creating a guest with more than 8 filesystems, you might run into an error like this one:

mount: could not find any free loop device

This is because your host kernel is limited in the number of loopback filesystems it can have mounted at any one time. The default limit is 8. I like to set this to 64.

To increase this value, you need to tell the kernel you want more devices:
* If you have the “loop” driver compiled into your kernel, you need to add “max_loop=64″ to your kernel boot arguments (the “append=” line in lilo.conf, or to the end of the “kernel” line in grub’s menu.lst), and reboot.
* If you have the “loop” driver compiled as a module, you need to insmod it with “max_loop=64″ added to its options. On Debian systems, you edit /etc/modules.conf and add “options loop max_loop=64″, and “rmmod loop” “modprobe loop”.

If you’re using devfs, then stop here. The new /dev/loop* will appear automatically. If not, you’ll need to edit and run MAKEDEV:
Change:

        loop)
                for part in 0 1 2 3 4 5 6 7
                do
                        makedev loop$part b 7 $part $disk
                done
                ;;

To:

        loop)
                for part in `seq 0 63`
                do
                        makedev loop$part b 7 $part $disk
                done
                ;;

And run “MAKEDEV loop”.

UMLazi 1.0.5.1

June 8th, 2004

Release 1.0.5.1 clears up all reported bugs from 1.0.5. Most importantly, “systemcheck” now works.

There’s improved support for Fedora Core 2, as well, thanks to Antonello Alonzi.

Converting root_fs into filesystem templates

June 7th, 2004

Here’s a quick little HOWTO on converting root_fs images (such as the ones found on “The UML Download Page”:http://user-mode-linux.sourceforge.net/dl-sf.html), into a UMLazi Filesystem Template.

Basically, a root_fs is a bunch of files inside a filesystem, and a filesystem template is a bunch of files inside a tarball. To convert between the two, you need only mount the root_fs (so you can get at the tasty files inside), and tar them up.

The first step is to download a root_fs. The User-Mode Linux “Download Page”:http://user-mode-linux.sf.net/dl-sf.net is a good source. Now, mount it, and make a tarball of it:

 # cd /data/umlazi/templates  (or wherever your templates are)
 # mkdir RootFS-Template (or whatever name you prefer)
 # mount -o loop /path/to/root_fs /mnt
 # tar -jcvf filesystem.tar.bz2 -C /mnt .
 # umount /mnt

Viola. Put “RootFS-Template” in your UMLazi Guest’s ‘templates’ file, and create yourself a Guest.

Known Issues with 1.0.5

June 7th, 2004

Antonello Alonzi reports the following bugs in 1.0.5:

* There’s a typo in “systemcheck” which prevents it from running. D’oh. Add a “)” on line 44, after the “file_exists /usr/bin/sar” block.
* On Fedora Core 2, the chkconfig syntax in the umlazi.init script isn’t complete. Locate the chkconfig line and add this line before it:

# description: This script won't touch a Guest if it's not "enabled"

* Also on Fedora Core 2, “nice” is in “/bin/”, not “/usr/bin”. This prevents Guests from starting on Fedora Core 2. Fix by changing “/usr/bin/nice” on line 515 of lib_uml to just “nice”.

Update: These bugfixes have been rolled into UMLazi 1.0.5.1.

UMLazi 1.0.5 Released

June 7th, 2004

I’m pleased to announce UMLazi 1.0.5. This release provides a few new features, and a bunch of behind the scenes enhancements.

Get it from the “Download Page”:http://umlazi.org/category/downloads/

Update: Please see the “errata thread”:http://umlazi.org/2004/06/07/known-issues-with-105/ for known issues with this release.

New Features:
* *Filesystem Growing* – Guest disks may be grown automatically by increasing the “size” variable in a volume’s configuration directory and running “umlazi resize {guestname}”. This currently only works for ext2, ext3, and swap, with partial support for XFS (xfs_growfs must still be executed from inside the Guest, since it doesn’t understand files that contain filesystems.)
* *Autorestart* – When activated, guests that die unexpectedly will be restarted automatically by UMLazi. To activate, create a file named “autorestart” in the Guest’s configuration directory prior to starting up the Guest.
* *Guest CPU and Network Utilization Data* – When activated CPU and Network performance information is logged to the the “logs” directory in the Guest’s homedir. Enable by creating a file named “generate_stats” in the Guest’s configuration directory.
* *CPU Prioritization* – Guests may be given CPU scheduling priority over (or under) other guests by putting a value between -20 and +20 in a file named “cpu_priority” in the Guest’s configuration directory. This sets the “nice” value of the process, so the lower the number, the higher the priority.

Enhancements:
* For those who just can’t get used to the Host Configuration Directory format, check out “txt2hcd” in the “scripts/” directory, and “cfg-template.txt”.
* “start” is more intelligent at detecting whether a Guest has sucessfully started or not.
* “stop” gives the Guest longer to shut down cleanly before resorting to tougher shutdown methods.
* “snapshot” has better error detection.
* Added a “systemcheck” command to verify that your system has all the components necessarily to run UMLazi. Run it with “umlazi systemcheck”.
* Added an init script. “umlazi.init” in the build directory. “make install-init-update-rc.d” to install it on Debian systems, and “make install-init-chkconfig” to install it on Fedora/Redhat systems.
* Local hacks may now be added to $(DATADIR)/lib/lib_umlazi_hacks
* All lib_config variables may now be overridden. For example: “DEBUG=11 umlazi create test-uml” will run umlazi with _full_ debugging.
* Added DEBUG level 11, for those who like watching pages and pages of shell script roll by whenever they run a command.
* Several installation path fixes and bugfixes thanks to Leendert Meyer.
* Minor bugfixes to binding and run control.

As usual, please direct your bug reports and support questions to support@umlazi.org.

Support Addresses

June 7th, 2004

Until we have the mailing lists and bugtracking systems up, please email your bug reports and support questions to support@umlazi.org.

Features

June 3rd, 2004

Features of UMLazi v1.0.0

UMLazi is a management tool for configuring and running User-Mode Linux based virtual machines. By setting a few simple configuration variables, UMLazi can control disk layout, network configuration, memory usage, processor priority.

Management Features:

  • Run control – Stop, start, create, and destroy UMLs. Set them to boot automatically when the host boots, and shut down when the host shuts down.

  • Resource Control – Set limits to the memory assigned to specific guests, and prioritize their CPU usage relative to other guests on the same host.
  • Disk Management – Define mountpoints and filesystems. UMLazi supports ext2, ext3, jfs, and XFS, mounted anyway you want them. Disk snapshots can be taken of running guests, allowing rollback to earlier disk states. Disks may be set as “undoable”, allowing you to make changes to the guest filesystems (installing a piece of software, for instance) then either commit the changes or discard them.
  • Console Access – Access the console of a UMLazi Guest as through you were sitting right in front of it. All UMLazi Guest console output is logged, and archived with every reboot.
  • Networking Configuration – Set up UMLazi Guests on their own private networks, or connect them to your live network in several different ways. Route, firewall, NAT, or bridge each guest individually. Set up as many virtual networks and virtual network interfaces as you wish to create arbitrarily complex networks layouts.

Security:

  • Jailed – Each UMLazi Guest is locked in chroot(2) jail, with the minimal files and permissions set to allow the guest to run.

  • Unprivileged Execution – Each UMLazi Guest runs as its own user, rather than running as root.
  • Built in Firewalling (with Shorewall) – Integrates with shorewall, to secure each guest behind a firewall. Only authorized traffic is allowed in or out, and all connections are logged.

Monitoring:

  • Cisco NetFlow Exports – Get individual traffic stats about each UML, on a per-service level.

  • AutoRestart – If a guest dies unexpectedly, UMLazi will automatically restart it.
  • Processor Utilization Stats – See which UMLazi Guests are using the most CPU.

UMLazi and Kernel 2.6 on the Host

May 2nd, 2004

I just upgraded the kernel on one of my dev hosts to 2.6.6-rc3-skas, and it’s working just fine.

Next: Guests with 2.6 series kernels.
Read the rest of this entry »

UMLconfig – Automatic guest configuration

May 1st, 2004

People who don’t use the filesystem templates provided on the download page are missing one of the nicest features of UMLazi: UMLconfig.

UMLconfig takes the UMLazi configuration directory you created on the host, and uses it inside the Guest to automatically configure things like network interfaces, the hostname, root password, and DNS settings. This greatly reduces the configuration burden involved in creating new guests.

Due to the large size of the templates.tar, umlconfig-0.7.0 is now it’s own download on the download page.

Networking Documentation

April 2nd, 2004

UML Networking can be kind of tricky if you don’t understand exactly how things are done with User-Mode-Linux.

Here’s a pre-release of the Networking Guide for UMLazi v1.0. It’s not much, but It’s Better Than Nothing™

UMLazi FAQ: “mount: special device /proc/mm does not exist.”

April 2nd, 2004
UMware DEBUG: Mounting required files and directories in jail.
mount: special device /proc/mm does not exist
UMware ERROR: Unable to mount jail files.

A few people have reported this error message when starting UMLazi guests.

This happens when UMLazi is run on a host without SKAS support. UMLazi 1.0.3.2 expects SKAS support in the host kernel, and throws a fatal error if it doesn’t exist.

Here’s the solution.
Read the rest of this entry »

Version 1.0.3.1

March 8th, 2004

Looks like the “2.4.24 kernel support” changes didn’t actually make it into the 1.0.3 release.

This update contains that patch, and is otherwise identical to 1.0.3.

UMLazi Binding

March 7th, 2004

New as of 1.0.3 is UMLazi Binding. “Binding” allows multiple Guests to share the same disk images for reading, while writing new data to private “Copy On Write” (COW) files.

This allows you to, for instance, create a new guest, get all the software you want installed on it, tweak it out the way you want it, freeze it, and spark up a farm of disk-identical guests.

Read on for some brief documentation on using binding.
Read the rest of this entry »

UMLazi 1.0.3 – Come and Get it

March 7th, 2004

UMLazi 1.0.3 is now released. Get it in the download page.

Changes include:

  • Support for newer kernels (post 2.4.24)
  • Binding support (Shared disk, with COWs)
  • Numerous bugfixes and enhancements (see extended)

Read the rest of this entry »

Version 1.0.2 Available For Download

November 28th, 2003

Release 1.0.2, along with some sample filesystem templates and a starter UML kernel, are up for download on the downloads page. Your feedback is welcome.

Installation Guide

November 28th, 2003

An Installation Guide for UMLazi v1.0 is now available.

Live Snapshots

November 1st, 2003

The UMLazi New Feature of the Day is Live Snapshotting.

Although it’s been possible to make undoable disks in UMLazi for some time, it hasn’t been easy to take a snapshot of the disks of a live, running system.

This release of UMLazi includes the “snapshot” command.  This allows you to take a disk snapshot of a live, running system, and restore from it later.

  • Take a snapshot:
    • # umlazi snapshot test-uml BeforeTrashing
  • Totally trash the test-uml:
    • # ssh root@test-uml rm -rf /
  • Restore the test-uml
    • # umlazi stop test-uml
    • # umlazi restore test-uml BeforeTrashing
    • # umlazi start test-uml

(If you don’t specify a name for the snapshot ["BeforeTrashing"], one will be assigned to it using the current date and time.)

Limitations:

  • While the snapshot is running, the UML is complete paused and unable to do any work.  Once the snapshot is complete, the host will continue where it left off.
  • Snapshots are slow.  In my testbed, it takes 1-2 minutes to take a snapshot.  The more data you have, the longer it will take.
  • Though snapshots are live, restores are not.   These are only disk snapshots, not memory or process snapshots, so the guest will need to be stopped while the restore is taking place.

Back from Vacation

September 24th, 2003

Just had a BEAUTIFUL vacation up in Sechelt, British Columbia, Canada.  I’ll post pictures in a few days.

Now, back to getting this release out the door!