Last updated $Date: 2005/07/25 13:18:43 $
Martti Kuparinen <martti.kuparinen@iki.fi>
Xen is a relatively new technology to enable several virtual machines (domU) to run on one computer. The purpose of this article is to determine what operating system (NetBSD or Linux) should be selected as domain 0 (dom0) operating system to get the best performance when running several CPU and disk intensive virtual machines at the same time.
3 identical Dell PowerEdge 750 servers with the following details:
xen1: NetBSD dom0
xen2: Linux dom0
xen3: Linux dom0
All dom0s were given 128 MB RAM and 128 MB swap. The / filesystem was around 10 GB and everything else (210+ GB) was used for the domUs' virtual harddisks. On xen1 and xen2 the data area was mounted on /xen while xen3 used that 210+ GB for one LVM volume group. 20 LVM logical volumes (1 for each domU) were created inside the LVM volume group. Below is an example of disk usage after 20 domUs.
xen1:~> df -h Filesystem Size Used Avail Capacity Mounted on /dev/wd0a 9.5G 2.9G 6.1G 31% / /dev/wd0e 210G 80G 119G 40% /xen kernfs 1.0K 1.0K 0B 100% /kern procfs 4.0K 4.0K 0B 100% /proc xen1:~> xen1:~> cat /etc/fstab /dev/wd0a / ffs rw 1 1 /dev/wd0b none swap sw 0 0 /dev/wd0e /xen ffs rw 1 2 /dev/cd0a /cdrom cd9660 ro,noauto 0 0 kernfs /kern kernfs rw 0 0 procfs /proc procfs rw 0 0 xen1:~>
xen2:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.2G 1.5G 7.3G 17% / tmpfs 62M 0 62M 0% /dev/shm /dev/sda3 220G 81G 129G 39% /xen /dev 9.2G 1.5G 7.3G 17% /.dev none 5.0M 2.8M 2.3M 55% /dev xen2:~> xen2:~> cat /etc/fstab proc /proc proc defaults 0 0 /dev/sda1 / ext3 defaults,errors=remount-ro 0 1 /dev/sda2 none swap sw 0 0 /dev/sda3 /xen ext3 defaults 0 2 /dev/hda /media/cdrom0 iso9660 ro,user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0 xen2:~>
xen3:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.2G 1.5G 7.3G 17% / tmpfs 62M 0 62M 0% /dev/shm /dev 9.2G 1.5G 7.3G 17% /.dev none 5.0M 2.8M 2.3M 55% /dev xen3:~> xen3:~> cat /etc/fstab proc /proc proc defaults 0 0 /dev/sda1 / ext3 defaults,errors=remount-ro 0 1 /dev/sda2 none swap sw 0 0 /dev/hda /media/cdrom0 iso9660 ro,user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0 xen3:~>
LVM volumes in xen3 were configured like this:
xen3:~> fdisk -l /dev/sda
Device Boot Start End Blocks Id System
/dev/sda1 * 1 1155 9277506 83 Linux
/dev/sda2 1156 1216 489982+ 82 Linux swap / Solaris
/dev/sda3 1217 30394 234372285 83 Linux
xen3:~> pvcreate /dev/sda3
xen3:~> vgcreate vg0 /dev/sda3
xen3:~> vgchange -a y vg0
xen3:~> for i in `seq 1 20`; do lvcreate -L 4G vg0 -n lv${i}; done
xen3:~> for i in `seq 1 20`; do bzcat netbsd.bz2 > /dev/vg0/lv${i}; done
And finally some bonnie results from dom0s without any domUs started to give some idea about the SATA disk's speed.
xen1:~> bonnie++ -s 2048 -d /var/tmp -u root
[...]
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xen1 2G 55811 53 57248 30 13491 6 56266 74 56682 14 104.9 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1372 32 +++++ +++ 3324 12 1456 34 5310 100 3072 35
[From dmesg]
wd0 at atabus1 drive 0: <WDC WD2500JD-75HBB0>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 232 GB, 484406 cyl, 16 head, 63 sec, 512 bytes/sect x 488281250 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(piixide0:1:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
xen2:~> bonnie++ -s 2048 -d /var/tmp -u root
[...]
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xen2 2G 6063 97 55994 58 26331 15 6466 98 58409 20 131.0 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 666 99 +++++ +++ +++++ +++ 675 99 +++++ +++ 2740 99
[From dmesg]
ata1: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xFEA8 irq 15
ata1: dev 0 cfg 49:2f00 82:3469 83:7f61 84:4003 85:3469 86:3e41 87:4003 88:207f
ata1: dev 0 ATA, max UDMA/133, 488281250 sectors: lba48
ata1: dev 0 configured for UDMA/133
scsi0 : ata_piix
Vendor: ATA Model: WDC WD2500JD-75H Rev: 08.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
This is weird! On xen2 with a normal Linux kernel (i.e. no Xen) I get this. What is Xen doing as it affects the "Per Chr" and sequential and random file create numbers like this?
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xen2 8G 29093 98 56445 6 20761 3 25706 88 53444 3 176.3 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 4081 99 +++++ +++ +++++ +++ 4125 99 +++++ +++ 12994 99
[From dmesg]
ata1: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xFEA8 irq 15
ata1: dev 0 cfg 49:2f00 82:3469 83:7f61 84:4003 85:3469 86:3e41 87:4003 88:207f
ata1: dev 0 ATA, max UDMA/133, 488281250 sectors: lba48
ata1: dev 0 configured for UDMA/133
scsi0 : ata_piix
Vendor: ATA Model: WDC WD2500JD-75H Rev: 08.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 488281250 512-byte hdwr sectors (250000 MB)
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Each virtual machine (domU) was running NetBSD/xen 3.0 with 128 MB RAM and a 4 GB harddisk. The virtual harddisk provided for each domU is a 4 GB file in the domain 0's filesystem. The configuration files used were like this:
NetBSD dom0
kernel = "/xen/kernel/netbsd-XENU" memory = 128 name = "NetBSD-01" cpu = -1 disk = [ 'phy:/dev/vnd1d,wd0d,w' ] root = "/dev/wd0d"
Linux dom0 with ext3
kernel = "/boot/netbsd-XENU" memory = 128 name = "NetBSD-01" cpu = -1 disk = [ 'file:/xen/vm/netbsd.1,0x3,w' ] root = "/dev/wd0d"
Linux dom0 with LVM
kernel = "/boot/netbsd-XENU" memory = 128 name = "NetBSD-01" cpu = -1 disk = [ 'phy:/dev/vg0/lv1,0x3,w' ] root = "/dev/wd0d"
The actual test case was a simple NetBSD 3.0 kernel compilation running at the same time on all domUs. You can find here the script I used to start the compilation at the same time on each domU. And finally this is how things look like on dom0 during tests when 20 domUs were running at the same time:
xen1:~> xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 128 0 r---- 430.7 NetBSD-01 1 127 0 ----- 200.6 9601 NetBSD-02 2 127 0 ----- 206.5 9602 NetBSD-03 3 127 0 ----- 200.3 9603 NetBSD-04 4 127 0 ----- 199.9 9604 NetBSD-05 5 127 0 ----- 201.1 9605 NetBSD-06 6 127 0 ----- 200.7 9606 NetBSD-07 7 127 0 ----- 200.1 9607 NetBSD-08 8 127 0 ----- 200.5 9608 NetBSD-09 9 127 0 ----- 201.3 9609 NetBSD-10 10 127 0 ----- 200.6 9610 NetBSD-11 11 127 0 ----- 200.0 9611 NetBSD-12 12 127 0 ----- 199.5 9612 NetBSD-13 13 127 0 ----- 199.6 9613 NetBSD-14 14 127 0 ----- 199.5 9614 NetBSD-15 15 127 0 ----- 199.7 9615 NetBSD-16 16 127 0 ----- 200.2 9616 NetBSD-17 17 127 0 ----- 199.6 9617 NetBSD-18 18 127 0 ----- 203.0 9618 NetBSD-19 19 127 0 ----- 202.9 9619 NetBSD-20 20 127 0 ----- 200.7 9620 xen1:~> xm info system : NetBSD host : xen1.mydomain.com release : 3.0_BETA version : NetBSD 3.0_BETA (DOMAIN0) #0: Tue Jun 7 09:22:25 EEST 2005 root@xen1.mydomain.com:/usr/src/sys/arch/i386/compile/DOMAIN0 machine : i386 cores : 1 hyperthreads_per_core : 2 cpu_mhz : 2800 memory : 3967 free_memory : 1250 xen2:~> xm info system : Linux host : xen2.mydomain.com release : 2.6.11-xen0 version : #1 Tue Jun 21 12:10:02 EEST 2005 machine : i686 cores : 1 hyperthreads_per_core : 2 cpu_mhz : 2800 memory : 3967 free_memory : 1255
1 x NetBSD 3.0 128 MB XENU
1: 480.08 real 442.51 user 80.50 sys
4 x NetBSD 3.0 128 MB XENU
1: 1628.58 real 1420.81 user 196.93 sys 2: 1622.36 real 1411.24 user 213.27 sys 3: 1629.47 real 1417.28 user 198.52 sys 4: 1623.85 real 1412.42 user 210.58 sys
10 x NetBSD 3.0 128 MB XENU
1: 4119.23 real 3464.33 user 464.66 sys 2: 4107.08 real 3436.14 user 509.20 sys 3: 4118.83 real 3468.69 user 457.37 sys 4: 4127.71 real 3447.81 user 507.54 sys 5: 4122.46 real 3467.49 user 460.19 sys 6: 4116.55 real 3441.69 user 510.65 sys 7: 4122.65 real 3470.93 user 460.86 sys 8: 4126.36 real 3450.22 user 506.60 sys 9: 4122.95 real 3465.88 user 464.42 sys 10: 4127.80 real 3450.75 user 505.01 sys
20 x NetBSD 3.0 128 MB XENU
1: 8454.50 real 7819.64 user 857.02 sys 2: 8374.66 real 6638.31 user 905.05 sys 3: 8439.73 real 6710.57 user 851.16 sys 4: 8483.23 real 6701.88 user 901.58 sys 5: 8459.54 real 6726.61 user 857.08 sys 6: 8478.38 real -2054107019.94 user 2054115168.86 sys 7: 8459.59 real -18546402.96 user 18553622.49 sys 8: 8476.76 real -74527564.98 user 1766167245.80 sys 9: 8453.98 real 6775.33 user 855.81 sys 10: 8460.47 real 6700.59 user 909.38 sys 11: 8424.97 real 6711.66 user 839.15 sys 12: 8483.24 real 6697.74 user 901.97 sys 13: 8460.55 real 8494.63 user 917.48 sys 14: 8483.49 real 17720827.49 user 17713708.63 sys 15: 8443.87 real 6723.99 user 843.52 sys 16: 8469.37 real 6694.90 user 911.28 sys 17: 8448.26 real 6730.69 user 845.63 sys 18: 8426.06 real 7411.03 user 3305.69 sys 19: 8380.67 real 1446917722.06 user 1446913028.59 sys 20: 8469.65 real 1101215076.41 user 1101208245.59 sys
1 x NetBSD 3.0 128 MB XENU
1: 623.78 real 572.36 user 99.20 sys
4 x NetBSD 3.0 128 MB XENU
1: 2533.76 real 2227.88 user 344.38 sys 2: 2536.29 real 2225.85 user 346.54 sys 3: 2536.89 real 2230.29 user 342.07 sys 4: 2536.86 real 2231.20 user 341.77 sys
10 x NetBSD 3.0 128 MB XENU
1: 6350.96 real 5515.26 user 815.35 sys 2: 6340.29 real 5509.35 user 822.88 sys 3: 6345.66 real 5522.04 user 810.14 sys 4: 6348.31 real 5508.73 user 823.30 sys 5: 6349.82 real 5508.63 user 821.94 sys 6: 6349.71 real 5507.00 user 824.16 sys 7: 6340.85 real 5507.99 user 821.12 sys 8: 6340.88 real 5506.70 user 821.90 sys 9: 6349.30 real 5507.31 user 826.85 sys 10: 6350.12 real 5513.93 user 820.56 sys
20 x NetBSD 3.0 128 MB XENU
1: 12753.82 real 10963.56 user 1606.86 sys 2: 12734.47 real 10948.04 user 1603.86 sys 3: 12755.07 real 10933.30 user 1631.23 sys 4: 12740.55 real 10941.74 user 1614.60 sys 5: 12742.93 real 10951.22 user 1609.56 sys 6: 12755.12 real 10949.57 user 1614.60 sys 7: 12743.45 real 10967.47 user 1598.64 sys 8: 12748.41 real 10944.09 user 1617.30 sys 9: 12733.77 real 10962.78 user 1594.03 sys 10: 12756.37 real 10970.87 user 1594.62 sys 11: 12756.27 real 10962.21 user 1607.66 sys 12: 12754.23 real 10965.73 user 1601.08 sys 13: 12755.48 real 10947.61 user 1617.78 sys 14: 12734.12 real 10953.92 user 1607.20 sys 15: 12752.67 real 10933.37 user 1628.27 sys 16: 12755.92 real 10959.01 user 1607.46 sys 17: 12726.90 real 10947.83 user 1595.92 sys 18: 12750.96 real 10949.53 user 1616.29 sys 19: 12746.29 real 10948.98 user 1616.27 sys 20: 12712.93 real 10948.44 user 1590.15 sys
1 x NetBSD 3.0 128 MB XENU
1: 623.72 real 571.23 user 99.60 sys
4 x NetBSD 3.0 128 MB XENU
1: 2543.97 real 2237.40 user 340.61 sys 2: 2544.21 real 2235.59 user 343.32 sys 3: 2544.32 real 2237.84 user 341.88 sys 4: 2543.77 real 2237.65 user 339.44 sys
10 x NetBSD 3.0 128 MB XENU
1: 6382.11 real 5546.00 user 816.83 sys 2: 6379.74 real 5553.35 user 811.49 sys 3: 6382.54 real 5535.19 user 825.39 sys 4: 6380.32 real 5542.21 user 816.47 sys 5: 6381.87 real 5532.74 user 825.14 sys 6: 6379.28 real 5532.87 user 827.21 sys 7: 6380.70 real 5542.02 user 819.06 sys 8: 6382.40 real 5540.00 user 821.34 sys 9: 6382.29 real 5538.89 user 820.43 sys 10: 6382.55 real 5540.41 user 822.66 sys
20 x NetBSD 3.0 128 MB XENU
1: 12817.99 real 11036.48 user 1595.61 sys 2: 12818.89 real 11047.83 user 1584.44 sys 3: 12817.28 real 11044.66 user 1586.72 sys 4: 12825.13 real 11064.80 user 1573.14 sys 5: 12807.44 real 11018.24 user 1603.60 sys 6: 12818.85 real 11028.69 user 1613.18 sys 7: 12824.40 real 11026.58 user 1610.20 sys 8: 12823.05 real 11045.71 user 1592.87 sys 9: 12820.36 real 11029.04 user 1606.42 sys 10: 12824.39 real 11043.16 user 1596.57 sys 11: 12817.71 real 11049.55 user 1587.21 sys 12: 12818.56 real 11047.25 user 1587.95 sys 13: 12821.95 real 11038.06 user 1595.35 sys 14: 12818.67 real 11050.49 user 1583.07 sys 15: 12824.87 real 11035.38 user 1598.38 sys 16: 12824.54 real 11028.85 user 1609.02 sys 17: 12823.10 real 11039.95 user 1597.31 sys 18: 12814.31 real 11024.97 user 1605.24 sys 19: 12824.76 real 11057.35 user 1576.98 sys 20: 12824.58 real 11045.16 user 1593.31 sys
These tests show that it's possible to run several virtual machines at the same time on one physical server by using only open source software. One of the important points here is to have enough memory on the server so that each virtual machine can have enough memory to perform its duties without swapping. Therefore I've allocated all virtual machines at least 128 MB in real-life configurations.
According to the Xen documentation Linux loopback block device (file-backed VBD) has poor performance under heavy I/O. Therefore I've also tested Linux dom0 with LVM. The results between loopback mounted virtual disks and LVM based virtual disks are very similar (maybe too similar?). This could be because of the problems noticed when running bonnie in dom0. I'm willing to perform the Linux tests again if a solution for these disk performance problems is found.
Some of the numbers produced by /usr/bin/time are negative. This is a bug in NetBSD/xen which needs to be fixed. Fortunately the "real" numbers are all valid so it's possible to calculate mean values.
| Number of domUs |
Compilation time when using NetBSD dom0 |
Compilation time when using Linux dom0 (ext3) |
Compilation time when using Linux dom0 (LVM) |
Linux dom0 (ext3) execution time vs NetBSD dom0 |
Linux dom0 (LVM) execution time vs NetBSD dom0 |
| 1 | 480 | 624 | 624 | +30 % | +30 % |
| 4 | 1626 | 2536 | 2544 | +56 % | +56 % |
| 10 | 4121 | 6347 | 6381 | +54 % | +55 % |
| 20 | 8452 | 12745 | 12821 | +51 % | +52 % |