VMAPI for VirtualBox

VMAPI is a collection of scripts that allows easy creation of virtual machines and network topologies in Oracle VM VirtualBox.

System Requirements

Recommended host system: Ubuntu Server 12.04 64-bit.

VMAPI is developed on Ubuntu 12.04, which is supported until April 2017. A Server installation is preferred over the Desktop edition, because it contains significantly fewer packages, so that there's less overhead. VirtualBox has a headless frontend which can run without a graphic interface.

VMAPI is likely to work on other linux hosts, but I don't have time and resource to test them.

64-bit CPU and 64-bit OS are required to run VirtualBox efficiently. In addition, Intel VT-x and VT-d (or AMD's equivalent) should be enabled in BIOS settings, so that VMs can execute at the same speed of the physical machine.

VirtualBox must be installed on the host system. On Ubuntu you can type sudo apt-get install virtualbox to install VirtualBox. I suggest not to compile VirtualBox from source because you are more likely to run into problems. You also need to install Oracle VM VirtualBox Extension Pack to get VRDP support, which is necessary when preparing OS templates.

Supported guest systems:

• Ubuntu Server 12.04 64-bit
• Ubuntu Server 14.04 64-bit
• Ubuntu Server 14.04 32-bit
• Ubuntu Server 14.10 32-bit
• FreeBSD 9.0-RELEASE 64-bit

It's not too difficult to support a new guest OS, if you understand the network configuration files of that OS. Guest OS must have an SSH server.

Environment Setup

Create a runner account

Each VM corresponds to a user-level process. It's a good idea to create a separate user account to run these processes, so that you won't accidentally kill a VM process (which would power off the VM). This user account is called a runner account.

You can create a runner account 'vbox1' with sudo adduser vbox1. It should be same as a regular user account (not a system / service account). It does not need (and should not be given) sudo privilege. NEVER run any VM under root account.

If you have many VMs to deal with, you can create multiple runner accounts. It's a good idea to create a group that contains all runner accounts.

Unless otherwise noted, all subsequent actions are executed in the shell of a runner account. You can easily get a shell of a runner account with sudo -iu vbox1.

Create directory structure

1. Create a VMAPI host directory (eg. /data1/vbox1) to store all files of a runner account. Make sure there is sufficient disk space. As a reference, a recently-created Ubuntu 12.04 VM, after installing ns-3.16 inside, takes 7.3GB.
2. Create subdirectories under the VMAPI host directory: mkdir osimage vmapi-host vm vmdata vmdata/hostport
3. Download VMAPI host scripts and place them into vmapi-host/ subdirectory. Make them executable.
4. Edit ~/.bashrc and append these line at the end:

   export VMAPI_ROOT=/data1/vbox1
   export PATH=$PATH:$VMAPI_ROOT/vmapi-host
   cd $VMAPI_ROOT

   Therefore you are changed into VMAPI host directory automatically, and can type VMAPI host commands without typing the full path.

Download guest OS templates

OS installation is a long and tedious process. VMAPI creates VMs by cloning guest OS template, so that a VM can be ready in one or two minutes.

A guest OS template consists of:

• VM hardware spec, which is a series of VBoxManage commands that configures the VM
• virtual disk image (VDI)
• SSH key pair for VM master account

You may download prepared guest OS templates. They should be uncompressed and placed in osimage/$template/

The procedure of creating a guest OS template is in "Advanced Topics".

VMAPI Host Commands

These commands can be executed on the host machine, under the runner account.

The quick sequence of creating a new VM is:

vmcreate vm25 UbuntuPrecise 22025
vmstart vm25
vmadduser vm25 user password

VM Networking

VirtualBox allows up to 8 network cards (NICs) per VM. VMAPI configures the first NIC in NAT + DHCP mode, so that you can SSH into the VM, and the VM can access Internet. The other 7 NICs are available for your experiments.

vmintnet command adds a NIC to a VM, and attaches it to an internal network. The internal network is identified by its name: each network name is a different network. Each internal network behaves like a Ethernet layer-2 switch interconnecting all NICs attached to it.

A network topology can be created among a set of VMs by creating a separate internal network for each link. For example, the following commands create an Internet2 topology:

vmcreate I2STTLng UbuntuPrecise 22121
vmcreate I2CHIC UbuntuPrecise 22122
vmcreate I2NEWY UbuntuPrecise 22123
vmcreate I2WASH UbuntuPrecise 22124
vmcreate I2ATLA UbuntuPrecise 22125
vmcreate I2HOUS UbuntuPrecise 22126
vmcreate I2LOSA UbuntuPrecise 22127
vmcreate I2SALT UbuntuPrecise 22128
vmcreate I2KANS UbuntuPrecise 22129

# STTLng-SALT
vmintnet I2STTLng eth1 I2l0 10.21.69.0 255.255.255.254
vmintnet I2SALT eth1 I2l0 10.21.69.1 255.255.255.254
# STTLng-LOSA
vmintnet I2STTLng eth2 I2l1 10.21.69.2 255.255.255.254
vmintnet I2LOSA eth1 I2l1 10.21.69.3 255.255.255.254
# NEWY-WASH
vmintnet I2NEWY eth1 I2l2 10.21.69.4 255.255.255.254
vmintnet I2WASH eth1 I2l2 10.21.69.5 255.255.255.254
# CHIC-KANS
vmintnet I2CHIC eth1 I2l3 10.21.69.6 255.255.255.254
vmintnet I2KANS eth1 I2l3 10.21.69.7 255.255.255.254
# CHIC-NEWY
vmintnet I2CHIC eth2 I2l4 10.21.69.8 255.255.255.254
vmintnet I2NEWY eth2 I2l4 10.21.69.9 255.255.255.254
# CHIC-ATLA
vmintnet I2CHIC eth3 I2l5 10.21.69.10 255.255.255.254
vmintnet I2ATLA eth1 I2l5 10.21.69.11 255.255.255.254
# CHIC-WASH
vmintnet I2CHIC eth4 I2l6 10.21.69.12 255.255.255.254
vmintnet I2WASH eth2 I2l6 10.21.69.13 255.255.255.254
# ATLA-HOUS
vmintnet I2ATLA eth2 I2l7 10.21.69.14 255.255.255.254
vmintnet I2HOUS eth1 I2l7 10.21.69.15 255.255.255.254
# WASH-ATLA
vmintnet I2WASH eth3 I2l8 10.21.69.16 255.255.255.254
vmintnet I2ATLA eth3 I2l8 10.21.69.17 255.255.255.254
# LOSA-SALT
vmintnet I2LOSA eth2 I2l9 10.21.69.18 255.255.255.254
vmintnet I2SALT eth2 I2l9 10.21.69.19 255.255.255.254
# HOUS-KANS
vmintnet I2HOUS eth2 I2l10 10.21.69.20 255.255.255.254
vmintnet I2KANS eth2 I2l10 10.21.69.21 255.255.255.254
# HOUS-LOSA
vmintnet I2HOUS eth3 I2l11 10.21.69.22 255.255.255.254
vmintnet I2LOSA eth3 I2l11 10.21.69.23 255.255.255.254
# SALT-KANS
vmintnet I2SALT eth3 I2l12 10.21.69.24 255.255.255.254
vmintnet I2KANS eth3 I2l12 10.21.69.25 255.255.255.254

for vmname in I2STTLng I2CHIC I2NEWY I2WASH I2ATLA I2HOUS I2LOSA I2SALT I2KANS; do vmssh $vmname sudo shutdown -r now; done

This would probably take some time, because VM must be rebooted after adding each NIC. Otherwise, if multiple NICs are added before rebooting, their names in OS cannot be reliably determined.

VirtualBox also supports bridges networking which exposes a VM's NIC to the physical network, and host-only networking which connects a set of VMs to a TAP interface on the host machine. VMAPI does not directly support these modes, but you can use vmintnet to add the interface, and manually modify the configuration with VBoxManage modifyvm command.

Advanced Topics

Create and Update Guest OS Template

To create a guest OS template, there's more work than installing an OS, but this cost is amortized over hundreds of VMs based on this template.

1. Use a separate runner account other than the one you use to run regular VMs.
2. Download the installation CD of the guest OS.
3. Create a VM, and start it.

   VBoxManage createvm --name UbuntuPrecise --basefolder /data1/vbox0/vm/ --register
   VBoxManage modifyvm UbuntuPrecise --ostype Ubuntu_64 --memory 1024 --cpus 4 --acpi on --ioapic on --hwvirtex on --nestedpaging on --boot1 disk --boot2 dvd --boot3 none --boot4 none
   VBoxManage modifyvm UbuntuPrecise --nic1 nat --nictype1 virtio --natnet1 "192.168.254/24" --natpf1 ssh,tcp,,2222,,22
   VBoxManage modifyvm UbuntuPrecise --audio none --clipboard disabled --usb off
   VBoxManage modifyvm UbuntuPrecise --vrde on --vrdeport 3389 --vrdeauthtype null
   
   VBoxManage createhd --filename /data1/vbox1/vm/UbuntuPrecise/main.vdi -size 12288
   VBoxManage storagectl UbuntuPrecise --name SATA --add sata
   VBoxManage storageattach UbuntuPrecise --storagectl SATA --port 0 --type hdd --medium /data1/vbox0/vm/UbuntuPrecise/main.vdi
   
   VBoxManage storagectl UbuntuPrecise --name IDE --add ide
   VBoxManage storageattach UbuntuPrecise --storagectl IDE --port 1 --device 0 --type dvddrive --medium /data1/dvds/ubuntu-12.04-server-amd64.iso
     # insert OS installation CD
   VBoxManage startvm UbuntuPrecise --type headless

   In these command lines: '2222' is SSH host port, '3389' is VRDP host port; they must be above 1024 and not occupied by other VMs or services. '12288' is VM's disk space (MB); it's very difficult to increase disk space after guest OS template is created, so allocate sufficient space for your experiments here. You may tune other parameters as needed, especially for non-Ubuntu systems.
4. On a machine with graphical interface, use a RDP client (Terminal Server Client, not VNC client) to connect to port 3389 of host machine. If firewall restricts this connection, you can specify another port number in the command line above, or use SSH tunnel.
5. Complete OS installation over RDP session, and install an SSH server. Create a 'master' user (with that name) and grant sudo privilege with NOPASSWD option.
6. Create a new SSH key pair: ssh-keygen -t rsa -N '' -f /data1/vbox0/vm/UbuntuPrecise/id_rsa Even if you already have an SSH key pair for personal or project use, you should create a new pair for guest OS templates. cat /data1/vbox0/vm/UbuntuPrecise/id_rsa.pub and copy the public key on your clipboard.
7. SSH into the VM: ssh -p 2222 master@localhost then enter the password chosen during OS installation. Inside the VM, umask 077; mkdir .ssh; vi .ssh/authorized_keys and paste the SSH public key. Exit from SSH session.
8. SSH into the VM with private key: ssh -i /data1/vbox1/vm/UbuntuPrecise/id_rsa.vm -p 2222 master@localhost
9. Inside the VM: update existing packages, and install necessary new packages, such as

   sudo apt-get install virtualbox-guest-utils build-essential php5-cli python3 \
     default-jdk ant curl traceroute tcpdump tshark libpcap-dev libexpat1-dev libssl-dev subversion git

   VirtualBox guest additions (Ubuntu package virtual-guest-utils) should always be installed if available for the guest OS.
10. Download VMAPI guest scripts and copy the scripts suitable for the guest OS into /home/master/vmapi-guest/
11. Make guest OS forget about Ethernet cards, so an Ethernet card with a different address is still named eth0. sudo rm /etc/udev/rules.d/*net* Exit from SSH session.
12. At host machine's runner account shell:

    VBoxManage guestcontrol UbuntuPrecise exec --image '/bin/rm' --username master --password 123456 \
      --wait-stderr -- /home/master/.bash_history
    VBoxManage guestcontrol UbuntuPrecise exec --image '/bin/sh' --username master --password 123456 \
     --wait-stderr -- -c 'sudo shutdown -h now'

    (replace '123456' with master's password). This makes guest OS forget about entered commands, and turns off the VM.

At this point, main.vdi (in /data1/vbox0/vm/UbuntuPrecise/) is the VDI for this new guest OS template, and id_rsa + id_rsa.pub is an SSH key pair that can manage the VM using master account. These files should be copied to /data1/vbox1/osimage/UbuntuPrecise/ so they become a guest OS template. If you have multiple runner accounts, you may copy them to a central location, and create symlinks in /data1/vbox1/osimage/UbuntuPrecise/

To upgrade a guest OS template, start the VM with VBoxManage startvm UbuntuPrecise --type headless, SSH into the VM, and upgrade the packages. Perform the last two steps above, and copy main.vdi to osimage directory.

VMAPI Host Directory

The subdirectories and files in VMAPI host directory are:

VMAPI Guest Commands

These commands are exposed by a guest OS. New guest OS templates are expected to support these commands. These scripts are places in /home/master/vmapi-guest/ directory, and are executed over SSH under master account with sudo prepended. They are typically called by VMAPI host scripts, and are not used directly.