VMAPI for VirtualBox
VMAPI is a collection of scripts that allows easy creation of virtual machines and network topologies in Oracle VM VirtualBox.
Recommended host system: Ubuntu Server 12.04 64-bit.
VMAPI is developed on Ubuntu 12.04, which is supported until April 2017. A Server installation is preferred over the Desktop edition, because it contains significantly fewer packages, so that there's less overhead. VirtualBox has a headless frontend which can run without a graphic interface.
VMAPI is likely to work on other linux hosts, but I don't have time and resource to test them.
64-bit CPU and 64-bit OS are required to run VirtualBox efficiently. In addition, Intel VT-x and VT-d (or AMD's equivalent) should be enabled in BIOS settings, so that VMs can execute at the same speed of the physical machine.
VirtualBox must be installed on the host system. On Ubuntu you can type
sudo apt-get install virtualbox to install VirtualBox. I suggest not to compile VirtualBox from source because you are more likely to run into problems. You also need to install Oracle VM VirtualBox Extension Pack to get VRDP support, which is necessary when preparing OS templates.
Supported guest systems:
- Ubuntu Server 12.04 64-bit
- Ubuntu Server 14.04 64-bit
- Ubuntu Server 14.04 32-bit
- Ubuntu Server 14.10 32-bit
- FreeBSD 9.0-RELEASE 64-bit
It's not too difficult to support a new guest OS, if you understand the network configuration files of that OS. Guest OS must have an SSH server.
Create a runner account
Each VM corresponds to a user-level process. It's a good idea to create a separate user account to run these processes, so that you won't accidentally kill a VM process (which would power off the VM). This user account is called a runner account.
You can create a runner account 'vbox1' with
sudo adduser vbox1. It should be same as a regular user account (not a system / service account). It does not need (and should not be given) sudo privilege. NEVER run any VM under root account.
If you have many VMs to deal with, you can create multiple runner accounts. It's a good idea to create a group that contains all runner accounts.
Unless otherwise noted, all subsequent actions are executed in the shell of a runner account. You can easily get a shell of a runner account with
sudo -iu vbox1.
Create directory structure
- Create a VMAPI host directory (eg. /data1/vbox1) to store all files of a runner account. Make sure there is sufficient disk space. As a reference, a recently-created Ubuntu 12.04 VM, after installing ns-3.16 inside, takes 7.3GB.
- Create subdirectories under the VMAPI host directory:
mkdir osimage vmapi-host vm vmdata vmdata/hostport
- Download VMAPI host scripts and place them into vmapi-host/ subdirectory. Make them executable.
- Edit ~/.bashrc and append these line at the end:
export VMAPI_ROOT=/data1/vbox1Therefore you are changed into VMAPI host directory automatically, and can type VMAPI host commands without typing the full path.
Download guest OS templates
OS installation is a long and tedious process. VMAPI creates VMs by cloning guest OS template, so that a VM can be ready in one or two minutes.
A guest OS template consists of:
- VM hardware spec, which is a series of VBoxManage commands that configures the VM
- virtual disk image (VDI)
- SSH key pair for VM master account
You may download prepared guest OS templates. They should be uncompressed and placed in osimage/$template/
The procedure of creating a guest OS template is in "Advanced Topics".
VMAPI Host Commands
These commands can be executed on the host machine, under the runner account.
|command||description||VM power state|
|vmcreate $vmname $template $sshhostport||create a new VM, and redirect host's $sshhostport to guest's port 22|
|vmadduser $vmname $newuser $pw||add a user to guest|
|vmdestroy $vmname||destroy a VM (unregister and delete all files)|
|vmstart $vmname||boot a VM|
|vmshutdown $vmname||shut down by sending shutdown command via SSH|
if the VM is still running after 15 seconds, perform hard power off
|vmssh $vmname <$command>||SSH into VM as master, and optionally run a command|
|vmintnet $vmname $int $netname $ip $mask||add an interface and attach to internal network|
(example: vmintnet vm2 eth1 net6 192.168.6.2 255.255.255.0)
|vmsaveall||save the states of all running VMs|
execute this before rebooting host machine
|vmrestoreall||resume all saved VMs|
The quick sequence of creating a new VM is:
vmcreate vm25 UbuntuPrecise 22025
vmadduser vm25 user password
VirtualBox allows up to 8 network cards (NICs) per VM. VMAPI configures the first NIC in NAT + DHCP mode, so that you can SSH into the VM, and the VM can access Internet. The other 7 NICs are available for your experiments.
vmintnet command adds a NIC to a VM, and attaches it to an internal network. The internal network is identified by its name: each network name is a different network. Each internal network behaves like a Ethernet layer-2 switch interconnecting all NICs attached to it.
A network topology can be created among a set of VMs by creating a separate internal network for each link. For example, the following commands create an Internet2 topology:
vmcreate I2STTLng UbuntuPrecise 22121 vmcreate I2CHIC UbuntuPrecise 22122 vmcreate I2NEWY UbuntuPrecise 22123 vmcreate I2WASH UbuntuPrecise 22124 vmcreate I2ATLA UbuntuPrecise 22125 vmcreate I2HOUS UbuntuPrecise 22126 vmcreate I2LOSA UbuntuPrecise 22127 vmcreate I2SALT UbuntuPrecise 22128 vmcreate I2KANS UbuntuPrecise 22129 # STTLng-SALT vmintnet I2STTLng eth1 I2l0 10.21.69.0 255.255.255.254 vmintnet I2SALT eth1 I2l0 10.21.69.1 255.255.255.254 # STTLng-LOSA vmintnet I2STTLng eth2 I2l1 10.21.69.2 255.255.255.254 vmintnet I2LOSA eth1 I2l1 10.21.69.3 255.255.255.254 # NEWY-WASH vmintnet I2NEWY eth1 I2l2 10.21.69.4 255.255.255.254 vmintnet I2WASH eth1 I2l2 10.21.69.5 255.255.255.254 # CHIC-KANS vmintnet I2CHIC eth1 I2l3 10.21.69.6 255.255.255.254 vmintnet I2KANS eth1 I2l3 10.21.69.7 255.255.255.254 # CHIC-NEWY vmintnet I2CHIC eth2 I2l4 10.21.69.8 255.255.255.254 vmintnet I2NEWY eth2 I2l4 10.21.69.9 255.255.255.254 # CHIC-ATLA vmintnet I2CHIC eth3 I2l5 10.21.69.10 255.255.255.254 vmintnet I2ATLA eth1 I2l5 10.21.69.11 255.255.255.254 # CHIC-WASH vmintnet I2CHIC eth4 I2l6 10.21.69.12 255.255.255.254 vmintnet I2WASH eth2 I2l6 10.21.69.13 255.255.255.254 # ATLA-HOUS vmintnet I2ATLA eth2 I2l7 10.21.69.14 255.255.255.254 vmintnet I2HOUS eth1 I2l7 10.21.69.15 255.255.255.254 # WASH-ATLA vmintnet I2WASH eth3 I2l8 10.21.69.16 255.255.255.254 vmintnet I2ATLA eth3 I2l8 10.21.69.17 255.255.255.254 # LOSA-SALT vmintnet I2LOSA eth2 I2l9 10.21.69.18 255.255.255.254 vmintnet I2SALT eth2 I2l9 10.21.69.19 255.255.255.254 # HOUS-KANS vmintnet I2HOUS eth2 I2l10 10.21.69.20 255.255.255.254 vmintnet I2KANS eth2 I2l10 10.21.69.21 255.255.255.254 # HOUS-LOSA vmintnet I2HOUS eth3 I2l11 10.21.69.22 255.255.255.254 vmintnet I2LOSA eth3 I2l11 10.21.69.23 255.255.255.254 # SALT-KANS vmintnet I2SALT eth3 I2l12 10.21.69.24 255.255.255.254 vmintnet I2KANS eth3 I2l12 10.21.69.25 255.255.255.254 for vmname in I2STTLng I2CHIC I2NEWY I2WASH I2ATLA I2HOUS I2LOSA I2SALT I2KANS; do vmssh $vmname sudo shutdown -r now; done
This would probably take some time, because VM must be rebooted after adding each NIC. Otherwise, if multiple NICs are added before rebooting, their names in OS cannot be reliably determined.
VirtualBox also supports bridges networking which exposes a VM's NIC to the physical network, and host-only networking which connects a set of VMs to a TAP interface on the host machine. VMAPI does not directly support these modes, but you can use vmintnet to add the interface, and manually modify the configuration with VBoxManage modifyvm command.
Create and Update Guest OS Template
To create a guest OS template, there's more work than installing an OS, but this cost is amortized over hundreds of VMs based on this template.
- Use a separate runner account other than the one you use to run regular VMs.
- Download the installation CD of the guest OS.
- Create a VM, and start it.
VBoxManage createvm --name UbuntuPrecise --basefolder /data1/vbox0/vm/ --register VBoxManage modifyvm UbuntuPrecise --ostype Ubuntu_64 --memory 1024 --cpus 4 --acpi on --ioapic on --hwvirtex on --nestedpaging on --boot1 disk --boot2 dvd --boot3 none --boot4 none VBoxManage modifyvm UbuntuPrecise --nic1 nat --nictype1 virtio --natnet1 "192.168.254/24" --natpf1 ssh,tcp,,2222,,22 VBoxManage modifyvm UbuntuPrecise --audio none --clipboard disabled --usb off VBoxManage modifyvm UbuntuPrecise --vrde on --vrdeport 3389 --vrdeauthtype null VBoxManage createhd --filename /data1/vbox1/vm/UbuntuPrecise/main.vdi -size 12288 VBoxManage storagectl UbuntuPrecise --name SATA --add sata VBoxManage storageattach UbuntuPrecise --storagectl SATA --port 0 --type hdd --medium /data1/vbox0/vm/UbuntuPrecise/main.vdi VBoxManage storagectl UbuntuPrecise --name IDE --add ide VBoxManage storageattach UbuntuPrecise --storagectl IDE --port 1 --device 0 --type dvddrive --medium /data1/dvds/ubuntu-12.04-server-amd64.iso # insert OS installation CD VBoxManage startvm UbuntuPrecise --type headlessIn these command lines: '2222' is SSH host port, '3389' is VRDP host port; they must be above 1024 and not occupied by other VMs or services. '12288' is VM's disk space (MB); it's very difficult to increase disk space after guest OS template is created, so allocate sufficient space for your experiments here. You may tune other parameters as needed, especially for non-Ubuntu systems.
- On a machine with graphical interface, use a RDP client (Terminal Server Client, not VNC client) to connect to port 3389 of host machine. If firewall restricts this connection, you can specify another port number in the command line above, or use SSH tunnel.
- Complete OS installation over RDP session, and install an SSH server. Create a 'master' user (with that name) and grant sudo privilege with NOPASSWD option.
- Create a new SSH key pair:
ssh-keygen -t rsa -N '' -f /data1/vbox0/vm/UbuntuPrecise/id_rsaEven if you already have an SSH key pair for personal or project use, you should create a new pair for guest OS templates.
cat /data1/vbox0/vm/UbuntuPrecise/id_rsa.puband copy the public key on your clipboard.
- SSH into the VM:
ssh -p 2222 master@localhostthen enter the password chosen during OS installation. Inside the VM,
umask 077; mkdir .ssh; vi .ssh/authorized_keysand paste the SSH public key. Exit from SSH session.
- SSH into the VM with private key:
ssh -i /data1/vbox1/vm/UbuntuPrecise/id_rsa.vm -p 2222 master@localhost
- Inside the VM: update existing packages, and install necessary new packages, such as
sudo apt-get install virtualbox-guest-utils build-essential php5-cli python3 \VirtualBox guest additions (Ubuntu package virtual-guest-utils) should always be installed if available for the guest OS.
default-jdk ant curl traceroute tcpdump tshark libpcap-dev libexpat1-dev libssl-dev subversion git
- Download VMAPI guest scripts and copy the scripts suitable for the guest OS into /home/master/vmapi-guest/
- Make guest OS forget about Ethernet cards, so an Ethernet card with a different address is still named eth0.
sudo rm /etc/udev/rules.d/*net*Exit from SSH session.
- At host machine's runner account shell:
VBoxManage guestcontrol UbuntuPrecise exec --image '/bin/rm' --username master --password 123456 \(replace '123456' with master's password). This makes guest OS forget about entered commands, and turns off the VM.
--wait-stderr -- /home/master/.bash_history
VBoxManage guestcontrol UbuntuPrecise exec --image '/bin/sh' --username master --password 123456 \
--wait-stderr -- -c 'sudo shutdown -h now'
At this point, main.vdi (in /data1/vbox0/vm/UbuntuPrecise/) is the VDI for this new guest OS template, and id_rsa + id_rsa.pub is an SSH key pair that can manage the VM using master account. These files should be copied to /data1/vbox1/osimage/UbuntuPrecise/ so they become a guest OS template. If you have multiple runner accounts, you may copy them to a central location, and create symlinks in /data1/vbox1/osimage/UbuntuPrecise/
To upgrade a guest OS template, start the VM with
VBoxManage startvm UbuntuPrecise --type headless, SSH into the VM, and upgrade the packages. Perform the last two steps above, and copy main.vdi to osimage directory.
VMAPI Host Directory
The subdirectories and files in VMAPI host directory are:
|$R||VMAPI root directory|
|$R/osimage||guest OS templates|
|$R/osimage/$template||a guest OS template named $template|
|$R/osimage/$template/main.vdi||main disk image|
|$R/osimage/$template/*.vdi||optional additional disk image(s)|
|$R/osimage/$template/id_rsa||SSH private key for master account|
|$R/osimage/$template/id_rsa.pub||SSH public key for master account|
|$R/osimage/$template/nictype||optional virtual NIC type if not 'virtio'|
|$R/osimage/$template/create.sh||optional a script to run during VM creation, will receive $vmname as $1, can configure using VBoxManage modifyvm and/or attach additional disks|
|$R/vmapi-host||VMAPI host scripts|
|$R/vm/$vmname||a VM named $vmname; contains definition file and disk images of a running VM|
|$R/vmdata/$vmname||metadata for VM $vmname|
|$R/vmdata/$vmname/sshconfig||a ssh_config file used by |
|$R/vmdata/$vmname/users||list of users named added by |
|$R/vmdata/$vmname/sshport||SSH host port|
|$R/vmdata/hostport||host port mapping; reduces the possibility of port conflicts, but port conflicts can still happen among multiple runner accounts or with other services|
|$R/vmdata/hostport/$port||VM name having this host port|
VMAPI Guest Commands
These commands are exposed by a guest OS. New guest OS templates are expected to support these commands. These scripts are places in /home/master/vmapi-guest/ directory, and are executed over SSH under master account with sudo prepended. They are typically called by VMAPI host scripts, and are not used directly.
|hostname.sh $hostname||set the hostname|
|sshhostkey.sh||generate a new SSH host key|
|ip.sh clear||remove all IP address configuration, then add eth0 as DHCP|
|ip.sh $int $ip $mask|
(example: ip.sh eth1 192.168.5.2 255.255.255.0)
|assign IP address to a network interface. interfaces must be added in-order (eth0,eth1,eth2,eth3,eth4,eth5,eth6,eth7); reboot is required after adding each interface|
|adduser.sh $newuser $pw||add a new user, and add it to sudoers list|
|shutdown.sh||shutdown OS and power off|