Virtual Machines With VDE Networking
For a long time now I've used kvm, the hardware-accelerated alternative to QEMU, for running Intel x86-based VM (Virtual Machine) guests. After a lot of investigation I decided to use VDE2 (Virtual Distributed Ethernet) to provide the networking support. I chose it because it provided the closest to a regular network experience of any of the alternatives, especially since it is a full Ethernet emulation. I use dnsmasq to provide DNS and DHCP services to the virtual network.
There are many nuances to getting a perfect VDE configuration so it is time I documented the configuration I've used with Ubuntu Gutsy and Hardy.
For generic situations libvirt and virt-manager are useful tools to help manage VM clusters. ubuntu-vm-builder is useful for creating VMs and adding them to libvirt hosts.
VDE and libvirt Don't Play Together
Unfortunately libvirt doesn't currently support VDE networks, although it is possible for someone to implement a VDE interface using the libvirt network API.
Install Packages
These are the required packages:
sudo apt-get install vde2 dnsmasq kvm
Optional simple GUI VM management:
sudo apt-get install qemuctl
Optional libvirt support (currently no use with VDE, and limited if the VMs are started with lots of customised options):
sudo apt-get install libvirt0 libvirt-bin virt-manager ubuntu-vm-builder
Add User(s) to Groups
Add the current user to network and virtual machine groups:
sudo adduser $(id -un) kvm sudo adduser $(id -un) vde2-net sudo adduser $(id -un) tun
Optional libvirt support:
sudo adduser $(id -un) libvirtd
Notice: At this point the user should log-out and log-in to make these new group memberships active. Check them with:
groups tj adm disk dialout cdrom floppy audio dip video plugdev fuse lpadmin admin kvm sambashare vde2-net libvirtd tun
Give New Devices tun Group Ownership
This is a udev rule that is fired when a new tun/tap device is created:
cat <<EOF | sudo tee /etc/udev/rules.d/41-permissions-tun.rules # provide non-root user access to tun/tap devices for Virtual Machines using VDE etc KERNEL=="tun", GROUP="tun" EOF
Because the user is a member of the 'tun' group they can read and write to tun/tap devices. This makes it possible for VM guests to run with regular user permissions and still use VDE networking.
To reread the udev rules do:
sudo udevadm control --reload_rules
Define the VDE Network Interface
I chose to name the virtual network interface kvm0 but you might call it anything that doesn't conflict with other devices. This stanza will ensure the interface is set-up at system start-up (auto kvm0). The absolute values I've defined such as the IP address should be changed to suit:
cat <<EOF | sudo tee -a /etc/network/interfaces
# KVM/QEMU Virtual Machine Network
auto kvm0
iface kvm0 inet static
address 10.254.1.254
netmask 255.255.255.0
pre-up /usr/bin/vde_switch --tap ${IFACE} --daemon --group vde2-net --sock /var/run/${IFACE}.ctl \
--mod 775 --mgmtmode 770 --mgmt /var/run/${IFACE}-manage --pidfile /var/run/${IFACE}_vde.pid
up /usr/sbin/dnsmasq --interface=${IFACE} --except-interface=lo --bind-interfaces --user=nobody \
--dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h \
--domain=lan.tjworld.net --pid-file=/var/run/${IFACE}_dnsmasq.pid --conf-file
up echo "`route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'`" > /var/run/${IFACE}_route
up iptables -t nat -A POSTROUTING -o `cat /var/run/${IFACE}_route` -j MASQUERADE
down iptables -t nat -D POSTROUTING -o `cat /var/run/${IFACE}_route` -j MASQUERADE
down kill -s TERM `cat /var/run/${IFACE}_dnsmasq.pid` && rm -f /var/run/${IFACE}_dnsmasq.pid
post-down kill -s HUP `cat /var/run/${IFACE}_vde.pid`
post-down rm -f /var/run/${IFACE}_route
EOF
The chosen sub-net and interface IP address shouldn't be the same as the host or other sub-nets in use. In this case the physical LAN sub-net is 10.254.251.0/24 and there are remote VPNs on other /24 sub-nets all routable locally.
dnsmasq is given its settings directly to avoid it conflicting with any other instance running that is using the default /etc/dnsmasq.conf settings, which all instances of dnsmasq will read unless the --conf-file (-C) option is given.
In this case it only attaches and binds to the kvm0 interface, so it won't interfere with other dnsmasq instances or, for example, bind, if that is running on the host's primary interfaces.
on the kvm0 interface any VMs that connect and request DHCP leases will get an IP v4 address in the range 10.254.1.1 - 253. The gateway is the address given to kvm0 - 10.254.1.254.
iptables is used to create a netfilters Network Address Translation (NAT) rule that routes all outbound packets to the current default route interface. Because this can change between kvm0 starting and stopping, the current default-route interface is written to a text file in /var/run/kvm0_route and used later to correctly remove the netfilters rule.
To control the interface manually do:
# start the interface sudo ifup kvm0 # stop the interface sudo ifdown kvm0
When it is running it should show:
ifconfig kvm0
kvm0 Link encap:Ethernet HWaddr 00:ff:84:1d:b3:f0
inet addr:10.254.1.254 Bcast:10.254.1.255 Mask:255.255.255.0
inet6 addr: fe80::2ff:84ff:fe1d:b3f0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:53 (53.0 B) TX bytes:6478 (6.3 KB)
And the netfilters rule:
sudo iptables -t nat -nvL POSTROUTING
Chain POSTROUTING (policy ACCEPT 2233 packets, 205K bytes)
pkts bytes target prot opt in out source destination
3 193 MASQUERADE all -- * eth0 0.0.0.0/0 0.0.0.0/0
Enhanced kvm Package
If using the official Ubuntu kvm package it is important to know that it is not compiled with support for VDE. kvm-72 introduced internal support for VDE networking. I've built kvm-72 for Gutsy, Hardy and Intrepid with support for VDE (and the new audio device support). The packages are available from my Ubuntu Personal Package Archive.
Using these VDE-enabled packages avoids having to wrap the kvm start-up in a call to the vdeq utility, whose only purpose is to link VDE to the guest VM.
To install it do:
echo 'deb http://ppa.launchpad.net/intuitivenipple/ubuntu hardy main' | sudo tee /etc/apt/sources.list.d/intuitivenipple.list sudo apt-get update sudo apt-get install kvm
KVM Start-up Scripts
Every VM configuration is going to be slightly different so I'm not going to dwell on the individual options. I'll focus on the options that are important to networking with VDE.
I'll start with an example start-up script for Ubuntu Gutsy Server using my enhanced kvm-72:
#!/bin/bash qemuctl -qemu kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \ -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@
You'll notice it is using qemuctl to provide basic GUI control. That can be removed easily (just delete "qemuctl -qemu").
The same script for the pre-kvm-72 hypervisor simply wraps the kvm start-up in a call to vdeq. Here it is:
#!/bin/bash qemuctl -qemu vdeq kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \ -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@
You can see the only difference is that qemuctl is told that the qemu binary is called "vdeq" not "kvm". Without qemuctl the start-up would be:
#!/bin/bash vdeq kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \ -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@
Simple!
Networking Options
-net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0
Create a network interface card in the VM based on the RTL8139 with the MAC address 56:44:45:30:30:32 and associate with the internal kvm/qemu vlan 0.
-net vde,sock=/var/run/kvm0.ctl,vlan=0
Create an external network link using VDE to the control socket /var/run/kvm0.ctl and associate it with the internal kvm/qemu vlan 0.
These two options can be thought of as the two ends of the wire. NIC is the device on the back of the virtual PC, VDE is the plug on the wire into the switch.
Note: When defining MAC addresses ensure the first value is an even number. In Ethernet conventions odd numbers are reserved for switches and can cause some unexpected problems for other devices on the network although the VM using the odd-numbered MAC won't be affected.
The same options can be used for Microsoft Windows VM guests.
Guest Network Settings
If the VM guest is configured to use DHCP to automatically get its network settings there should be zero configuration of the networking in the guest operating system.
If using a static definition, the details are:
- IP address in the range 10.254.1.1 to 10.254.1.253 (ensure it doesn't conflict with what dnsmasq is issuing. Best way is to allocate from 10.254.1.150 and above for statics, and reduce the range of IPs in the dnsmasq pool to end at 10.254.1.149)
- Default Gateway is 10.254.1.254 (IP of kvm0)
- DNS server is 10.254.1.254 (IP of kvm0)
- Broadcast Address is 10.254.1.255
Using
Because this is a virtual Ethernet there are no restrictions caused by port-mapping or NAT. Restrictions can be placed in the netfilters rules of the host using iptables if desired. Because it is a proper Ethernet, networks of VMs can be tested and used to emulate production environments for testing. routing rules can be set using ip route to test advanced topologies in a totally virtual environment.
