wiki:Linux/Ubuntu/VirtualMachinesWithVDENetworking

Virtual Machines With VDE Networking

For a long time now I've used  kvm, the hardware-accelerated alternative to  QEMU, for running Intel x86-based VM (Virtual Machine) guests. After a lot of investigation I decided to use  VDE2 (Virtual Distributed Ethernet) to provide the networking support. I chose it because it provided the closest to a regular network experience of any of the alternatives, especially since it is a full Ethernet emulation. I use  dnsmasq to provide DNS and DHCP services to the virtual network.

There are many nuances to getting a perfect VDE configuration so it is time I documented the configuration I've used with Ubuntu Gutsy and Hardy.

For generic situations  libvirt and  virt-manager are useful tools to help manage VM clusters.  ubuntu-vm-builder is useful for creating VMs and adding them to libvirt hosts.

VDE and libvirt Don't Play Together

Unfortunately libvirt doesn't currently support VDE networks, although it is possible for someone to implement a VDE interface using the libvirt network API.

Install Packages

These are the required packages:

sudo apt-get install vde2 dnsmasq kvm

Optional simple GUI VM management:

sudo apt-get install qemuctl

Optional libvirt support (currently no use with VDE, and limited if the VMs are started with lots of customised options):

sudo apt-get install libvirt0 libvirt-bin virt-manager ubuntu-vm-builder

Add User(s) to Groups

Add the current user to network and virtual machine groups:

sudo adduser $(id -un) kvm
sudo adduser $(id -un) vde2-net
sudo adduser $(id -un) tun

Optional libvirt support:

sudo adduser $(id -un) libvirtd

Notice: At this point the user should log-out and log-in to make these new group memberships active. Check them with:

groups
tj adm disk dialout cdrom floppy audio dip video plugdev fuse lpadmin admin kvm sambashare vde2-net libvirtd tun

Give New Devices tun Group Ownership

This is a udev rule that is fired when a new tun/tap device is created:

cat <<EOF | sudo tee /etc/udev/rules.d/41-permissions-tun.rules
# provide non-root user access to tun/tap devices for Virtual Machines using VDE etc
KERNEL=="tun", GROUP="tun"
EOF

Because the user is a member of the 'tun' group they can read and write to tun/tap devices. This makes it possible for VM guests to run with regular user permissions and still use VDE networking.

To reread the udev rules do:

sudo udevadm control --reload_rules

Define the VDE Network Interface

I chose to name the virtual network interface kvm0 but you might call it anything that doesn't conflict with other devices. This stanza will ensure the interface is set-up at system start-up (auto kvm0). The absolute values I've defined such as the IP address should be changed to suit:

cat <<EOF | sudo tee -a /etc/network/interfaces
# KVM/QEMU Virtual Machine Network
auto kvm0
iface kvm0 inet static
	address 10.254.1.254
	netmask 255.255.255.0
	pre-up /usr/bin/vde_switch --tap ${IFACE} --daemon --group vde2-net --sock /var/run/${IFACE}.ctl \
	 --mod 775 --mgmtmode 770 --mgmt /var/run/${IFACE}-manage --pidfile /var/run/${IFACE}_vde.pid
	up /usr/sbin/dnsmasq --interface=${IFACE}  --except-interface=lo --bind-interfaces --user=nobody \
	 --dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h \
	 --domain=lan.tjworld.net --pid-file=/var/run/${IFACE}_dnsmasq.pid --conf-file
	up echo "`route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'`" > /var/run/${IFACE}_route
	up iptables -t nat -A POSTROUTING -o `cat /var/run/${IFACE}_route` -j MASQUERADE
	down iptables -t nat -D POSTROUTING -o `cat /var/run/${IFACE}_route` -j MASQUERADE
	down kill -s TERM `cat /var/run/${IFACE}_dnsmasq.pid` && rm -f /var/run/${IFACE}_dnsmasq.pid
	post-down kill -s HUP `cat /var/run/${IFACE}_vde.pid`
	post-down rm -f /var/run/${IFACE}_route 
EOF

The chosen sub-net and interface IP address shouldn't be the same as the host or other sub-nets in use. In this case the physical LAN sub-net is 10.254.251.0/24 and there are remote VPNs on other /24 sub-nets all routable locally.

dnsmasq is given its settings directly to avoid it conflicting with any other instance running that is using the default /etc/dnsmasq.conf settings, which all instances of dnsmasq will read unless the --conf-file (-C) option is given.

In this case it only attaches and binds to the kvm0 interface, so it won't interfere with other dnsmasq instances or, for example, bind, if that is running on the host's primary interfaces.

on the kvm0 interface any VMs that connect and request DHCP leases will get an IP v4 address in the range 10.254.1.1 - 253. The gateway is the address given to kvm0 - 10.254.1.254.

iptables is used to create a netfilters Network Address Translation (NAT) rule that routes all outbound packets to the current default route interface. Because this can change between kvm0 starting and stopping, the current default-route interface is written to a text file in /var/run/kvm0_route and used later to correctly remove the netfilters rule.

To control the interface manually do:

# start the interface
sudo ifup kvm0

# stop the interface
sudo ifdown kvm0

When it is running it should show:

ifconfig kvm0

kvm0      Link encap:Ethernet  HWaddr 00:ff:84:1d:b3:f0  
          inet addr:10.254.1.254  Bcast:10.254.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2ff:84ff:fe1d:b3f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:53 (53.0 B)  TX bytes:6478 (6.3 KB)

And the netfilters rule:

sudo iptables -t nat -nvL POSTROUTING
Chain POSTROUTING (policy ACCEPT 2233 packets, 205K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    3   193 MASQUERADE  all  --  *      eth0    0.0.0.0/0            0.0.0.0/0        

Enhanced kvm Package

If using the official Ubuntu kvm package it is important to know that it is not compiled with support for VDE. kvm-72 introduced internal support for VDE networking. I've built kvm-72 for Gutsy, Hardy and Intrepid with support for VDE (and the new audio device support). The packages are available from  my Ubuntu Personal Package Archive.

Using these VDE-enabled packages avoids having to wrap the kvm start-up in a call to the vdeq utility, whose only purpose is to link VDE to the guest VM.

To install it do:

echo 'deb http://ppa.launchpad.net/intuitivenipple/ubuntu hardy main' | sudo tee /etc/apt/sources.list.d/intuitivenipple.list
sudo apt-get update
sudo apt-get install kvm

KVM Start-up Scripts

Every VM configuration is going to be slightly different so I'm not going to dwell on the individual options. I'll focus on the options that are important to networking with VDE.

I'll start with an example start-up script for Ubuntu Gutsy Server using my enhanced kvm-72:

#!/bin/bash
qemuctl -qemu kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \
 -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@

You'll notice it is using qemuctl to provide basic GUI control. That can be removed easily (just delete "qemuctl -qemu").

The same script for the pre-kvm-72 hypervisor simply wraps the kvm start-up in a call to vdeq. Here it is:

#!/bin/bash
qemuctl -qemu vdeq kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \
 -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@

You can see the only difference is that qemuctl is told that the qemu binary is called "vdeq" not "kvm". Without qemuctl the start-up would be:

#!/bin/bash
vdeq kvm -name Gutsy-Server -boot d -m 450 -hda /home/all/VirtualMachines/Ubuntu-Gutsy-Server-x86.qcow2 -k en-gb \
 -net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0 -net vde,sock=/var/run/kvm0.ctl,vlan=0 $@

Simple!

Networking Options

-net nic,model=rtl8139,macaddr=56:44:45:30:30:32,vlan=0

Create a network interface card in the VM based on the RTL8139 with the MAC address 56:44:45:30:30:32 and associate with the internal kvm/qemu vlan 0.

-net vde,sock=/var/run/kvm0.ctl,vlan=0

Create an external network link using VDE to the control socket /var/run/kvm0.ctl and associate it with the internal kvm/qemu vlan 0.

These two options can be thought of as the two ends of the wire. NIC is the device on the back of the virtual PC, VDE is the plug on the wire into the switch.

Note: When defining MAC addresses ensure the first value is an even number. In Ethernet conventions odd numbers are reserved for switches and can cause some unexpected problems for other devices on the network although the VM using the odd-numbered MAC won't be affected.

The same options can be used for Microsoft Windows VM guests.

Guest Network Settings

If the VM guest is configured to use DHCP to automatically get its network settings there should be zero configuration of the networking in the guest operating system.

If using a static definition, the details are:

  • IP address in the range 10.254.1.1 to 10.254.1.253 (ensure it doesn't conflict with what dnsmasq is issuing. Best way is to allocate from 10.254.1.150 and above for statics, and reduce the range of IPs in the dnsmasq pool to end at 10.254.1.149)
  • Default Gateway is 10.254.1.254 (IP of kvm0)
  • DNS server is 10.254.1.254 (IP of kvm0)
  • Broadcast Address is 10.254.1.255

Using

Because this is a virtual Ethernet there are no restrictions caused by port-mapping or NAT. Restrictions can be placed in the netfilters rules of the host using iptables if desired. Because it is a proper Ethernet, networks of VMs can be tested and used to emulate production environments for testing. routing rules can be set using ip route to test advanced topologies in a totally virtual environment.