DnsMasq reports Address Already In Use
This can be infuriating to diagnose:
$ sudo /etc/init.d/dnsmasq start * Starting DNS forwarder and DHCP server dnsmasq dnsmasq: failed to create listening socket: Address already in use * failed
On the system Bind v9 named is in use. It is bound to an IP address range via the /etc/bind/named.conf.options setting:
options {
directory "/var/cache/bind";
auth-nxdomain no; # conform to RFC1035
listen-on { 10.254.251.0/24; };
listen-on-v6 { any; };
};
This subnet attaches to either the wireless or wired (wlan0 or eth0) interfaces.
The system also has a virtual interface to support networking of multiple KVM virtual machines:
$ ifconfig kvm0
kvm0 Link encap:Ethernet HWaddr 00:ff:91:b6:77:b4
inet addr:10.254.1.254 Bcast:10.254.1.255 Mask:255.255.255.0
inet6 addr: fe80::2ff:91ff:feb6:77b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:158 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:53 (53.0 B) TX bytes:31423 (30.6 KB)
It is configured from /etc/network/interfaces:
# KVM/QEMU Virtual Machine Network
auto kvm0
iface kvm0 inet static
address 10.254.1.254
netmask 255.255.255.0
pre-up /usr/bin/vde_switch --tap kvm0 --daemon --group vde2-net \
--sock /var/run/kvm0.ctl --mod 775 --mgmtmode 770 --mgmt /var/run/kvm0-manage --pidfile /var/run/kvm0_vde.pid
pre-up /etc/init.d/dnsmasq restart
up iptables -t nat -A POSTROUTING -o `route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'` -j MASQUERADE
down iptables -t nat -D POSTROUTING -o `route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'` -j MASQUERADE
post-down kill -s HUP `cat /var/run/vde_kvm0.pid`
DnsMasq is used to provide DNS and DHCP services to the virtual machines which are linked using VDE v2 (vde_switch). Its configuration was:
user=nobody interface=kvm0 domain=lan.tjworld.net dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h
When the interface tried to start DnsMasq reported "Address already in use". This seemed strange because it was configured to listen on the kvm0 interface (10.254.1.254), and Bind is only configured to listen on the 10.254.251.0/32 sub-net. Netstat confirmed this:
$ sudo netstat -tpln | egrep ':53 ' tcp 0 0 10.254.251.51:53 0.0.0.0:* LISTEN 27514/named tcp6 0 0 fe80::219:d2ff:fe1a::53 :::* LISTEN 27514/named tcp6 0 0 fe80::2ff:91ff:feb6::53 :::* LISTEN 27514/named tcp6 0 0 ::1:53 :::* LISTEN 27514/named
I began to think DnsMasq was ignoring its configuration file. What infuriated me, and had me shouting at the PC "which %^&*$ address is in use!?", was the fact that DnsMasq doesn't report the problem address.
After a lot of reading of the man-pages I discovered this little gem:
-z, --bind-interfaces
On systems which support it, dnsmasq binds the wildcard address, even when it is listening on only some interfaces. It then discards requests that it shouldn’t reply to. This has the advantage of working even when interfaces come and go and change address. This option forces dnsmasq to really bind only the interfaces it is listening on. About the only time when this is useful is when running another nameserver (or another instance of dnsmasq) on the same machine. Setting this option also enables multiple instances of dnsmasq which provide DHCP service to run in the same machine.
So, in summary, even if an interface is specifically declared to be used, DnsMasq still listens on all interfaces!
The (simple) solution is to add the bind-interfaces option to the configuration:
user=nobody interface=kvm0 # only attach to the required interfaces, not to *:53 bind-interfaces domain=lan.tjworld.net dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h
There might still be a clash with Bind if Bind is listening on all (any) IP v6 addresses:
$ sudo /etc/init.d/dnsmasq start * Starting DNS forwarder and DHCP server dnsmasq dnsmasq: failed to bind listening socket for fe80::2ff:91ff:feb6:77b4: Address already in use * failed
This is the kvm0 interface's IP v6 address. Bind needs to be told to listen on a limited range for IP v6 too:
options {
// ...
listen-on-v6 { ip6-localhost; };
};
After stopping both then restarting in the order
- Bind
- DnsMasq
they are finally co-existing happily:
~$ sudo netstat -tupln | grep ':53 ' tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 1396/dnsmasq tcp 0 0 10.254.1.254:53 0.0.0.0:* LISTEN 1396/dnsmasq tcp 0 0 10.254.251.51:53 0.0.0.0:* LISTEN 1371/named tcp6 0 0 ::1:53 :::* LISTEN 1396/dnsmasq tcp6 0 0 fe80::2ff:91ff:feb6::53 :::* LISTEN 1396/dnsmasq udp 0 0 127.0.0.1:53 0.0.0.0:* 1396/dnsmasq udp 0 0 10.254.1.254:53 0.0.0.0:* 1396/dnsmasq udp 0 0 10.254.251.51:53 0.0.0.0:* 1371/named udp6 0 0 ::1:53 :::* 1396/dnsmasq udp6 0 0 fe80::2ff:91ff:feb6::53 :::* 1396/dnsmasq
One thing to notice is that DnsMasq is partially ignoring the bind-interfaces directive - in the list above it has still bound to localhost for IP v4 and IP v6! To stop this as well add the except-interface directive to DnsMasq's configuration:
user=nobody interface=kvm0 # only attach to the required interfaces, not to *:53 bind-interfaces # Even with that, needs to be explicitly stopped from binding to localhost except-interface=lo domain=lan.tjworld.net dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h
So the absolutely final (yeah right!) status is:
$ sudo /etc/init.d/dnsmasq restart * Restarting DNS forwarder and DHCP server dnsmasq [ OK ] $ sudo netstat -tupln | grep ':53 ' tcp 0 0 10.254.1.254:53 0.0.0.0:* LISTEN 3468/dnsmasq tcp 0 0 10.254.251.51:53 0.0.0.0:* LISTEN 1371/named tcp6 0 0 fe80::2ff:91ff:feb6::53 :::* LISTEN 3468/dnsmasq udp 0 0 10.254.1.254:53 0.0.0.0:* 3468/dnsmasq udp 0 0 10.254.251.51:53 0.0.0.0:* 1371/named udp6 0 0 fe80::2ff:91ff:feb6::53 :::* 3468/dnsmasq
