wiki:Linux/DnsMasqAddressAlreadyInUse

Version 3 (modified by tj, 9 years ago) (diff)

--

DnsMasq reports Address Already In Use

This can be infuriating to diagnose:

$ sudo /etc/init.d/dnsmasq start
 * Starting DNS forwarder and DHCP server dnsmasq                                                             
dnsmasq: failed to create listening socket: Address already in use
 * failed

On the system Bind v9 named is in use. It is bound to an IP address range via the /etc/bind/named.conf.options setting:

options {
	directory "/var/cache/bind";
	auth-nxdomain no;    # conform to RFC1035
	listen-on { 10.254.251.0/24; };
	listen-on-v6 { any; };
};

This subnet attaches to either the wireless or wired (wlan0 or eth0) interfaces.

The system also has a virtual interface to support networking of multiple KVM virtual machines:

$ ifconfig kvm0
kvm0      Link encap:Ethernet  HWaddr 00:ff:91:b6:77:b4  
          inet addr:10.254.1.254  Bcast:10.254.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2ff:91ff:feb6:77b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:158 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:53 (53.0 B)  TX bytes:31423 (30.6 KB)

It is configured from /etc/network/interfaces:

# KVM/QEMU Virtual Machine Network
auto kvm0
iface kvm0 inet static
	address 10.254.1.254
	netmask 255.255.255.0
	pre-up /usr/bin/vde_switch --tap kvm0 --daemon --group vde2-net \
         --sock /var/run/kvm0.ctl --mod 775 --mgmtmode 770 --mgmt /var/run/kvm0-manage --pidfile /var/run/kvm0_vde.pid
	pre-up /etc/init.d/dnsmasq restart
	up iptables -t nat -A POSTROUTING -o `route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'` -j MASQUERADE
	down iptables -t nat -D POSTROUTING -o `route -n | sed -n 's/^0\.0\.0\.0 .* \(.*\)$/\1/p'` -j MASQUERADE
	post-down kill -s HUP `cat /var/run/vde_kvm0.pid`

DnsMasq is used to provide DNS and DHCP services to the virtual machines which are linked using VDE v2 (vde_switch). Its configuration was:

user=nobody
interface=kvm0
domain=lan.tjworld.net
dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h

When the interface tried to start DnsMasq reported "Address already in use". This seemed strange because it was configured to listen on the kvm0 interface (10.254.1.254), and Bind is only configured to listen on the 10.254.251.0/32 sub-net. Netstat confirmed this:

$ sudo netstat -tpln | egrep ':53 '
tcp        0      0 10.254.251.51:53        0.0.0.0:*               LISTEN      27514/named     
tcp6       0      0 fe80::219:d2ff:fe1a::53 :::*                    LISTEN      27514/named     
tcp6       0      0 fe80::2ff:91ff:feb6::53 :::*                    LISTEN      27514/named     
tcp6       0      0 ::1:53                  :::*                    LISTEN      27514/named  

I began to think DnsMasq was ignoring its configuration file. What infuriated me, and had me shouting at the PC "which %&*$ address is in use!?", was the fact that DnsMasq doesn't report the problem address.

After a lot of reading of the man-pages I discovered this little gem:

-z, --bind-interfaces

On systems which support it, dnsmasq binds the wildcard address, even when it is listening on only some interfaces. It then discards requests that it shouldn’t reply to. This has the advantage of working even when interfaces come and go and change address. This option forces dnsmasq to really bind only the interfaces it is listening on. About the only time when this is useful is when running another nameserver (or another instance of dnsmasq) on the same machine. Setting this option also enables multiple instances of dnsmasq which provide DHCP service to run in the same machine.


So, in summary, even if an interface is specifically declared to be used, DnsMasq still listens on all interfaces!


The (simple) solution is to add the bind-interfaces option to the configuration:

user=nobody
interface=kvm0
# only attach to the required interfaces, not to *:53
bind-interfaces
domain=lan.tjworld.net
dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h

There might still be a clash with Bind if Bind is listening on all (any) IP v6 addresses:

$ sudo /etc/init.d/dnsmasq start
 * Starting DNS forwarder and DHCP server dnsmasq                                                             
dnsmasq: failed to bind listening socket for fe80::2ff:91ff:feb6:77b4: Address already in use
 * failed

This is the kvm0 interface's IP v6 address. Bind needs to be told to listen on a limited range for IP v6 too:

options {
	// ...
	listen-on-v6 { ip6-localhost; };
};

After stopping both then restarting in the order

  1. Bind
  2. DnsMasq

they are finally co-existing happily:

~$ sudo netstat -tupln | grep ':53 '
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      1396/dnsmasq    
tcp        0      0 10.254.1.254:53         0.0.0.0:*               LISTEN      1396/dnsmasq    
tcp        0      0 10.254.251.51:53        0.0.0.0:*               LISTEN      1371/named      
tcp6       0      0 ::1:53                  :::*                    LISTEN      1396/dnsmasq    
tcp6       0      0 fe80::2ff:91ff:feb6::53 :::*                    LISTEN      1396/dnsmasq    
udp        0      0 127.0.0.1:53            0.0.0.0:*                           1396/dnsmasq    
udp        0      0 10.254.1.254:53         0.0.0.0:*                           1396/dnsmasq    
udp        0      0 10.254.251.51:53        0.0.0.0:*                           1371/named      
udp6       0      0 ::1:53                  :::*                                1396/dnsmasq    
udp6       0      0 fe80::2ff:91ff:feb6::53 :::*                                1396/dnsmasq    

One thing to notice is that DnsMasq is partially ignoring the bind-interfaces directive - in the list above it has still bound to localhost for IP v4 and IP v6! To stop this as well add the except-interface directive to DnsMasq's configuration:

user=nobody
interface=kvm0
# only attach to the required interfaces, not to *:53
bind-interfaces
# Even with that, needs to be explicitly stopped from binding to localhost
except-interface=lo
domain=lan.tjworld.net
dhcp-range=kvm,10.254.1.1,10.254.1.253,255.255.255.0,10.254.1.255,8h