Discussion:
linux bridge does not forward arp reply back packets in a vmware vm
(too old to reply)
Adrian P
2017-12-15 13:37:39 UTC
Permalink
Hello,

I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.

From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.

Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:

# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface

The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.

The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.

The problem is the following:

With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.

To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.

I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.

Any thoughts?

Many thanks in advance.

Best regards,
Adrian
Stephen Hemminger
2017-12-15 15:55:12 UTC
Permalink
On Fri, 15 Dec 2017 15:37:39 +0200
Post by Adrian P
Hello,
I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.
From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.
# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface
The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.
The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.
With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.
To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.
I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.
Any thoughts?
Many thanks in advance.
Best regards,
Adrian
Does each tap instance and the ens160 have a different and valid Ethernet
address? Also make sure the these are in the bridge forwarding table.
Adrian P
2017-12-15 16:29:58 UTC
Permalink
On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 15:37:39 +0200
Post by Adrian P
Hello,
I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.
From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.
# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface
The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.
The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.
With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.
To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.
I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.
Any thoughts?
Many thanks in advance.
Best regards,
Adrian
Does each tap instance and the ens160 have a different and valid Ethernet
address? Also make sure the these are in the bridge forwarding table.
Yes, they have valid Ethernet addresses, and they do show up in the
forwarding table twice, see below:

# ip addr
<...>
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
brq025a9a94-58 state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:fea6:b378/64 scope link
valid_lft forever preferred_lft forever
4: tap2eb4cad6-***@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue master brq025a9a94-58 state UP qlen 1000
link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
valid_lft forever preferred_lft forever
inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
valid_lft forever preferred_lft forever
6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe9a:495/64 scope link
valid_lft forever preferred_lft forever

# brctl showmacs brq025a9a94-58
port no mac addr is local? ageing timer
1 00:50:56:a6:b3:78 yes 0.00
1 00:50:56:a6:b3:78 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
Stephen Hemminger
2017-12-16 01:47:09 UTC
Permalink
On Fri, 15 Dec 2017 18:29:58 +0200
Post by Adrian P
On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 15:37:39 +0200
Post by Adrian P
Hello,
I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.
From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.
# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface
The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.
The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.
With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.
To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.
I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.
Any thoughts?
Many thanks in advance.
Best regards,
Adrian
Does each tap instance and the ens160 have a different and valid Ethernet
address? Also make sure the these are in the bridge forwarding table.
Yes, they have valid Ethernet addresses, and they do show up in the
# ip addr
<...>
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
brq025a9a94-58 state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:fea6:b378/64 scope link
valid_lft forever preferred_lft forever
qdisc noqueue master brq025a9a94-58 state UP qlen 1000
link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
valid_lft forever preferred_lft forever
inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
valid_lft forever preferred_lft forever
6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe9a:495/64 scope link
valid_lft forever preferred_lft forever
# brctl showmacs brq025a9a94-58
port no mac addr is local? ageing timer
1 00:50:56:a6:b3:78 yes 0.00
1 00:50:56:a6:b3:78 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
Since there are multiple entries per port maybe you are also using VLANs?
Adrian P
2017-12-16 07:12:13 UTC
Permalink
On Sat, Dec 16, 2017 at 3:47 AM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 18:29:58 +0200
Post by Adrian P
On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 15:37:39 +0200
Post by Adrian P
Hello,
I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.
From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.
# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface
The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.
The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.
With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.
To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.
I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.
Any thoughts?
Many thanks in advance.
Best regards,
Adrian
Does each tap instance and the ens160 have a different and valid Ethernet
address? Also make sure the these are in the bridge forwarding table.
Yes, they have valid Ethernet addresses, and they do show up in the
# ip addr
<...>
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
brq025a9a94-58 state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:fea6:b378/64 scope link
valid_lft forever preferred_lft forever
qdisc noqueue master brq025a9a94-58 state UP qlen 1000
link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
valid_lft forever preferred_lft forever
inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
valid_lft forever preferred_lft forever
6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe9a:495/64 scope link
valid_lft forever preferred_lft forever
# brctl showmacs brq025a9a94-58
port no mac addr is local? ageing timer
1 00:50:56:a6:b3:78 yes 0.00
1 00:50:56:a6:b3:78 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
Since there are multiple entries per port maybe you are also using VLANs?
So I have checked one more time: the host is a vmware centos7 VM, with
a single interface (ens160 that can be seen above) connected to a
vmware virtual switch port group configured in promiscuous mode, that
removes the VLAN tagging (similar with access mode port in a physical
switch).

I have another environment that behaves the same, where the host is a
physical server, and the interface is a bonded interface, with two
slaves, and I also have two entries for each local interface in the
forwarding table:

# ip addr
<...>
6: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 01000000000000000000004c4930394833 state UP qlen
1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
7: eno6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 02000000000000000000004c4930394833 state UP qlen
1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
<...>
10: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue master brq75a55ef7-4a state UP qlen 1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fe15:b4ff:fe13:e6a3/64 scope link
valid_lft forever preferred_lft forever
<...>
14: brq75a55ef7-4a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.55/24 brd 10.20.21.255 scope global brq75a55ef7-4a
valid_lft forever preferred_lft forever
15: tap44bc34bb-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq75a55ef7-4a state UNKNOWN qlen 1000
link/ether fe:16:3e:cc:dc:ec brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fecc:dcec/64 scope link
valid_lft forever preferred_lft forever

# brctl show
bridge name bridge id STP enabled interfaces
brq75a55ef7-4a 8000.fc15b413e6a3 no bond2
tap44bc34bb-e2

# brctl showmacs brq75a55ef7-4a | grep yes
2 fc:15:b4:13:e6:a3 yes 0.00
2 fc:15:b4:13:e6:a3 yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
Adrian P
2017-12-16 14:19:03 UTC
Permalink
Investigating this further, I have noticed that the mac address of the
eth0 interface from the cirros VM instance does not appear in the
bridge forwarding table, and this explains why everything starts
working only when I set ageing time to 0, because in that case all
packets are flooded on all ports and the bridge behaves like a hub.

So now the question is: why the bridge does not learn the mac address
of the eth0 interface from the cirros VM instance? I am able to see
the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
going out from the cirros VM instance on tap interface, so the bridge
should learn the mac address and add it to the forwarding table.

The reply back to the arp request (Reply 10.20.21.1 is-at
00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
now I know why: there is no mac address in the forwarding table, so
the bridge does not know on which port to send the arp reply back.

This happens with tap interfaces only. I can see many mac addresses
associated with "physical" interface ens160 (that is interface number
1) in the forwarding table, but in case of the tap interfaces, there
are only two entries, and both entries shows the mac address of the
tap interfaces only:

# brctl showmacs brq025a9a94-58
port no mac addr is local? ageing timer
<...>
1 00:50:56:a6:f8:ba no 13.66
1 00:60:16:6d:b3:f8 no 85.01
1 00:60:16:6d:dd:b8 no 239.39
2 52:db:7c:ac:4b:25 yes 0.00
2 52:db:7c:ac:4b:25 yes 0.00
<...>
1 fc:15:b4:13:b9:f0 no 0.21
1 fc:15:b4:13:b9:f1 no 19.40
1 fc:15:b4:13:b9:f5 no 17.40
3 fe:16:3e:9a:04:95 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
Post by Adrian P
On Sat, Dec 16, 2017 at 3:47 AM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 18:29:58 +0200
Post by Adrian P
On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
Post by Stephen Hemminger
On Fri, 15 Dec 2017 15:37:39 +0200
Post by Adrian P
Hello,
I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.
From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.
# brctl show
bridge name bridge id STP enabled interfaces
brq025a9a94-58 8000.005056a6b378 no ens160
tap2eb4cad6-cd
<----- neutron DHCP agent tap interface
tap6d31a191-9f
<----- cirros VM instance tap interface
The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.
The tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.
With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.
To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.
I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.
Any thoughts?
Many thanks in advance.
Best regards,
Adrian
Does each tap instance and the ens160 have a different and valid Ethernet
address? Also make sure the these are in the bridge forwarding table.
Yes, they have valid Ethernet addresses, and they do show up in the
# ip addr
<...>
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
brq025a9a94-58 state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:fea6:b378/64 scope link
valid_lft forever preferred_lft forever
qdisc noqueue master brq025a9a94-58 state UP qlen 1000
link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
valid_lft forever preferred_lft forever
inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
valid_lft forever preferred_lft forever
6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe9a:495/64 scope link
valid_lft forever preferred_lft forever
# brctl showmacs brq025a9a94-58
port no mac addr is local? ageing timer
1 00:50:56:a6:b3:78 yes 0.00
1 00:50:56:a6:b3:78 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
2 8a:b2:15:4c:96:55 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
3 fe:16:3e:9a:04:95 yes 0.00
Since there are multiple entries per port maybe you are also using VLANs?
So I have checked one more time: the host is a vmware centos7 VM, with
a single interface (ens160 that can be seen above) connected to a
vmware virtual switch port group configured in promiscuous mode, that
removes the VLAN tagging (similar with access mode port in a physical
switch).
I have another environment that behaves the same, where the host is a
physical server, and the interface is a bonded interface, with two
slaves, and I also have two entries for each local interface in the
# ip addr
<...>
6: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 01000000000000000000004c4930394833 state UP qlen
1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
7: eno6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 02000000000000000000004c4930394833 state UP qlen
1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
<...>
10: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue master brq75a55ef7-4a state UP qlen 1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fe15:b4ff:fe13:e6a3/64 scope link
valid_lft forever preferred_lft forever
<...>
14: brq75a55ef7-4a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
inet 10.20.21.55/24 brd 10.20.21.255 scope global brq75a55ef7-4a
valid_lft forever preferred_lft forever
15: tap44bc34bb-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq75a55ef7-4a state UNKNOWN qlen 1000
link/ether fe:16:3e:cc:dc:ec brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fecc:dcec/64 scope link
valid_lft forever preferred_lft forever
# brctl show
bridge name bridge id STP enabled interfaces
brq75a55ef7-4a 8000.fc15b413e6a3 no bond2
tap44bc34bb-e2
# brctl showmacs brq75a55ef7-4a | grep yes
2 fc:15:b4:13:e6:a3 yes 0.00
2 fc:15:b4:13:e6:a3 yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
Stephen Hemminger
2017-12-16 17:35:20 UTC
Permalink
On Sat, 16 Dec 2017 16:19:03 +0200
Post by Adrian P
Investigating this further, I have noticed that the mac address of the
eth0 interface from the cirros VM instance does not appear in the
bridge forwarding table, and this explains why everything starts
working only when I set ageing time to 0, because in that case all
packets are flooded on all ports and the bridge behaves like a hub.
So now the question is: why the bridge does not learn the mac address
of the eth0 interface from the cirros VM instance? I am able to see
the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
going out from the cirros VM instance on tap interface, so the bridge
should learn the mac address and add it to the forwarding table.
The reply back to the arp request (Reply 10.20.21.1 is-at
00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
now I know why: there is no mac address in the forwarding table, so
the bridge does not know on which port to send the arp reply back.
This happens with tap interfaces only. I can see many mac addresses
associated with "physical" interface ens160 (that is interface number
1) in the forwarding table, but in case of the tap interfaces, there
are only two entries, and both entries shows the mac address of the
VMWare does ARP spoofing maybe it consumes the ARP
Adrian P
2017-12-16 18:14:19 UTC
Permalink
On Sat, Dec 16, 2017 at 7:35 PM, Stephen Hemminger
Post by Stephen Hemminger
On Sat, 16 Dec 2017 16:19:03 +0200
Post by Adrian P
Investigating this further, I have noticed that the mac address of the
eth0 interface from the cirros VM instance does not appear in the
bridge forwarding table, and this explains why everything starts
working only when I set ageing time to 0, because in that case all
packets are flooded on all ports and the bridge behaves like a hub.
So now the question is: why the bridge does not learn the mac address
of the eth0 interface from the cirros VM instance? I am able to see
the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
going out from the cirros VM instance on tap interface, so the bridge
should learn the mac address and add it to the forwarding table.
The reply back to the arp request (Reply 10.20.21.1 is-at
00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
now I know why: there is no mac address in the forwarding table, so
the bridge does not know on which port to send the arp reply back.
This happens with tap interfaces only. I can see many mac addresses
associated with "physical" interface ens160 (that is interface number
1) in the forwarding table, but in case of the tap interfaces, there
are only two entries, and both entries shows the mac address of the
VMWare does ARP spoofing maybe it consumes the ARP
The flow is like this:

vmware <--> ens160 <--> bridge <--> tap if <--> eth0

So my understanding is that when the arp request goes out on the eth0,
through the tap interface, the bridge should learn the eth0 mac
address and add it to the forwarding table. But this does not happen,
so when the arp reply back comes back, the bridge does not know on
which interface to send the arp reply, and this is why nothing works.

I have another bare metal environment, without vmware, and I still
have this issue.

I suspect this is a bug related to the bridge and the tap interface,
since I see many mac address learned by the bridge on the ens160
interface... or maybe some tap / bridge setting? I have no clue. Where
should I report such issue?
Adrian P
2017-12-16 20:01:11 UTC
Permalink
Post by Adrian P
On Sat, Dec 16, 2017 at 7:35 PM, Stephen Hemminger
Post by Stephen Hemminger
On Sat, 16 Dec 2017 16:19:03 +0200
Post by Adrian P
Investigating this further, I have noticed that the mac address of the
eth0 interface from the cirros VM instance does not appear in the
bridge forwarding table, and this explains why everything starts
working only when I set ageing time to 0, because in that case all
packets are flooded on all ports and the bridge behaves like a hub.
So now the question is: why the bridge does not learn the mac address
of the eth0 interface from the cirros VM instance? I am able to see
the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
going out from the cirros VM instance on tap interface, so the bridge
should learn the mac address and add it to the forwarding table.
The reply back to the arp request (Reply 10.20.21.1 is-at
00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
now I know why: there is no mac address in the forwarding table, so
the bridge does not know on which port to send the arp reply back.
This happens with tap interfaces only. I can see many mac addresses
associated with "physical" interface ens160 (that is interface number
1) in the forwarding table, but in case of the tap interfaces, there
are only two entries, and both entries shows the mac address of the
VMWare does ARP spoofing maybe it consumes the ARP
vmware <--> ens160 <--> bridge <--> tap if <--> eth0
So my understanding is that when the arp request goes out on the eth0,
through the tap interface, the bridge should learn the eth0 mac
address and add it to the forwarding table. But this does not happen,
so when the arp reply back comes back, the bridge does not know on
which interface to send the arp reply, and this is why nothing works.
I have another bare metal environment, without vmware, and I still
have this issue.
I suspect this is a bug related to the bridge and the tap interface,
since I see many mac address learned by the bridge on the ens160
interface... or maybe some tap / bridge setting? I have no clue. Where
should I report such issue?
Further investigation reveals something strange: when the
communication starts with an arp request (which happens almost all the
time), the bridge wrongly assigns the eth0 mac address to port 1,
instead of port 3.

Flow again:

default gw --- vmware --- [ ens160 bridge tap ] --- eth0

On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
mac address is fa:16:3e:9a:04:95

What I have found is that in the forwarding table, the bridge wrongly
assigns the eth0 mac address to the port 1, which is ens160 interface,
instead of assigning it to the port 3, which is the tap interface.
This happens only if the arp table in the cirros VM instance does not
contain the mac address of the destination I am pinging (default gw in
this case), so the cirros VM sends an arp request. See below the eth0
mac address wrongly assigned in the forwarding table to the port 1:

# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
1 fa:16:3e:9a:04:95 no 0.67

However, if I manually add the mac address of the destination IP I am
pining into the cirros VM instance arp table, and there is no arp
request sent, just icmp packets going out, then the bridge correctly
assigns the eth0 mac address to the port 3, which is the tap
interface, and everything starts working fine. See below the eth0 mac
address correctly assigned in the forwarding table to the port 3:

# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
3 fa:16:3e:9a:04:95 no 9.26
Toshiaki Makita
2017-12-18 02:54:20 UTC
Permalink
On 2017/12/17 5:01, Adrian P wrote:
...
Post by Adrian P
Further investigation reveals something strange: when the
communication starts with an arp request (which happens almost all the
time), the bridge wrongly assigns the eth0 mac address to port 1,
instead of port 3.
default gw --- vmware --- [ ens160 bridge tap ] --- eth0
On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
mac address is fa:16:3e:9a:04:95
What I have found is that in the forwarding table, the bridge wrongly
assigns the eth0 mac address to the port 1, which is ens160 interface,
instead of assigning it to the port 3, which is the tap interface.
This happens only if the arp table in the cirros VM instance does not
contain the mac address of the destination I am pinging (default gw in
this case), so the cirros VM sends an arp request. See below the eth0
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
1 fa:16:3e:9a:04:95 no 0.67
However, if I manually add the mac address of the destination IP I am
pining into the cirros VM instance arp table, and there is no arp
request sent, just icmp packets going out, then the bridge correctly
assigns the eth0 mac address to the port 3, which is the tap
interface, and everything starts working fine. See below the eth0 mac
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
3 fa:16:3e:9a:04:95 no 9.26
It looks like the broadcast arp request is coming back from gw or
vmware. Would you check it?
--
Toshiaki Makita
Adrian Pascalau
2017-12-18 08:05:47 UTC
Permalink
On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
Post by Toshiaki Makita
...
Post by Adrian P
Further investigation reveals something strange: when the
communication starts with an arp request (which happens almost all the
time), the bridge wrongly assigns the eth0 mac address to port 1,
instead of port 3.
default gw --- vmware --- [ ens160 bridge tap ] --- eth0
On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
mac address is fa:16:3e:9a:04:95
What I have found is that in the forwarding table, the bridge wrongly
assigns the eth0 mac address to the port 1, which is ens160 interface,
instead of assigning it to the port 3, which is the tap interface.
This happens only if the arp table in the cirros VM instance does not
contain the mac address of the destination I am pinging (default gw in
this case), so the cirros VM sends an arp request. See below the eth0
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
1 fa:16:3e:9a:04:95 no 0.67
However, if I manually add the mac address of the destination IP I am
pining into the cirros VM instance arp table, and there is no arp
request sent, just icmp packets going out, then the bridge correctly
assigns the eth0 mac address to the port 3, which is the tap
interface, and everything starts working fine. See below the eth0 mac
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
3 fa:16:3e:9a:04:95 no 9.26
It looks like the broadcast arp request is coming back from gw or
vmware. Would you check it?
So if I look into the flow below, once more:

default gw --- physical switch --- vmware --- [ ens160 bridge tap ] --- eth0

The arp request goes out on the eth0 interface, and enters the bridge
on the tap interface. The bridge assigns the eth0 mac address to the
tap interface, and sends the arp request out on the ens160 interface.
Now some deice on the left side of the bridge (either de vmware,
physical switch or the default gw), broadcasts that arp requests back,
therefore the same arp request enters back the bridge on the ens160
interface, and the bridge assigns the source mac address of the arp
request (which is still the eth0 mac address) to the ens160 port in
the forwarding table, which causes the behavior I have noticed...

This explains why I see multiple times the arp request when using
tcpdump. By why there are three? Maybe two devices broadcasts back the
arp request?

# tcpdump -n -i ens160 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46

What is strange in the above tcpdump is that the first arp request is
28 bytes in length (I suppose this is the one that enters the bridge
on the tap interface), and the 2 others is 46 bytes in length (I
suppose these are the ones that enters the bridge on the ens160
interface). If I open the capture in wireshark, I see some padding
added in the last two arp requests, which does not appear in the first
arp request:

Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst:
Broadcast (ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
Type: ARP (0x0806)
Padding: 000000000000000000000000000000000000

Any thoughts how I can find out who duplicates the two additional arps?
Adrian Pascalau
2017-12-19 19:18:38 UTC
Permalink
On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
Post by Adrian Pascalau
On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
Post by Toshiaki Makita
...
Post by Adrian P
Further investigation reveals something strange: when the
communication starts with an arp request (which happens almost all the
time), the bridge wrongly assigns the eth0 mac address to port 1,
instead of port 3.
default gw --- vmware --- [ ens160 bridge tap ] --- eth0
On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
mac address is fa:16:3e:9a:04:95
What I have found is that in the forwarding table, the bridge wrongly
assigns the eth0 mac address to the port 1, which is ens160 interface,
instead of assigning it to the port 3, which is the tap interface.
This happens only if the arp table in the cirros VM instance does not
contain the mac address of the destination I am pinging (default gw in
this case), so the cirros VM sends an arp request. See below the eth0
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
1 fa:16:3e:9a:04:95 no 0.67
However, if I manually add the mac address of the destination IP I am
pining into the cirros VM instance arp table, and there is no arp
request sent, just icmp packets going out, then the bridge correctly
assigns the eth0 mac address to the port 3, which is the tap
interface, and everything starts working fine. See below the eth0 mac
# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
3 fa:16:3e:9a:04:95 no 9.26
It looks like the broadcast arp request is coming back from gw or
vmware. Would you check it?
default gw --- physical switch --- vmware --- [ ens160 bridge tap ] --- eth0
The arp request goes out on the eth0 interface, and enters the bridge
on the tap interface. The bridge assigns the eth0 mac address to the
tap interface, and sends the arp request out on the ens160 interface.
Now some deice on the left side of the bridge (either de vmware,
physical switch or the default gw), broadcasts that arp requests back,
therefore the same arp request enters back the bridge on the ens160
interface, and the bridge assigns the source mac address of the arp
request (which is still the eth0 mac address) to the ens160 port in
the forwarding table, which causes the behavior I have noticed...
This explains why I see multiple times the arp request when using
tcpdump. By why there are three? Maybe two devices broadcasts back the
arp request?
# tcpdump -n -i ens160 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46
What is strange in the above tcpdump is that the first arp request is
28 bytes in length (I suppose this is the one that enters the bridge
on the tap interface), and the 2 others is 46 bytes in length (I
suppose these are the ones that enters the bridge on the ens160
interface). If I open the capture in wireshark, I see some padding
added in the last two arp requests, which does not appear in the first
Broadcast (ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
Type: ARP (0x0806)
Padding: 000000000000000000000000000000000000
Any thoughts how I can find out who duplicates the two additional arps?
Is there a way to trace the packet as it passes through the bridge?
Something like a debug mode?
Toshiaki Makita
2017-12-20 05:04:27 UTC
Permalink
Post by Adrian Pascalau
On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
Post by Adrian Pascalau
On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
Post by Toshiaki Makita
It looks like the broadcast arp request is coming back from gw or
vmware. Would you check it?
default gw --- physical switch --- vmware --- [ ens160 bridge tap ] --- eth0
The arp request goes out on the eth0 interface, and enters the bridge
on the tap interface. The bridge assigns the eth0 mac address to the
tap interface, and sends the arp request out on the ens160 interface.
Now some deice on the left side of the bridge (either de vmware,
physical switch or the default gw), broadcasts that arp requests back,
therefore the same arp request enters back the bridge on the ens160
interface, and the bridge assigns the source mac address of the arp
request (which is still the eth0 mac address) to the ens160 port in
the forwarding table, which causes the behavior I have noticed...
This explains why I see multiple times the arp request when using
tcpdump. By why there are three? Maybe two devices broadcasts back the
arp request?
# tcpdump -n -i ens160 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46
What is strange in the above tcpdump is that the first arp request is
28 bytes in length (I suppose this is the one that enters the bridge
on the tap interface), and the 2 others is 46 bytes in length (I
suppose these are the ones that enters the bridge on the ens160
interface). If I open the capture in wireshark, I see some padding
added in the last two arp requests, which does not appear in the first
Broadcast (ff:ff:ff:ff:ff:ff)
Destination: Broadcast (ff:ff:ff:ff:ff:ff)
Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
Type: ARP (0x0806)
Padding: 000000000000000000000000000000000000
Any thoughts how I can find out who duplicates the two additional arps?
Is there a way to trace the packet as it passes through the bridge?
Something like a debug mode?
FWIW you can use "bridge monitor" command to monitor fdb update.
tcpdump has --direction option so you can only capture either of "out"
packets or "in" packets.
--
Toshiaki Makita
Continue reading on narkive:
Loading...