[Bridge] Problem with bridge (mcast-to-ucast + hairpin) and Broadcom's 802.11f in their FullMAC fw

[Stephen Hemminger]

On Tue, 27 Feb 2018 11:08:20 +0100

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.
1) IFLA_BRPORT_MCAST_TO_UCAST
2) Clients isolation in hostapd
3) Hairpin mode enabled
https://patchwork.kernel.org/patch/9530669/
https://lwn.net/Articles/347344/
Short version: in that setup packets received from a bridged wireless
interface can be handled back to it for transmission.
Now, Broadcom's firmware for their FullMAC chipsets in AP mode
supports an obsoleted 802.11f AKA IAPP standard. It's a roaming
standard that was replaced by 802.11r.
ff ff ff ff ff ff ec 10 7b 5f ?? ?? 00 06 00 01 af 81 01 00
(just masked 2 bytes of my MAC)
https://patchwork.kernel.org/patch/10191451/
The problem is that bridge (in setup as above) handles such a packet
back to the device.
That makes Broadcom's FullMAC firmware believe that a given station
just connected to another AP in a network (which doesn't even exist).
As a result firmware immediately disassociates that station. It's
simply impossible to connect to the router. Every association is
followed by immediate disassociation.
Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

ebtables is your friend in dealing with weird and broken devices.

[Linus Lüssing]

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.

Hi Rafał,

Thanks for reporting this issue!

Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Just to avoid some potential confusion: This is more an issue of
hairpinning than it is an issue of multicast-to-unicast per se,
right?

That is, if you set this to 0 manually:
/sys/class/net/<ap-iface>/brport/multicast_to_unicast
Then the issue still occurs, right?

Regards, Linus

[Linus Lüssing]

Post by Linus Lüssing

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.

Hi Rafał,
Thanks for reporting this issue!

Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Just to avoid some potential confusion: This is more an issue of
hairpinning than it is an issue of multicast-to-unicast per se,
right?
/sys/class/net/<ap-iface>/brport/multicast_to_unicast
Then the issue still occurs, right?

Btw., if in OpenWRT/LEDE you set 'option multicast_to_unicast 0'
in /etc/config/network for the bridge (and not bridge port)
then netifd should refrain from setting the bridge hairpinning and
AP isolation on wireless devices, too, if I remember correctly.

Can you confirm that the issue disappears for you then?

[Rafał Miłecki]

Post by Linus Lüssing

Post by Linus Lüssing

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.

Hi Rafał,
Thanks for reporting this issue!

Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Just to avoid some potential confusion: This is more an issue of
hairpinning than it is an issue of multicast-to-unicast per se,
right?
/sys/class/net/<ap-iface>/brport/multicast_to_unicast
Then the issue still occurs, right?

Btw., if in OpenWRT/LEDE you set 'option multicast_to_unicast 0'
in /etc/config/network for the bridge (and not bridge port)
then netifd should refrain from setting the bridge hairpinning and
AP isolation on wireless devices, too, if I remember correctly.
Can you confirm that the issue disappears for you then?

Yes, absolutely. This reverts OpenWrt/LEDE to the old setup (no
mcast-to-ucast + hairpin) and it works. I was hoping we can make
mcast-to-ucast + hairpin work with Broadcom's 802.11f however instead
of moving OpenWrt/LEDE back to the old setup.

[Rafał Miłecki]

Post by Linus Lüssing

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.

Hi Rafał,
Thanks for reporting this issue!

Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Just to avoid some potential confusion: This is more an issue of
hairpinning than it is an issue of multicast-to-unicast per se,
right?
/sys/class/net/<ap-iface>/brport/multicast_to_unicast
Then the issue still occurs, right?

I can't really tell, I didn't go into details on bridge +
mcast-to-unicast + hairpin.

By default OpenWrt/LEDE sets
/sys/class/net/<ap-iface>/brport/multicast_to_unicast
to 1. Changing it to 0 (with a simple echo) doesn't /fix/ the problem.

[Linus Lüssing]

On 27 February 2018 at 18:05, Stephen Hemminger

[...]

Post by Stephen Hemminger
ebtables is your friend in dealing with weird and broken devices.

It may be weird, not sure if actually broken. Anyway I'd like to have
some generic solution instead of telling every user to use ebtables to
workaround the problem.

I agree that a "broken by default" in OpenWRT/LEDE for a variety
of Broadcom devices is not really acceptable.

Technically we could teach netifd in OpenWRT/LEDE to configure
ebtables accordingly, at least for a list of affected devices,
so that users would not have to. However, as ebtables is not
managed by the fw3 in OpenWRT/LEDE, that would probably interfer
with user provided ebtables rules and scripts...

That said I think we still should look for a solution for existing
firmwares. I guess it may takes months to years to never to release
new firmwares for all supported chipsets.

Hm, we could change the default in OpenWRT/LEDE for
multicast-to-unicast (or more precisely bridge hairpinning) to
disabled again for now.

[Stephen Hemminger]

On Mon, 12 Mar 2018 23:42:48 +0100

2) Blame bridge + mcast-to-ucast + hairpin for 802.11f incompatibility
If we agree that 802.11f support in FullMAC firmware is acceptable, then
we have to make sure Linux's bridge doesn't break it by passing 802.11f
(broadcast) frames back to the source interface. That would require a
check like in below diff + proper code for handling such packets. I'm
afraid I'm not familiar with bridge code enough to complete that.
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index edae702..9e5d6ea 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -126,6 +126,27 @@ static void br_do_proxy_arp(struct sk_buff *skb, struct net_bridge *br,
}
}
+static bool br_skb_is_iapp_add_packet(struct sk_buff *skb)
+{
+ const u8 iapp_add_packet[6] __aligned(2) = {
+ 0x00, 0x01, 0xaf, 0x81, 0x01, 0x00,
+ };
+#if !defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+ const u16 *a = (const u16 *)skb->data;
+ const u16 *b = (const u16 *)iapp_add_packet;
+#endif
+
+ if (skb->len != 6)
+ return false;
+
+#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+ return !(((*(const u32 *)skb->data) ^ (*(const u32 *)iapp_add_packet)) |
+ ((*(const u16 *)(skb->data + 4)) ^ (*(const u16 *)(iapp_add_packet + 4))));
+#else
+ return !((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]));
+#endif
+}
+
/* note: already called with rcu_read_lock */
int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
@@ -155,6 +176,8 @@ int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb
if (is_multicast_ether_addr(dest)) {
/* by definition the broadcast is also a multicast address */
if (is_broadcast_ether_addr(dest)) {
+ if (br_skb_is_iapp_add_packet(skb))
+ pr_warn("This packet should not be passed back to the source interface!\n");
pkt_type = BR_PKT_BROADCAST;
local_rcv = true;
} else {

Don't like bridge doing special case code for magic received values directly in input path.
Really needs to be generic which is why I suggested ebtables.

[Rafał Miłecki]

Post by Stephen Hemminger
On Mon, 12 Mar 2018 23:42:48 +0100

2) Blame bridge + mcast-to-ucast + hairpin for 802.11f incompatibility
If we agree that 802.11f support in FullMAC firmware is acceptable, then
we have to make sure Linux's bridge doesn't break it by passing 802.11f
(broadcast) frames back to the source interface. That would require a
check like in below diff + proper code for handling such packets. I'm
afraid I'm not familiar with bridge code enough to complete that.
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index edae702..9e5d6ea 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -126,6 +126,27 @@ static void br_do_proxy_arp(struct sk_buff *skb, struct net_bridge *br,
}
}
+static bool br_skb_is_iapp_add_packet(struct sk_buff *skb)
+{
+ const u8 iapp_add_packet[6] __aligned(2) = {
+ 0x00, 0x01, 0xaf, 0x81, 0x01, 0x00,
+ };
+#if !defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+ const u16 *a = (const u16 *)skb->data;
+ const u16 *b = (const u16 *)iapp_add_packet;
+#endif
+
+ if (skb->len != 6)
+ return false;
+
+#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+ return !(((*(const u32 *)skb->data) ^ (*(const u32 *)iapp_add_packet)) |
+ ((*(const u16 *)(skb->data + 4)) ^ (*(const u16 *)(iapp_add_packet + 4))));
+#else
+ return !((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]));
+#endif
+}
+
/* note: already called with rcu_read_lock */
int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
@@ -155,6 +176,8 @@ int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb
if (is_multicast_ether_addr(dest)) {
/* by definition the broadcast is also a multicast address */
if (is_broadcast_ether_addr(dest)) {
+ if (br_skb_is_iapp_add_packet(skb))
+ pr_warn("This packet should not be passed back to the source interface!\n");
pkt_type = BR_PKT_BROADCAST;
local_rcv = true;
} else {

Don't like bridge doing special case code for magic received values directly in input path.
Really needs to be generic which is why I suggested ebtables.

We need in-bridge solution only if we decide to support FullMAC
firmwares with 802.11f implementation.

In that case is this possible to use ebtables as a workaround at all?
Can I really use ebtables to set switch to don't pass 802.11f ADD frames
back to the original interface?

[Felix Fietkau]

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.
1) IFLA_BRPORT_MCAST_TO_UCAST
2) Clients isolation in hostapd
3) Hairpin mode enabled
https://patchwork.kernel.org/patch/9530669/
https://lwn.net/Articles/347344/
Short version: in that setup packets received from a bridged wireless
interface can be handled back to it for transmission.
Now, Broadcom's firmware for their FullMAC chipsets in AP mode
supports an obsoleted 802.11f AKA IAPP standard. It's a roaming
standard that was replaced by 802.11r.
ff ff ff ff ff ff ec 10 7b 5f ?? ?? 00 06 00 01 af 81 01 00
(just masked 2 bytes of my MAC)
https://patchwork.kernel.org/patch/10191451/
The problem is that bridge (in setup as above) handles such a packet
back to the device.
That makes Broadcom's FullMAC firmware believe that a given station
just connected to another AP in a network (which doesn't even exist).
As a result firmware immediately disassociates that station. It's
simply impossible to connect to the router. Every association is
followed by immediate disassociation.
Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Let's look at it from a different angle: Since these packets are
forwarded as normal packets by the bridge, and the Broadcom firmware
reacts to them in this nasty way, that's basically local DoS security
issue. In my opinion that matters a lot more than having support for an
obsolete feature that almost nobody will ever want to use.

I think the right approach to deal with this issue is to drop these
garbage packets in both the receive and transmit path of brcmfmac.

- Felix

[Felix Fietkau]

[resent with fixed typo in linux-wireless address]

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.
1) IFLA_BRPORT_MCAST_TO_UCAST
2) Clients isolation in hostapd
3) Hairpin mode enabled
https://patchwork.kernel.org/patch/9530669/
https://lwn.net/Articles/347344/
Short version: in that setup packets received from a bridged wireless
interface can be handled back to it for transmission.
Now, Broadcom's firmware for their FullMAC chipsets in AP mode
supports an obsoleted 802.11f AKA IAPP standard. It's a roaming
standard that was replaced by 802.11r.
ff ff ff ff ff ff ec 10 7b 5f ?? ?? 00 06 00 01 af 81 01 00
(just masked 2 bytes of my MAC)
https://patchwork.kernel.org/patch/10191451/
The problem is that bridge (in setup as above) handles such a packet
back to the device.
That makes Broadcom's FullMAC firmware believe that a given station
just connected to another AP in a network (which doesn't even exist).
As a result firmware immediately disassociates that station. It's
simply impossible to connect to the router. Every association is
followed by immediate disassociation.
Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

Let's look at it from a different angle: Since these packets are
forwarded as normal packets by the bridge, and the Broadcom firmware
reacts to them in this nasty way, that's basically local DoS security
issue. In my opinion that matters a lot more than having support for an
obsolete feature that almost nobody will ever want to use.

I think the right approach to deal with this issue is to drop these
garbage packets in both the receive and transmit path of brcmfmac.

- Felix

[Rafał Miłecki]

Sending with a fixed linux-wireless ML address. Please kindly send your

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.
1) IFLA_BRPORT_MCAST_TO_UCAST
2) Clients isolation in hostapd
3) Hairpin mode enabled
https://patchwork.kernel.org/patch/9530669/
https://lwn.net/Articles/347344/
Short version: in that setup packets received from a bridged wireless
interface can be handled back to it for transmission.
Now, Broadcom's firmware for their FullMAC chipsets in AP mode
supports an obsoleted 802.11f AKA IAPP standard. It's a roaming
standard that was replaced by 802.11r.
ff ff ff ff ff ff ec 10 7b 5f ?? ?? 00 06 00 01 af 81 01 00
(just masked 2 bytes of my MAC)
https://patchwork.kernel.org/patch/10191451/
The problem is that bridge (in setup as above) handles such a packet
back to the device.
That makes Broadcom's FullMAC firmware believe that a given station
just connected to another AP in a network (which doesn't even exist).
As a result firmware immediately disassociates that station. It's
simply impossible to connect to the router. Every association is
followed by immediate disassociation.
Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

[Rafał Miłecki]

On 28 February 2018 at 12:31, Arend van Spriel

Post by Rafał Miłecki
Sending with a fixed linux-wireless ML address. Please kindly send your

I've problem when using OpenWrt/LEDE on a home router with Broadcom's
FullMAC WiFi chipset.
1) IFLA_BRPORT_MCAST_TO_UCAST
2) Clients isolation in hostapd
3) Hairpin mode enabled
https://patchwork.kernel.org/patch/9530669/
https://lwn.net/Articles/347344/
Short version: in that setup packets received from a bridged wireless
interface can be handled back to it for transmission.
Now, Broadcom's firmware for their FullMAC chipsets in AP mode
supports an obsoleted 802.11f AKA IAPP standard. It's a roaming
standard that was replaced by 802.11r.
ff ff ff ff ff ff ec 10 7b 5f ?? ?? 00 06 00 01 af 81 01 00
(just masked 2 bytes of my MAC)
https://patchwork.kernel.org/patch/10191451/
The problem is that bridge (in setup as above) handles such a packet
back to the device.

From reading the referenced links I understand the hairpin mode is causing
the packet to be sent back to the device, and the hairpin mode is required
for MCAST_TO_UCAST, right?

Post by Rafał Miłecki

That makes Broadcom's FullMAC firmware believe that a given station
just connected to another AP in a network (which doesn't even exist).
As a result firmware immediately disassociates that station. It's
simply impossible to connect to the router. Every association is
followed by immediate disassociation.
Can you see any solution for this problem? Is that an option to stop
multicast-to-unicast from touching 802.11f packets? Some other ideas?
Obviously I can't modify Broadcom's firmware and drop that obsoleted
standard.

As far as I can tell you are correct that the 802.11f amendment was never
adopted into the 802.11 standard. I will ask internally if we still have a
reason for carrying it in our firmware.

Thanks! Having at least a way to disable it would be nice.

That said I think we still should look for a solution for existing
firmwares. I guess it may takes months to years to never to release
new firmwares for all supported chipsets.