Cannot access internet with virtual switch

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot access internet with virtual switch

Aham Brahmasmi
Hello misc,

Problem
A physical server with a switch (add em0 up) cannot access the internet.
However, the same host with a bridge (add em0 up) can access the
internet.

Steps
$ ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 22:22:22:22:22:22
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
...
$ doas route -n show
Routing tables

Internet:
Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
default         20.20.20.1         UGS        0     1XXX     -     8 em0
224/4           127.0.0.1          URS        0        0 32768     8 lo0
127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
...
$ doas ifconfig switch0 create
$ doas ifconfig switch0 add em0
$ doas ifconfig switch0 up
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
31 packets transmitted, 0 packets received, 100.0% packet loss
$ ifconfig
em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        lladdr 22:22:22:22:22:22
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
switch0: flags=41<UP,RUNNING>
        index 6 llprio 3
        groups: switch
        datapath xxxxxxxxxxxxxxxxxx maxflow 10000 maxgroup 1000
        em0 flags=0<>
                port 1 ifpriority 0 ifcost 0
...
$ doas route -n show
Routing tables

Internet:
Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
default         20.20.20.1         UGS        0     1XXX     -     8 em0
224/4           127.0.0.1          URS        0        0 32768     8 lo0
127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
$ doas ifconfig switch0 destroy
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms

Repeating the above steps with bridge0 does let the ping pass through
after the bridge is brought up. The only delta between the switch and
bridge output is in the ifconfig.
$ ifconfig
bridge0: flags=41<UP,RUNNING>
        index 8 llprio 3
        groups: bridge
        priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
        em0 flags=3<LEARNING,DISCOVER>
                port 1 ifpriority 0 ifcost 0
...

I have run "doas route -n monitor" in a separate session while doing
this. However, I cannot comprehend the output. pf is not involved -
running tcpdump -nettti pflog0 with the catchall "block log" produces
the normal output of blocked packets with the bridge. However, it stops
producing the normal output of blocked packets with the switch. Once the
switch is destroyed, it is back to normal blocked packets output.

What am I doing wrong/missing? The only thing that stands out to me is
the em0 flags=0<> line in the ifconfig for the switch. And I do not know
what to make of it.

Regards,
ab
---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi <[hidden email]> wrote:

> Hello misc,
>
> Problem
> A physical server with a switch (add em0 up) cannot access the internet.
> However, the same host with a bridge (add em0 up) can access the
> internet.
>
> Steps
> $ ifconfig
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 22:22:22:22:22:22
>         index 1 priority 0 llprio 3
>         groups: egress
>         media: Ethernet autoselect (1000baseT full-duplex,master)
>         status: active
>         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> ...
> $ doas route -n show
> Routing tables
>
> Internet:
> Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> default         20.20.20.1         UGS        0     1XXX     -     8 em0
> 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> ...
> $ doas ifconfig switch0 create
> $ doas ifconfig switch0 add em0
> $ doas ifconfig switch0 up
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> ^C
> --- 8.8.8.8 ping statistics ---
> 31 packets transmitted, 0 packets received, 100.0% packet loss

Hi,

Seems you haven't started switchd(8), or connected your switch to it
-- it shouldn't forward traffic until you do so.

> $ ifconfig
> em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 22:22:22:22:22:22
>         index 1 priority 0 llprio 3
>         groups: egress
>         media: Ethernet autoselect (1000baseT full-duplex,master)
>         status: active
>         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> switch0: flags=41<UP,RUNNING>
>         index 6 llprio 3
>         groups: switch
>         datapath xxxxxxxxxxxxxxxxxx maxflow 10000 maxgroup 1000
>         em0 flags=0<>
>                 port 1 ifpriority 0 ifcost 0
> ...
> $ doas route -n show
> Routing tables
>
> Internet:
> Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> default         20.20.20.1         UGS        0     1XXX     -     8 em0
> 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> $ doas ifconfig switch0 destroy
> $ ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
>
> Repeating the above steps with bridge0 does let the ping pass through
> after the bridge is brought up. The only delta between the switch and
> bridge output is in the ifconfig.
> $ ifconfig
> bridge0: flags=41<UP,RUNNING>
>         index 8 llprio 3
>         groups: bridge
>         priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
>         em0 flags=3<LEARNING,DISCOVER>
>                 port 1 ifpriority 0 ifcost 0
> ...
>
> I have run "doas route -n monitor" in a separate session while doing
> this. However, I cannot comprehend the output. pf is not involved -
> running tcpdump -nettti pflog0 with the catchall "block log" produces
> the normal output of blocked packets with the bridge. However, it stops
> producing the normal output of blocked packets with the switch. Once the
> switch is destroyed, it is back to normal blocked packets output.
>
> What am I doing wrong/missing? The only thing that stands out to me is
> the em0 flags=0<> line in the ifconfig for the switch. And I do not know
> what to make of it.
>
> Regards,
> ab
> ---------|---------|---------|---------|---------|---------|---------|--
>

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
> Sent: Saturday, April 07, 2018 at 5:02 AM
> From: "Ayaka Koshibe" <[hidden email]>
> To: "Aham Brahmasmi" <[hidden email]>
> Cc: [hidden email]
> Subject: Re: Cannot access internet with virtual switch
>
> On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi <[hidden email]> wrote:
> > Hello misc,
> >
> > Problem
> > A physical server with a switch (add em0 up) cannot access the internet.
> > However, the same host with a bridge (add em0 up) can access the
> > internet.
> >
> > Steps
> > $ ifconfig
> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr 22:22:22:22:22:22
> >         index 1 priority 0 llprio 3
> >         groups: egress
> >         media: Ethernet autoselect (1000baseT full-duplex,master)
> >         status: active
> >         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> > ...
> > $ doas route -n show
> > Routing tables
> >
> > Internet:
> > Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> > default         20.20.20.1         UGS        0     1XXX     -     8 em0
> > 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> > 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> > 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> > 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> > 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> > 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> > 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> > $ ping 8.8.8.8
> > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> > ...
> > $ doas ifconfig switch0 create
> > $ doas ifconfig switch0 add em0
> > $ doas ifconfig switch0 up
> > $ ping 8.8.8.8
> > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > ^C
> > --- 8.8.8.8 ping statistics ---
> > 31 packets transmitted, 0 packets received, 100.0% packet loss
>
> Hi,
>
> Seems you haven't started switchd(8), or connected your switch to it
> -- it shouldn't forward traffic until you do so.


Hi Koshibe-san,

Thank you for your reply.

I have started switchd and connected to it. However, I still cannot
ping 8.8.8.8. Starting switchd in debug mode results in output which
broadly says error and closes the switch.

Steps (after the above switch0 up)
$ cat /etc/switchd.conf
listen on 0.0.0.0 tls port 6633
$ doas switchd -dvvvv
listen on 0.0.0.0 6633

(On another session)
$ switchctl connect /dev/switch0

(Back to main session)
ofrelay_input_done: ...
/dev/switch0 > any: ...
switch_learn: ...
packet_input: ...
any > /dev/switch0: ...
(above block repeated multiple times)
...
ofrelay_input_done: connection 1.1: 76 bytes from switch 1
0401004c 00000013 00020004 040d00a9 00000013 ffffffff 00000001 00100000
00000000 00000010 ffffffff ffff0000 00000000 00c88be2 d687ac1f 6b2e22ce
8100026f 08004500 006f42d2
/dev/switch0 > any: version 1_3 type ERROR length 76 xid 19
        error type BAD_ACTION code 4
ofp13_input: message not supported: ERROR
ofrelay_close: connection 1.1 closed
switch_remove: switch 1 removed.

(Another session)
$ tail -10 /var/log/messages
Apr 9 XX:XX:XX MachineName /bsd: arp: attempt to add entry for GATEWAY_IP
on em0 by XX:XX:XX:XX:XX:XX on tap0
(above message repeated infrequently)

If it helps in any way, this machine is a dedicated/bare-metal machine
on a large dedicated/bare-metal machine provider's network. The em0
interface is in the egress group, has a public IP and is connected to
the internet via the provider's network equipment.

The end goal in using the switch is to enable multiple OpenBSD VM's with
with non-contiguous public IPs to be connected to the Internet as real
hosts. In https://www.openbsd.org/faq/faq6.html#VMMnet, this is the
Option 4, except using a switch instead of a bridge and public IPs
on the host network.

Regards,
ab
---------|---------|---------|---------|---------|---------|---------|--

>
> > $ ifconfig
> > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr 22:22:22:22:22:22
> >         index 1 priority 0 llprio 3
> >         groups: egress
> >         media: Ethernet autoselect (1000baseT full-duplex,master)
> >         status: active
> >         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> > switch0: flags=41<UP,RUNNING>
> >         index 6 llprio 3
> >         groups: switch
> >         datapath xxxxxxxxxxxxxxxxxx maxflow 10000 maxgroup 1000
> >         em0 flags=0<>
> >                 port 1 ifpriority 0 ifcost 0
> > ...
> > $ doas route -n show
> > Routing tables
> >
> > Internet:
> > Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> > default         20.20.20.1         UGS        0     1XXX     -     8 em0
> > 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> > 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> > 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> > 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> > 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> > 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> > 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> > $ doas ifconfig switch0 destroy
> > $ ping 8.8.8.8
> > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> >
> > Repeating the above steps with bridge0 does let the ping pass through
> > after the bridge is brought up. The only delta between the switch and
> > bridge output is in the ifconfig.
> > $ ifconfig
> > bridge0: flags=41<UP,RUNNING>
> >         index 8 llprio 3
> >         groups: bridge
> >         priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
> >         em0 flags=3<LEARNING,DISCOVER>
> >                 port 1 ifpriority 0 ifcost 0
> > ...
> >
> > I have run "doas route -n monitor" in a separate session while doing
> > this. However, I cannot comprehend the output. pf is not involved -
> > running tcpdump -nettti pflog0 with the catchall "block log" produces
> > the normal output of blocked packets with the bridge. However, it stops
> > producing the normal output of blocked packets with the switch. Once the
> > switch is destroyed, it is back to normal blocked packets output.
> >
> > What am I doing wrong/missing? The only thing that stands out to me is
> > the em0 flags=0<> line in the ifconfig for the switch. And I do not know
> > what to make of it.
> >
> > Regards,
> > ab
> > ---------|---------|---------|---------|---------|---------|---------|--
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
> Sent: Monday, April 09, 2018 at 6:50 PM
> From: "Aham Brahmasmi" <[hidden email]>
> To: [hidden email]
> Subject: Re: Cannot access internet with virtual switch
>
> > Sent: Saturday, April 07, 2018 at 5:02 AM
> > From: "Ayaka Koshibe" <[hidden email]>
> > To: "Aham Brahmasmi" <[hidden email]>
> > Cc: [hidden email]
> > Subject: Re: Cannot access internet with virtual switch
> >
> > On Fri, Apr 6, 2018 at 4:40 PM, Aham Brahmasmi <[hidden email]> wrote:
> > > Hello misc,
> > >
> > > Problem
> > > A physical server with a switch (add em0 up) cannot access the internet.
> > > However, the same host with a bridge (add em0 up) can access the
> > > internet.
> > >
> > > Steps
> > > $ ifconfig
> > > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > >         lladdr 22:22:22:22:22:22
> > >         index 1 priority 0 llprio 3
> > >         groups: egress
> > >         media: Ethernet autoselect (1000baseT full-duplex,master)
> > >         status: active
> > >         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> > > ...
> > > $ doas route -n show
> > > Routing tables
> > >
> > > Internet:
> > > Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> > > default         20.20.20.1         UGS        0     1XXX     -     8 em0
> > > 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> > > 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> > > 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> > > 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> > > 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> > > 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> > > 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> > > $ ping 8.8.8.8
> > > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> > > ...
> > > $ doas ifconfig switch0 create
> > > $ doas ifconfig switch0 add em0
> > > $ doas ifconfig switch0 up
> > > $ ping 8.8.8.8
> > > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > > ^C
> > > --- 8.8.8.8 ping statistics ---
> > > 31 packets transmitted, 0 packets received, 100.0% packet loss
> >
> > Hi,
> >
> > Seems you haven't started switchd(8), or connected your switch to it
> > -- it shouldn't forward traffic until you do so.
>
>
> Hi Koshibe-san,
>
> Thank you for your reply.
>
> I have started switchd and connected to it. However, I still cannot
> ping 8.8.8.8. Starting switchd in debug mode results in output which
> broadly says error and closes the switch.
>
> Steps (after the above switch0 up)
> $ cat /etc/switchd.conf
> listen on 0.0.0.0 tls port 6633
> $ doas switchd -dvvvv
> listen on 0.0.0.0 6633
>
> (On another session)
> $ switchctl connect /dev/switch0
>
> (Back to main session)
> ofrelay_input_done: ...
> /dev/switch0 > any: ...
> switch_learn: ...
> packet_input: ...
> any > /dev/switch0: ...
> (above block repeated multiple times)
> ...
> ofrelay_input_done: connection 1.1: 76 bytes from switch 1
> 0401004c 00000013 00020004 040d00a9 00000013 ffffffff 00000001 00100000
> 00000000 00000010 ffffffff ffff0000 00000000 00c88be2 d687ac1f 6b2e22ce
> 8100026f 08004500 006f42d2
> /dev/switch0 > any: version 1_3 type ERROR length 76 xid 19
>         error type BAD_ACTION code 4
> ofp13_input: message not supported: ERROR
> ofrelay_close: connection 1.1 closed
> switch_remove: switch 1 removed.
>
> (Another session)
> $ tail -10 /var/log/messages
> Apr 9 XX:XX:XX MachineName /bsd: arp: attempt to add entry for GATEWAY_IP
> on em0 by XX:XX:XX:XX:XX:XX on tap0
> (above message repeated infrequently)
>
> If it helps in any way, this machine is a dedicated/bare-metal machine
> on a large dedicated/bare-metal machine provider's network. The em0
> interface is in the egress group, has a public IP and is connected to
> the internet via the provider's network equipment.
>
> The end goal in using the switch is to enable multiple OpenBSD VM's with
> with non-contiguous public IPs to be connected to the Internet as real
> hosts. In https://www.openbsd.org/faq/faq6.html#VMMnet, this is the
> Option 4, except using a switch instead of a bridge and public IPs
> on the host network.
>
> Regards,
> ab
> ---------|---------|---------|---------|---------|---------|---------|--

Hi,

I have tried to locate the piece of code that might be causing the
switch to close.

In order to do so, I first looked at the specification 1.3.5 [1] for
Openflow protocol, specifically the ERROR message. This is because the
error that causes the switch to close is the "message not supported:
ERROR" message. This led me to page 113 (out of 177). Reading through
it led me to the following:

"If the error message is in response to a specific message from the
controller, then the xid field of the header in the error message must
match that of the offending message"

The OFP ERROR message has an xid of 19. Looking at the log, the message
just previous to that had an xid of 19, which implies that the previous
message caused the error. The full output from "doas switchd -dvvvv"
for that message is:

any > /dev/switch0: version 1_3 type PACKET_OUT length 169 xid 19
        buffer NO_BUFFER in_port <1> actions_len 16
                action OUTPUT len 16 port ANY max_len NO_BUFFER

Now looking at the ERROR message, it contains error type BAD_ACTION
and code 4. In source, BAD_ACTION is OFP_ERRTYPE_BAD_ACTION. Code 4
according to the enum ofp_bad_action_code on page 115 of the spec is
OFPBAC_BAD_OUT_PORT. OFP_ERRTYPE_BAD_ACTION is defined in sys/net/ofp.h
and used in the function swofp_validate_action in sys/net/switchofp.c.

Based on the above information, my guess is that the following line in
the function swofp_validate_action causes the error to occur:
case OFP_ACTION_OUTPUT:
...
        case OFP_PORT_ANY:
                *err = OFP_ERRACTION_OUT_PORT;
                return (-1);

This informs us that for a PACKET_OUT with action OUTPUT, it cannot
have its port as ANY. Now, I do not know why for a PACKET_OUT message,
an action OUTPUT cannot have port as ANY. More importantly, I do not
know why the controller seems to be sending the PACKET_OUT with action
OUTPUT and port ANY.

Beyond this, unfortunately, my skills more or less reach their limit.
If possible, I would request Reyk, Goda-san or Yasuoka-san as well to
please share any insights/help me. If required, I will share the
1100+ line log output of "doas switchd -dvvvv".

Regards,
ab
[1] - https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-switch-v1.3.5.pdf
---------|---------|---------|---------|---------|---------|---------|--

>
> >
> > > $ ifconfig
> > > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > >         lladdr 22:22:22:22:22:22
> > >         index 1 priority 0 llprio 3
> > >         groups: egress
> > >         media: Ethernet autoselect (1000baseT full-duplex,master)
> > >         status: active
> > >         inet 20.20.20.20 netmask 0xffffff00 broadcast 20.20.20.255
> > > switch0: flags=41<UP,RUNNING>
> > >         index 6 llprio 3
> > >         groups: switch
> > >         datapath xxxxxxxxxxxxxxxxxx maxflow 10000 maxgroup 1000
> > >         em0 flags=0<>
> > >                 port 1 ifpriority 0 ifcost 0
> > > ...
> > > $ doas route -n show
> > > Routing tables
> > >
> > > Internet:
> > > Destination     Gateway            Flags   Refs      Use   Mtu  Prio Iface
> > > default         20.20.20.1         UGS        0     1XXX     -     8 em0
> > > 224/4           127.0.0.1          URS        0        0 32768     8 lo0
> > > 127/8           127.0.0.1          UGRS       0        0 32768     8 lo0
> > > 127.0.0.1       127.0.0.1          UHhl       1        X 32768     1 lo0
> > > 20.20.20/24     20.20.20.20        UCn        1      9XX     -     4 em0
> > > 20.20.20.1      33:33:33:33:33:33  UHLch      1     1XXX     -     3 em0
> > > 20.20.20.20     44:44:44:44:44:44  UHLl       0        X     -     1 em0
> > > 20.20.20.255    20.20.20.20        UHb        0        0     -     1 em0
> > > $ doas ifconfig switch0 destroy
> > > $ ping 8.8.8.8
> > > PING 8.8.8.8 (8.8.8.8): 56 data bytes
> > > 64 bytes from 8.8.8.8: icmp_seq=0 ttl=61 time=x.xxx ms
> > >
> > > Repeating the above steps with bridge0 does let the ping pass through
> > > after the bridge is brought up. The only delta between the switch and
> > > bridge output is in the ifconfig.
> > > $ ifconfig
> > > bridge0: flags=41<UP,RUNNING>
> > >         index 8 llprio 3
> > >         groups: bridge
> > >         priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rtsp
> > >         em0 flags=3<LEARNING,DISCOVER>
> > >                 port 1 ifpriority 0 ifcost 0
> > > ...
> > >
> > > I have run "doas route -n monitor" in a separate session while doing
> > > this. However, I cannot comprehend the output. pf is not involved -
> > > running tcpdump -nettti pflog0 with the catchall "block log" produces
> > > the normal output of blocked packets with the bridge. However, it stops
> > > producing the normal output of blocked packets with the switch. Once the
> > > switch is destroyed, it is back to normal blocked packets output.
> > >
> > > What am I doing wrong/missing? The only thing that stands out to me is
> > > the em0 flags=0<> line in the ifconfig for the switch. And I do not know
> > > what to make of it.
> > >
> > > Regards,
> > > ab
> > > ---------|---------|---------|---------|---------|---------|---------|--
> > >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
> This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> an action OUTPUT cannot have port as ANY. More importantly, I do not
> know why the controller seems to be sending the PACKET_OUT with action
> OUTPUT and port ANY.

A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
so it would probably help to see which message (if any) the switch
sent to the controller to receive that PACKET_OUT.

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
> Sent: Wednesday, April 11, 2018 at 10:18 AM

> From: "Ayaka Koshibe" <[hidden email]>
> To: [hidden email]
> Subject: Re: Cannot access internet with virtual switch
>
> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> > an action OUTPUT cannot have port as ANY. More importantly, I do not
> > know why the controller seems to be sending the PACKET_OUT with action
> > OUTPUT and port ANY.
>
> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
> so it would probably help to see which message (if any) the switch
> sent to the controller to receive that PACKET_OUT.
Thank you Koshibe-san for your reply.

From what I understand, the PACKET_IN for that PACKET_OUT seems to be
the following:

ofrelay_input_done: connection 1.1: 179 bytes from switch 1
/dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
        buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 0x0000000000000000
        match type OXM length 24 (padded to 26)
        ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
                1
        ox match class OPENFLOW_BASIC type META hasmask no length 8
                0
switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1

I have also attached the complete output of "doas switchd -dvvvv". This
is because I do not know whether the above message is the correct
PACKET_OUT message corresponding to the PACKET_IN message.

Regards,
ab
---------|---------|---------|---------|---------|---------|---------|--

s.log (68K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
On Wed, Apr 11, 2018 at 6:25 AM, Aham Brahmasmi <[hidden email]> wrote:

>> Sent: Wednesday, April 11, 2018 at 10:18 AM
>> From: "Ayaka Koshibe" <[hidden email]>
>> To: [hidden email]
>> Subject: Re: Cannot access internet with virtual switch
>>
>> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
>> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
>> > an action OUTPUT cannot have port as ANY. More importantly, I do not
>> > know why the controller seems to be sending the PACKET_OUT with action
>> > OUTPUT and port ANY.
>>
>> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
>> so it would probably help to see which message (if any) the switch
>> sent to the controller to receive that PACKET_OUT.
>
> Thank you Koshibe-san for your reply.
>
> From what I understand, the PACKET_IN for that PACKET_OUT seems to be
> the following:
>
> ofrelay_input_done: connection 1.1: 179 bytes from switch 1
> /dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
>         buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 0x0000000000000000
>         match type OXM length 24 (padded to 26)
>         ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
>                 1
>         ox match class OPENFLOW_BASIC type META hasmask no length 8
>                 0
> switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
> packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1

This seems to be the right message. It looks like switchd will set the
output port to ANY if it sees a loop (which port 1 -> 1 suggests),
intended for dropping the packet. Do you have loops?

> I have also attached the complete output of "doas switchd -dvvvv". This
> is because I do not know whether the above message is the correct
> PACKET_OUT message corresponding to the PACKET_IN message.
>
> Regards,
> ab
> ---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
> Sent: Thursday, April 12, 2018 at 11:24 AM
> From: "Ayaka Koshibe" <[hidden email]>
> To: [hidden email]
> Subject: Re: Cannot access internet with virtual switch
>
> On Wed, Apr 11, 2018 at 6:25 AM, Aham Brahmasmi <[hidden email]> wrote:
> >> Sent: Wednesday, April 11, 2018 at 10:18 AM
> >> From: "Ayaka Koshibe" <[hidden email]>
> >> To: [hidden email]
> >> Subject: Re: Cannot access internet with virtual switch
> >>
> >> > This informs us that for a PACKET_OUT with action OUTPUT, it cannot
> >> > have its port as ANY. Now, I do not know why for a PACKET_OUT message,
> >> > an action OUTPUT cannot have port as ANY. More importantly, I do not
> >> > know why the controller seems to be sending the PACKET_OUT with action
> >> > OUTPUT and port ANY.
> >>
> >> A PACKET_OUT is usually a response to some message e.g. a PACKET_IN,
> >> so it would probably help to see which message (if any) the switch
> >> sent to the controller to receive that PACKET_OUT.
> >
> > Thank you Koshibe-san for your reply.
> >
> > From what I understand, the PACKET_IN for that PACKET_OUT seems to be
> > the following:
> >
> > ofrelay_input_done: connection 1.1: 179 bytes from switch 1
> > /dev/switch0 > any: version 1_3 type PACKET_IN length 179 xid 81972
> >         buffer NO_BUFFER length 129 reason REASON_NO_MATCH table <0> cookie 0x0000000000000000
> >         match type OXM length 24 (padded to 26)
> >         ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
> >                 1
> >         ox match class OPENFLOW_BASIC type META hasmask no length 8
> >                 0
> > switch_learn: updated mac ac:1f:6b:2e:22:ce on switch 1 port 1
> > packet_input: ac:1f:6b:2e:22:ce -> 00:c8:8b:e2:d6:87, port 1 -> 1
>
> This seems to be the right message. It looks like switchd will set the
> output port to ANY if it sees a loop (which port 1 -> 1 suggests),
> intended for dropping the packet. Do you have loops?

Thank you Koshibe-san for your reply.

I looked up switch loops in order to understand your insight. Based on
the configuration of hostname.switch0, I do not think there is a loop
in the virtual switch. However, I may be wrong since I am not good at
networks.

$ cat /etc/hostname.switch0
add em0
up

Here, em0 is the egress interface connected to the dedicated/bare-metal
machine provider's network. This provider's network is beyond my
control. As such, there might be a loop in the provider's network.

Is there a way that I can verify if there is a loop in the network?

If it helps in any way, a bridge works on the same network.

$ cat /etc/hostname.bridge0
add em0

Regards,
ab
---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
> $ cat /etc/hostname.switch0
> add em0
> up
>
> Here, em0 is the egress interface connected to the dedicated/bare-metal
> machine provider's network. This provider's network is beyond my
> control. As such, there might be a loop in the provider's network.

(Sorry, was meaning to respond but got busy)
That's my suspicion, the network you're connected to, rather than your
switch. Say, if you are still trying to use switch(4), there's a
chance that this might help:

https://marc.info/?l=openbsd-tech&m=152424475112610&w=2

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
> > $ cat /etc/hostname.switch0
> > add em0
> > up
> >
> > Here, em0 is the egress interface connected to the dedicated/bare-metal
> > machine provider's network. This provider's network is beyond my
> > control. As such, there might be a loop in the provider's network.
>
> (Sorry, was meaning to respond but got busy)
> That's my suspicion, the network you're connected to, rather than your
> switch. Say, if you are still trying to use switch(4), there's a
> chance that this might help:
>
> https://marc.info/?l=openbsd-tech&m=152424475112610&w=2

Thank you Koshibe-san for your reply.

Please, no sorry. I am grateful for your help.

Currently, I am using a bridge. From what I understand from the patch
and the cvsweb history, that patch is waiting for ok and commit.

This is my first time running a patch, so I may have messed up the
steps. Based on some inspiration from [1], I have used the following
steps for running the patch on -current:

$ doas user mod -G wsrc <user>
$ exit
<relogin to let user mod take effect>
$ cd /usr
$ cvs -qd [hidden email]:/cvs checkout -P src
$ cd /usr/src
$ cvs -q up -Pd -A
$ cd /usr/src/usr.sbin/switchd
$ vi ofp13.c
<make patch changes>
$ doas user mod -G wobj <user>
$ exit
<relogin to let user mod take effect>
$ cd /usr/src/usr.sbin/switchd
$ make obj
$ make
$ doas make install

Then the usual switch0, switchd and switchctl steps, as mentioned in
my previous mails.

This time, the switch does not close. However, I am still unable to
ping 8.8.8.8 with the switch. As usual, I am able to ping 8.8.8.8
using a bridge.

There is a continuous stream of messages when running "switchd -dvv":
...
switch_learn: updated mac 11:11:11:11:11:11 on switch 1 port 1
packet_input: 11:11:11:11:11:11 -> 22:22:22:22:22:22, port 1 -> 1
ofrelay_input_done: connection 1.1: 1528 bytes from switch 1
/dev/switch0 > any: version 1_3 type PACKET_IN length 1528 xid 11362
        buffer NO_BUFFER length 1478 reason REASON_NO_MATCH table <0> cookie 0x0
000000000000000
        match type OXM length 24 (padded to 26)
        ox match class OPENFLOW_BASIC type IN_PORT hasmask no length 4
                1
        ox match class OPENFLOW_BASIC type META hasmask no length 8
                0

<repeat above message multiple times with incrementing xids 11363...>
...
switch_learn: updated mac 22:22:22:22:22:22 on switch 1 port 1
packet_input: 22:22:22:22:22:22 -> ff:ff:ff:ff:ff:ff, port 1 -> 4294967295
any > /dev/switch0: version 1_3 type PACKET_OUT length 100 xid 10
        buffer NO_BUFFER in_port <1> actions_len 16
                action OUTPUT len 16 port FLOOD max_len NO_BUFFER
ofrelay_input_done: connection 1.1: 110 bytes from switch 1
...
<repeat port 1 -> 1 messages with incrementing xids 11366...>
<infrequent repeat port 1 -> 4294967295 messages with incrementing xids 11...>

I have noticed in the ifconfig output that the bridge has "rstp" while
there is no "rstp" or "stp" for the switch. It may be possible that the
provider has redundant internet routes, and that the bridge with the
spanning tree protocol might be able to handle it while the switch
may not be. This is just my guess as a layman user and I may be wrong
since I do not know whether spanning tree protocol is applicable for
the switch. I base this on the absence of stp options in the man pages
for switch/switchd/switchctl/switchd.conf.

Furthermore, the output of "switchctl show summary" displays a large
number of macs with ages within 10 seconds, although the switch has
been up for upwards of 1000 seconds. These macs seem to never go above
10 seconds. Two macs in particular are always age 0s. There is only one
switch - /dev/switch0.

Finally, I have a query - What does the tap0 interface do? I ask this
because currently I do not pass in/out tap0 traffic in pf. This is
because I do not know what tap0 is and what it does. With a "block log"
in pf and "tcpdump -nettti pflog0", I get the following output:

tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
<timestamp> rule 1/(match) block in on tap0:<RandomPublicIP> >
239.255.255.250.1900: udp 94 (DF) [tos 0x38] [ttl 1]
...
<repeat above message every 10 seconds...>

If I run a ping 8.8.8.8, the ping packets do not show up in the above
output.

Regards,
ab
[1] - https://ftp.openbsd.org/pub/OpenBSD/patches/6.3/common/005_httpd.patch.sig
---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
> Currently, I am using a bridge. From what I understand from the patch
> and the cvsweb history, that patch is waiting for ok and commit.

I've actually held back on that diff since it's a bit insufficient by itself.

> This time, the switch does not close. However, I am still unable to
> ping 8.8.8.8 with the switch. As usual, I am able to ping 8.8.8.8
> using a bridge.

Actually, you said that you had just em0 on that switch. Can you try
adding a local port (addlocal instead of add) alongside em0? It will
be a vether(4) interface that needs to be given em0's current address,
in its place.

> There is a continuous stream of messages when running "switchd -dvv":
> ...

I can't say what they are without the full output, but you will tend
to see broadcasts (periodic or otherwise) like your second example
even on your bridge. From a second look at your earlier logs, it seems
the 1->1 'loops' are generated by the switch seeing VLAN traffic in
other parts of the network.

> Finally, I have a query - What does the tap0 interface do? I ask this
> because currently I do not pass in/out tap0 traffic in pf. This is
> because I do not know what tap0 is and what it does.

tap0 is a control interface created by switchd to communicate with
switches. It won't carry normal network traffic.

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
On Sat, May 12, 2018 at 8:54 PM, Ayaka Koshibe <[hidden email]> wrote:
>> Currently, I am using a bridge. From what I understand from the patch
>> and the cvsweb history, that patch is waiting for ok and commit.
>
> I've actually held back on that diff since it's a bit insufficient by itself.

Sorry, I should elaborate. If you plan to try the local port, you may
still want the patch in place to prevent the switch from
disconnecting.

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
In reply to this post by Ayaka Koshibe
> tap0 is a control interface created by switchd to communicate with
> switches. It won't carry normal network traffic.

(Last bit of noise, and I'm done)
Actually, I think I'm wrong here. I'll need to dig a bit more to
answer correctly.

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
In reply to this post by Ayaka Koshibe
Thank you Koshibe-san for your reply.

> I've actually held back on that diff since it's a bit insufficient by itself.

Ok.
 
> Actually, you said that you had just em0 on that switch. Can you try
> adding a local port (addlocal instead of add) alongside em0? It will
> be a vether(4) interface that needs to be given em0's current address,
> in its place.

Should I be doing the following? And if yes, what address should em0
have?

$ cat /etc/hostname.vether0
inet 1.2.3.4 255.255.255.0
$ cat /etc/hostname.em0
inet ?.?.?.? 255.255.255.0
$ doas ifconfig switch0 create
$ doas ifconfig switch0 add em0
$ doas ifconfig switch0 addlocal vether0
$ doas ifconfig switch0 up

Here, 1.2.3.4 is the external public IP address of the machine
originally assigned to em0.

> > There is a continuous stream of messages when running "switchd -dvv":
> > ...
>
> I can't say what they are without the full output, but you will tend
> to see broadcasts (periodic or otherwise) like your second example
> even on your bridge. From a second look at your earlier logs, it seems
> the 1->1 'loops' are generated by the switch seeing VLAN traffic in
> other parts of the network.

Ok. Would sharing the full output of "switchd -dvv" help?

Interestingly, while searching for addlocal, I encountered a
presentation on switch [1]. On page 13 of that pdf, there is mention of
the switch sharing the STP code with bridge. Would it be correct to
assume that there would be no loops if there was STP in the switch?

Regards,
ab
[1] - https://www.openbsd.org/papers/bsdcan2016-switchd.pdf
---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Ayaka Koshibe
On Mon, May 14, 2018 at 1:01 PM, Aham Brahmasmi <[hidden email]> wrote:

> Thank you Koshibe-san for your reply.
>
>> I've actually held back on that diff since it's a bit insufficient by itself.
>
> Ok.
>
>> Actually, you said that you had just em0 on that switch. Can you try
>> adding a local port (addlocal instead of add) alongside em0? It will
>> be a vether(4) interface that needs to be given em0's current address,
>> in its place.
>
> Should I be doing the following? And if yes, what address should em0
> have?
>
> $ cat /etc/hostname.vether0
> inet 1.2.3.4 255.255.255.0
> $ cat /etc/hostname.em0
> inet ?.?.?.? 255.255.255.0
> $ doas ifconfig switch0 create
> $ doas ifconfig switch0 add em0
> $ doas ifconfig switch0 addlocal vether0
> $ doas ifconfig switch0 up
>
> Here, 1.2.3.4 is the external public IP address of the machine
> originally assigned to em0.

em0 shouldn't have an address, and you'll also want to explicitly
enable vether0. Otherwise that looks fine.

>> > There is a continuous stream of messages when running "switchd -dvv":
>> > ...
>>
>> I can't say what they are without the full output, but you will tend
>> to see broadcasts (periodic or otherwise) like your second example
>> even on your bridge. From a second look at your earlier logs, it seems
>> the 1->1 'loops' are generated by the switch seeing VLAN traffic in
>> other parts of the network.
>
> Ok. Would sharing the full output of "switchd -dvv" help?

I wouldn't worry much about it, unless adding the local port doesn't
work for you.

> Interestingly, while searching for addlocal, I encountered a
> presentation on switch [1]. On page 13 of that pdf, there is mention of
> the switch sharing the STP code with bridge. Would it be correct to
> assume that there would be no loops if there was STP in the switch?

Even with a bridge, you'd need to enable STP and set priority values
on the ports for it to work, so you're correct - if there were any
loops, the bridge probably wouldn't have worked either. But you've
also seen that, for switch(4), the STP-related options aren't
available in ifconfig, and as far as I can tell switchd doesn't do
topology/loop detection (and probably won't want to rely on (R)STP to
do so). So, the code might be shared, but is likely not used.


> Regards,
> ab
> [1] - https://www.openbsd.org/papers/bsdcan2016-switchd.pdf
> ---------|---------|---------|---------|---------|---------|---------|--

Reply | Threaded
Open this post in threaded view
|

Re: Cannot access internet with virtual switch

Aham Brahmasmi
Thank you Koshibe-san for your reply.

Here is the output of ping, after the steps:
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: Network is down
ping: wrote 8.8.8.8 64 chars, ret=-1
...

So, it seems the ping fails, except, this time there is some output.


> > Interestingly, while searching for addlocal, I encountered a
> > presentation on switch [1]. On page 13 of that pdf, there is mention of
> > the switch sharing the STP code with bridge. Would it be correct to
> > assume that there would be no loops if there was STP in the switch?
>
> Even with a bridge, you'd need to enable STP and set priority values
> on the ports for it to work, so you're correct - if there were any
> loops, the bridge probably wouldn't have worked either. But you've
> also seen that, for switch(4), the STP-related options aren't
> available in ifconfig, and as far as I can tell switchd doesn't do
> topology/loop detection (and probably won't want to rely on (R)STP to
> do so). So, the code might be shared, but is likely not used.

I do not know much about the network stack, but I went grepping in the
source, and I encountered a bstp_create function call in the
switch_clone_create function within sys/net/if_switch.c file. This
bstp_create function seems to be defined in sys/net/bridgestp.c [2].
I may be wrong here, but the bridge seems to have some kind of STP,
since there is the "rstp" in the ifconfig output. Whether the switch
does indeed use STP, and if it does, does it work, is something that I
unfortunately cannot determine.

For tap0, I ran "tcpdump -nettti tap0" on a normal machine, without
the above vether. I saw a large stream of messages. Then I ran it on
em0 interface. The "tcpdump -nettti em0" closely matched the output of
tap0 interface output. This led me to try to understand tap interfaces.
I encountered an article [3] and an image [4]. My hunch is that tap0
is sort of the mirror equivalent of em0. In terms of that image, em0
is the green physical NIC and tap0 is the Virtual Uplink Port. And since
they are connected to two ends of the same (imaginary) cable, they will
have the same set of messages. This leads me to believe that I should
pass all traffic on tap0. Again, this is based on my uneducated guess
and searching, so I could be wrong.

Passing all traffic on tap0 still does not lead to the ping reply.
However, doing a tcpdump on em0 shows an echo request and echo reply
whiile tcpdump on tap0 only shows an echo reply. This is irrespective of
addlocal vether0. Also, this is irrespective of the pf configuration.
Now, why does the echo request not show up on tap0 and what is exactly
stopping the echo reply to reach the ping command - I cannot determine.
I have done the usual block log, tcpdump -netti pflog0 routine. There
is infrequent igmp2 output, but nothing related to icmp. In fact, I
have also used a one line "pass" file for pf configuration, still no
ping reply.

Regards,
ab
[1] - http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/if_switch.c?rev=1.23&content-type=text/x-cvsweb-markup
[2] - http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/bridgestp.c?rev=1.65&content-type=text/x-cvsweb-markup
[3] - www.innervoice.in/blogs/2013/12/08/tap-interfaces-linux-bridge/
[4] - https://i2.wp.com/www.innervoice.in/blogs/wp-content/uploads/2013/12/VirtualNetwotk.png
---------|---------|---------|---------|---------|---------|---------|--