Hello
I have the following setup:
+------------------------------------+
| KVM CENTOS7 3.10.0-327.el7.x86_64 |
| +------+ | +--------+
| | VM1 | +--------+ | | SWITCH |
| | vio0-+--vnet0--+mytrunk0+--enp1s0+--+ TRUNK |
| +------+ +--------+ | | PORT |
+------------------------------------+ +--------+
The enp1s0 physical interface dump from ethtool is pasted at the end of this email as well as brctl output and interfaces config (please scroll down).
I am trying to achieve transparent VLAN bridging without defining any subinterfaces on the KVM box. The goal is for the KVM box just forward frames from physical port to VM and vice-versa. Any VM connected to mytrunk0 should be able to use any VLAN ID without the hypervisor having to be configured on it as well. The VMs (to be) connected to mytrunk0 will be a part of several dozen VLANs therefore having to manually define one bridge for each one of those just won't scale from a management perspective.
What I am seeing is:
0 - After KVM host boot i add enp1s0 to the bridge using the command "brctl addif mytrunk0 enp1s0"
1 - VM1 (OpenBSD 6.0) is properly tagging packets out on vnet0 (tcpdump output below):
# tcpdump -n -e -i vnet0
<...>
09:52:42.786031 52:54:00:40:24:a5 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 1000, p 3, ethertype ARP, Request who-has 10.0.0.19 tell 10.0.0.222, length 28
2 - enp2s0 tells me (INCORRECTLY) it is forwarding this packet out enp1s0 *WITH* the tag:
# tcpdump -n -e -i enp1s0
<...>
09:54:47.107468 52:54:00:40:24:a5 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 1000, p 3, ethertype ARP, Request who-has 10.0.0.19 tell 10.0.0.222, length 28
3 - Switch trunk port learns no MAC address on vlan 1000. As soon as I configure the switch as ACCESS port I learn the VM MAC address. This tells me the VLAN tag is being removed from the ethernet frame before sending it to the switch. VM1 learns NO mac address from anywhere with switch being configured either as trunk or access (makes no difference).
4 - I tried disabling all possible NIC offloads with ethtool with no luck. It seems the NIC is somehow stripping the VLAN tag out of the packet before putting on the wire.
# ethtool -K enp1s0 gro off gso off rxvlan off txvlan off rx off tx off sg off rxhash off
5 - I also tried Ubuntu 16.04 and 16.10, and also tried replacing e1000e NIC with a realtek one, getting the exact same results.
Any help figuring this out is greatly appreciated. Not sure I am missing/skipping a step or if this is a bug.
Thank you
Paulo
# ethtool -i enp1s0
driver: e1000e
version: 3.2.5-k
firmware-version: 1.8-0
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# ethtool -d enp1s0
MAC Registers
-------------
0x00000: CTRL (Device control register) 0x18100248
Endian mode (buffers): little
Link reset: reset
Set link up: 1
Invert Loss-Of-Signal: no
Receive flow control: enabled
Transmit flow control: enabled
VLAN mode: disabled
Auto speed detect: disabled
Speed select: 1000Mb/s
Force speed: no
Force duplex: no
0x00008: STATUS (Device status register) 0x00080783
Duplex: full
Link up: link config
TBI mode: disabled
Link speed: 1000Mb/s
Bus type: PCI
Bus speed: 33MHz
Bus width: 32-bit
0x00100: RCTL (Receive control register) 0x0400801A
Receiver: enabled
Store bad packets: disabled
Unicast promiscuous: enabled
Multicast promiscuous: enabled
Long packet: disabled
Descriptor minimum threshold size: 1/2
Broadcast accept mode: accept
VLAN filter: disabled
Canonical form indicator: disabled
Discard pause frames: filtered
Pass MAC control frames: don't pass
Receive buffer size: 2048
0x02808: RDLEN (Receive desc length) 0x00001000
0x02810: RDH (Receive desc head) 0x00000003
0x02818: RDT (Receive desc tail) 0x00000000
0x02820: RDTR (Receive delay timer) 0x00000020
0x00400: TCTL (Transmit ctrl register) 0x3103F0FA
Transmitter: enabled
Pad short packets: enabled
Software XOFF Transmission: disabled
Re-transmit on late collision: enabled
0x03808: TDLEN (Transmit desc length) 0x00001000
0x03810: TDH (Transmit desc head) 0x00000003
0x03818: TDT (Transmit desc tail) 0x00000003
0x03820: TIDV (Transmit delay timer) 0x00000008
PHY type: unknown
# brctl show
bridge name bridge id STP enabled interfaces
mytrunk0 8000.6805ca4124ff no enp1s0
vnet0
# cat /etc/sysconfig/network-scripts/ifcfg-mytrunk0
DEVICE=mytrunk0
STP=no
TYPE=Bridge
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=mytrunk0
_ONBOOT_=yes
NM_CONTROLLED=no
# cat /etc/sysconfig/network-scripts/ifcfg-enp1s0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
NAME=enp1s0
DEVICE=enp1s0
_ONBOOT_=yes
BRIDGE=mytrunk0
# virsh dumpxml VM1
<...>
<interface type='bridge'>
<mac address='52:54:00:40:24:a5'/>
<source bridge='mytrunk0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
I have the following setup:
+------------------------------------+
| KVM CENTOS7 3.10.0-327.el7.x86_64 |
| +------+ | +--------+
| | VM1 | +--------+ | | SWITCH |
| | vio0-+--vnet0--+mytrunk0+--enp1s0+--+ TRUNK |
| +------+ +--------+ | | PORT |
+------------------------------------+ +--------+
The enp1s0 physical interface dump from ethtool is pasted at the end of this email as well as brctl output and interfaces config (please scroll down).
I am trying to achieve transparent VLAN bridging without defining any subinterfaces on the KVM box. The goal is for the KVM box just forward frames from physical port to VM and vice-versa. Any VM connected to mytrunk0 should be able to use any VLAN ID without the hypervisor having to be configured on it as well. The VMs (to be) connected to mytrunk0 will be a part of several dozen VLANs therefore having to manually define one bridge for each one of those just won't scale from a management perspective.
What I am seeing is:
0 - After KVM host boot i add enp1s0 to the bridge using the command "brctl addif mytrunk0 enp1s0"
1 - VM1 (OpenBSD 6.0) is properly tagging packets out on vnet0 (tcpdump output below):
# tcpdump -n -e -i vnet0
<...>
09:52:42.786031 52:54:00:40:24:a5 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 1000, p 3, ethertype ARP, Request who-has 10.0.0.19 tell 10.0.0.222, length 28
2 - enp2s0 tells me (INCORRECTLY) it is forwarding this packet out enp1s0 *WITH* the tag:
# tcpdump -n -e -i enp1s0
<...>
09:54:47.107468 52:54:00:40:24:a5 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 1000, p 3, ethertype ARP, Request who-has 10.0.0.19 tell 10.0.0.222, length 28
3 - Switch trunk port learns no MAC address on vlan 1000. As soon as I configure the switch as ACCESS port I learn the VM MAC address. This tells me the VLAN tag is being removed from the ethernet frame before sending it to the switch. VM1 learns NO mac address from anywhere with switch being configured either as trunk or access (makes no difference).
4 - I tried disabling all possible NIC offloads with ethtool with no luck. It seems the NIC is somehow stripping the VLAN tag out of the packet before putting on the wire.
# ethtool -K enp1s0 gro off gso off rxvlan off txvlan off rx off tx off sg off rxhash off
5 - I also tried Ubuntu 16.04 and 16.10, and also tried replacing e1000e NIC with a realtek one, getting the exact same results.
Any help figuring this out is greatly appreciated. Not sure I am missing/skipping a step or if this is a bug.
Thank you
Paulo
# ethtool -i enp1s0
driver: e1000e
version: 3.2.5-k
firmware-version: 1.8-0
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# ethtool -d enp1s0
MAC Registers
-------------
0x00000: CTRL (Device control register) 0x18100248
Endian mode (buffers): little
Link reset: reset
Set link up: 1
Invert Loss-Of-Signal: no
Receive flow control: enabled
Transmit flow control: enabled
VLAN mode: disabled
Auto speed detect: disabled
Speed select: 1000Mb/s
Force speed: no
Force duplex: no
0x00008: STATUS (Device status register) 0x00080783
Duplex: full
Link up: link config
TBI mode: disabled
Link speed: 1000Mb/s
Bus type: PCI
Bus speed: 33MHz
Bus width: 32-bit
0x00100: RCTL (Receive control register) 0x0400801A
Receiver: enabled
Store bad packets: disabled
Unicast promiscuous: enabled
Multicast promiscuous: enabled
Long packet: disabled
Descriptor minimum threshold size: 1/2
Broadcast accept mode: accept
VLAN filter: disabled
Canonical form indicator: disabled
Discard pause frames: filtered
Pass MAC control frames: don't pass
Receive buffer size: 2048
0x02808: RDLEN (Receive desc length) 0x00001000
0x02810: RDH (Receive desc head) 0x00000003
0x02818: RDT (Receive desc tail) 0x00000000
0x02820: RDTR (Receive delay timer) 0x00000020
0x00400: TCTL (Transmit ctrl register) 0x3103F0FA
Transmitter: enabled
Pad short packets: enabled
Software XOFF Transmission: disabled
Re-transmit on late collision: enabled
0x03808: TDLEN (Transmit desc length) 0x00001000
0x03810: TDH (Transmit desc head) 0x00000003
0x03818: TDT (Transmit desc tail) 0x00000003
0x03820: TIDV (Transmit delay timer) 0x00000008
PHY type: unknown
# brctl show
bridge name bridge id STP enabled interfaces
mytrunk0 8000.6805ca4124ff no enp1s0
vnet0
# cat /etc/sysconfig/network-scripts/ifcfg-mytrunk0
DEVICE=mytrunk0
STP=no
TYPE=Bridge
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=mytrunk0
_ONBOOT_=yes
NM_CONTROLLED=no
# cat /etc/sysconfig/network-scripts/ifcfg-enp1s0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
NAME=enp1s0
DEVICE=enp1s0
_ONBOOT_=yes
BRIDGE=mytrunk0
# virsh dumpxml VM1
<...>
<interface type='bridge'>
<mac address='52:54:00:40:24:a5'/>
<source bridge='mytrunk0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
