Hi.
I did same tests on 2 other machines. One of them has same board, CPU
and NIC, second one has just same NIC but different board and CPU.
On first machine(same as initial) I got same results(too much rx_missed_error),
On the second machine I got the output as It should be(almost none
rx_missed_error) and I got zero latency on even 12m pps.
That test tells me there is a issue about board or CPU I use on my
first test. But I'm not sure how I am going to debug it. Here is
device informations
NICs are same on all device as I gave before
Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Failed(initial) device:
Base Board Information
Manufacturer: Intel
Product Name: Greencity
CPU name: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Second(successful) device:
Base Board Information
Manufacturer: INTEL Corporation
Product Name: DENLOW_WS
CPU name: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz
On Wed, Jun 12, 2019 at 11:59 AM İbrahim Ercan <ibrahim.metu@xxxxxxxxx> wrote:
>
> I removed switch but still see rx_missed_errors.
>
> Ethtool(enp7s0f0) stat: 374068 ( 374,068) <= fdir_miss /sec
> Ethtool(enp7s0f0) stat: 22269145 ( 22,269,145) <= rx_bytes /sec
> Ethtool(enp7s0f0) stat: 566912795 ( 566,912,795) <= rx_bytes_nic /sec
> Ethtool(enp7s0f0) stat: 8568315 ( 8,568,315) <= rx_missed_errors /sec
> Ethtool(enp7s0f0) stat: 371152 ( 371,152) <= rx_packets /sec
> Ethtool(enp7s0f0) stat: 297774 ( 297,774) <= rx_pkts_nic /sec
>
> I also tried it with another copper NIC (Intel I350 Gigabit Network
> Connection) on same firewall. I loaded same xdp code and send about
> 1.5m pps. Here I didn't see any error
> Ethtool(enp9s0f0) stat: 95234803 ( 95,234,803) <= rx_bytes /sec
> Ethtool(enp9s0f0) stat: 95234803 ( 95,234,803) <=
> rx_long_byte_count /sec
> Ethtool(enp9s0f0) stat: 1488043 ( 1,488,043) <= rx_packets /sec
>
> Then I tried to change SFP module (with same model) but got same
> results. I compared 'ethtool -m' output of attacker and firewall. I
> noticed signal powers different
>
> on attacker side
> # ethtool -m enp9s0f1
> Identifier : 0x03 (SFP)
> Extended identifier : 0x04 (GBIC/SFP
> defined by 2-wire interface ID)
> Connector : 0x07 (LC)
> Transceiver codes : 0x10 0x00 0x00
> 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 10G Ethernet: 10G Base-SR
> Encoding : 0x06 (64B/66B)
> BR, Nominal : 10300MBd
> Rate identifier : 0x00 (unspecified)
> Length (SMF,km) : 0km
> Length (SMF) : 0m
> Length (50um) : 80m
> Length (62.5um) : 30m
> Length (Copper) : 0m
> Length (OM3) : 300m
> Laser wavelength : 850nm
> Vendor name : FINISAR CORP.
> Vendor OUI : 00:90:65
> Vendor PN : FTLX8571D3BCL
> Vendor rev : A
> Optical diagnostics support : Yes
> Laser bias current : 18.160 mA
> Laser output power : 0.5945 mW / -2.26 dBm
> Receiver signal average optical power : 0.6328 mW / -1.99 dBm
> Module temperature : 35.47 degrees C /
> 95.84 degrees F
> Module voltage : 3.3568 V
> Alarm/warning flags implemented : Yes
> Laser bias current high alarm : Off
> Laser bias current low alarm : Off
> Laser bias current high warning : Off
> Laser bias current low warning : Off
> Laser output power high alarm : Off
> Laser output power low alarm : Off
> Laser output power high warning : Off
> Laser output power low warning : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser rx power high alarm : Off
> Laser rx power low alarm : Off
> Laser rx power high warning : Off
> Laser rx power low warning : Off
> Laser bias current high alarm threshold : 11.800 mA
> Laser bias current low alarm threshold : 4.000 mA
> Laser bias current high warning threshold : 10.800 mA
> Laser bias current low warning threshold : 5.000 mA
> Laser output power high alarm threshold : 0.8318 mW / -0.80 dBm
> Laser output power low alarm threshold : 0.2512 mW / -6.00 dBm
> Laser output power high warning threshold : 0.6607 mW / -1.80 dBm
> Laser output power low warning threshold : 0.3162 mW / -5.00 dBm
> Module temperature high alarm threshold : 78.00 degrees C /
> 172.40 degrees F
> Module temperature low alarm threshold : -13.00 degrees C /
> 8.60 degrees F
> Module temperature high warning threshold : 73.00 degrees C /
> 163.40 degrees F
> Module temperature low warning threshold : -8.00 degrees C /
> 17.60 degrees F
> Module voltage high alarm threshold : 3.7000 V
> Module voltage low alarm threshold : 2.9000 V
> Module voltage high warning threshold : 3.6000 V
> Module voltage low warning threshold : 3.0000 V
> Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm
> Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
> Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm
> Laser rx power low warning threshold : 0.0158 mW / -18.01 dBm
>
> on firewall side (It didn't change after I replace SFP with another)
> # ethtool -m enp7s0f0
> Identifier : 0x03 (SFP)
> Extended identifier : 0x04 (GBIC/SFP
> defined by 2-wire interface ID)
> Connector : 0x07 (LC)
> Transceiver codes : 0x10 0x00 0x00
> 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 10G Ethernet: 10G Base-SR
> Encoding : 0x06 (64B/66B)
> BR, Nominal : 10300MBd
> Rate identifier : 0x00 (unspecified)
> Length (SMF,km) : 0km
> Length (SMF) : 0m
> Length (50um) : 80m
> Length (62.5um) : 30m
> Length (Copper) : 0m
> Length (OM3) : 300m
> Laser wavelength : 850nm
> Vendor name : FINISAR CORP.
> Vendor OUI : 00:90:65
> Vendor PN : FTLX8571D3BCL
> Vendor rev : A
> Option values : 0x00 0x1a
> Option : RX_LOS implemented
> Option : TX_FAULT implemented
> Option : TX_DISABLE implemented
> BR margin, max : 0%
> BR margin, min : 0%
> Vendor SN : AP40XS0
> Date code : 130202
> Optical diagnostics support : Yes
> Laser bias current : 7.762 mA
> Laser output power : 0.6590 mW / -1.81 dBm
> Receiver signal average optical power : 0.4653 mW / -3.32 dBm
> Module temperature : 30.99 degrees C /
> 87.78 degrees F
> Module voltage : 3.3468 V
> Alarm/warning flags implemented : Yes
> Laser bias current high alarm : Off
> Laser bias current low alarm : Off
> Laser bias current high warning : Off
> Laser bias current low warning : Off
> Laser output power high alarm : Off
> Laser output power low alarm : Off
> Laser output power high warning : Off
> Laser output power low warning : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser rx power high alarm : Off
> Laser rx power low alarm : Off
> Laser rx power high warning : Off
> Laser rx power low warning : Off
> Laser bias current high alarm threshold : 13.200 mA
> Laser bias current low alarm threshold : 4.000 mA
> Laser bias current high warning threshold : 12.600 mA
> Laser bias current low warning threshold : 5.000 mA
> Laser output power high alarm threshold : 1.0000 mW / 0.00 dBm
> Laser output power low alarm threshold : 0.2512 mW / -6.00 dBm
> Laser output power high warning threshold : 0.7943 mW / -1.00 dBm
> Laser output power low warning threshold : 0.3162 mW / -5.00 dBm
> Module temperature high alarm threshold : 78.00 degrees C /
> 172.40 degrees F
> Module temperature low alarm threshold : -13.00 degrees C /
> 8.60 degrees F
> Module temperature high warning threshold : 73.00 degrees C /
> 163.40 degrees F
> Module temperature low warning threshold : -8.00 degrees C /
> 17.60 degrees F
> Module voltage high alarm threshold : 3.7000 V
> Module voltage low alarm threshold : 2.9000 V
> Module voltage high warning threshold : 3.6000 V
> Module voltage low warning threshold : 3.0000 V
> Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm
> Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
> Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm
> Laser rx power low warning threshold : 0.0158 mW / -18.01 dBm
>
> May that signal power difference be the problem?
>
>
>
>
>
> On Wed, Jun 12, 2019 at 10:53 AM Jesper Dangaard Brouer
> <brouer@xxxxxxxxxx> wrote:
> >
> >
> > On Wed, 12 Jun 2019 09:57:02 +0300 İbrahim Ercan <ibrahim.metu@xxxxxxxxx> wrote:
> >
> > > I removed bridge and did same tests again. Unfortunately result is
> > > same :/
> >
> > I sort of expected that, as the ethtool "rx_missed_errors" counter
> > says, that packets are dropped inside the NIC, before reaching Linux.
> > Something more fundamental is wrong with your setup.
> >
> > You mentioned there was a switch between the machines in your lab. One
> > possibility is that the switch is somehow corrupting the frames before
> > the reach the NIC, e.g. in these overload DDoS scenarios. Try to
> > remove the switch from the equation (by directly connecting machine
> > back-to-back), to identify where the pitfall is...
> >
> > --
> > Best regards,
> > Jesper Dangaard Brouer
> > MSc.CS, Principal Kernel Engineer at Red Hat
> > LinkedIn: http://www.linkedin.com/in/brouer
|