I have seen this problem before on a network of IBM x336s, w/ the BCM interface. Our analysts determined it was the catalyst router. As I wasn't directly involved in the diagnosis, I can't provide many details on how it was resolved, except to say that we escalated it to our Cisco engineer. Apparently, it was a problem with IOS, not with the servers, as it manifested even worse on the windows boxen. (same IBM h/ware.) -G Regards, Gavin McDonald ======================== EVI Logistic Enterprises email: me@xxxxxxxxxxxx phone: (604) 313-3845 > -----Original Message----- > From: redhat-list-bounces@xxxxxxxxxx [mailto:redhat-list- > bounces@xxxxxxxxxx] On Behalf Of Jimmy Stewpot > Sent: Monday, February 13, 2006 7:45 AM > To: redhat-list@xxxxxxxxxx > Subject: Issues with Performance > > Hello, > > I am currently trying to diagnose a particularly strange issue with a > Redhat Enterprise Linux v3.0 Update 5 system. The particular issue is as > follows. > > I am attempting to run an ICMP monitoring package from the system to > monitor all of the devices around a network. (circa 10,000). When the > ICMP monitoring software is enabled the server becomes unusable via SSH. > By saying unusable I mean the system is very laggy often taking several > minutes to respond to a command. However if I am on the system console > the system is still usable and responds with NO delay. > > Originally I suspected that the issue was related to system load or > network congestion. However I started to monitor the switch ports on the > catalyst 6509 ports with snmp and the utilisation never reaches above > 1Mbps. > > The system is an IBM x336 with dual 2.8Ghz XEONs, 4gb ram and SCSI > drives it has dual port broad com 10/100/1000 interfaces. If I monitor > iostat/vmstat or top I see that the system averages around 90% idle. If > I look at SAR for a long term performance report I see similar > information. > > On the server it directly connects to the 6509 @ 100Mbps Full duplex > there are no errors on the interface using the bcm5700 driver provided > by broadcom. Since looking into the issue I initially thought that the > problem was with the SSH but the problem appears to be more wide spread. > > If I do a rspan on the interface that the server is connected to I see > that the Ethernet frames go both directions however If I do a tcpdump on > the server I am not seeing the packets with the syntax > > 'tcpdump -vvv -n icmp'. > > The SSH sessions and the ICMP return packets not being seen I am sure > are related. In all cases the ports have been forced to 100-FULL Duplex > and I have tested at several other port link speeds and modes without > any success. > > One thing that is slightly different to many configurations is that the > gateway is running CatOS, it then routes in hybrid mode and has VRRP > configured on the device's gateway. However I do not see why that would > cause any problems as the routing is not flipping and there are no layer > 2/3 issues that I can detect. > > What I am trying to diagnose/determine is why the Linux box would not > be seeing the packets returning even though we see them on an rspan as > leaving the switch interface. There are no iptables rules on the server > at all and from what I can see the sysctl stuff is all setup correctly. > When I turn on the debug syslog I still get no error messages or > notifications. > > Where should I be looking now and is there any advice that you guys can > give me? > > Regards, > > Jimmy. > > > > > ===System Information=== > > /sbin/lspci -v > 00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a) > Subsystem: IBM: Unknown device 02dd > Flags: bus master, fast devsel, latency 0 > Memory at <ignored> (32-bit, non-prefetchable) [size=4K] > Capabilities: <available only to root> > > 00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting > Register (rev 0a) > Subsystem: IBM: Unknown device 02dd > Flags: fast devsel > > 00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port > A0 (rev 0a) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=02, subordinate=04, sec-latency=0 > Capabilities: <available only to root> > > 00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port > B0 (rev 0a) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 > Memory behind bridge: ce000000-cfffffff > Capabilities: <available only to root> > > 00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port > B1 (rev 0a) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 > Memory behind bridge: cc000000-cdffffff > Capabilities: <available only to root> > > 00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port > C0 (rev 0a) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=07, subordinate=09, sec-latency=0 > I/O behind bridge: 00004000-0000ffff > Memory behind bridge: ca000000-cbffffff > Capabilities: <available only to root> > > 00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended > Configuration Registers (rev 0a) > Subsystem: IBM: Unknown device 02dd > Flags: fast devsel > > 00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 > (rev 02) (prog-if 00 [UHCI]) > Subsystem: IBM: Unknown device 02dd > Flags: bus master, medium devsel, latency 0, IRQ 16 > I/O ports at 2200 [size=32] > > 00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 > (rev 02) (prog-if 00 [UHCI]) > Subsystem: IBM: Unknown device 02dd > Flags: bus master, medium devsel, latency 0, IRQ 19 > I/O ports at 2600 [size=32] > > 00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI > Controller (rev 02) (prog-if 20 [EHCI]) > Subsystem: IBM: Unknown device 02dd > Flags: bus master, medium devsel, latency 0, IRQ 23 > Memory at f0000000 (32-bit, non-prefetchable) [size=1K] > Capabilities: <available only to root> > > 00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI > Bridge (rev c2) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=01, subordinate=01, sec-latency=32 > I/O behind bridge: 00003000-00003fff > Memory behind bridge: de000000-dfffffff > Prefetchable memory behind bridge: d0000000-ddffffff > > 00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev > 02) > Flags: bus master, medium devsel, latency 0 > > 00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100 > Storage Controller (rev 02) (prog-if 8a [Master SecP PriP]) > Subsystem: IBM: Unknown device 02dd > Flags: bus master, medium devsel, latency 0, IRQ 17 > I/O ports at <ignored> > I/O ports at <ignored> > I/O ports at <ignored> > I/O ports at <ignored> > I/O ports at 0480 [size=16] > Memory at c8000000 (32-bit, non-prefetchable) [size=1K] > > 00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev > 02) > Subsystem: IBM: Unknown device 02dd > Flags: medium devsel, IRQ 17 > I/O ports at 0440 [size=32] > > 01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY > [Radeon 7000/VE] (prog-if 00 [VGA]) > Subsystem: IBM: Unknown device 02c8 > Flags: bus master, stepping, medium devsel, latency 64, IRQ 20 > Memory at d0000000 (32-bit, prefetchable) [size=128M] > I/O ports at 3000 [size=256] > Memory at dfff0000 (32-bit, non-prefetchable) [size=64K] > Expansion ROM at <unassigned> [disabled] [size=128K] > Capabilities: <available only to root> > > 02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00 > [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=02, secondary=03, subordinate=03, sec-latency=64 > Capabilities: <available only to root> > > 02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00 > [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=02, secondary=04, subordinate=04, sec-latency=64 > Capabilities: <available only to root> > > 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 > Gigabit Ethernet PCI Express (rev 01) > Subsystem: IBM: Unknown device 02c6 > Flags: bus master, fast devsel, latency 0, IRQ 16 > Memory at cfff0000 (64-bit, non-prefetchable) [size=64K] > Capabilities: <available only to root> > > 06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 > Gigabit Ethernet PCI Express (rev 01) > Subsystem: IBM: Unknown device 02c6 > Flags: bus master, fast devsel, latency 0, IRQ 16 > Memory at cdff0000 (64-bit, non-prefetchable) [size=64K] > Capabilities: <available only to root> > > 07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) > (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=07, secondary=08, subordinate=08, sec-latency=64 > Memory behind bridge: ca000000-cbffffff > Capabilities: <available only to root> > > 07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) > (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=07, secondary=09, subordinate=09, sec-latency=64 > I/O behind bridge: 00004000-0000ffff > Capabilities: <available only to root> > > 08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07) > Subsystem: IBM: Unknown device 028e > Flags: bus master, stepping, 66Mhz, medium devsel, latency 64, > IRQ 38 > Memory at cbfff000 (32-bit, non-prefetchable) [size=4K] > Expansion ROM at <unassigned> [disabled] [size=512K] > Capabilities: <available only to root> > > Module Size Used by Tainted: P > bcm5700 105380 2 > ide-cd 34016 0 (autoclean) > cdrom 32608 0 (autoclean) [ide-cd] > sg 37260 0 (autoclean) > autofs4 16984 0 (autoclean) (unused) > audit 90584 2 > basp 62020 0 (unused) > microcode 6912 0 (autoclean) > keybdev 2976 0 (unused) > mousedev 5656 0 (unused) > hid 22308 0 (unused) > input 6176 0 [keybdev mousedev hid] > ehci-hcd 20776 0 (unused) > usb-uhci 26988 0 (unused) > usbcore 81088 1 [hid ehci-hcd usb-uhci] > ext3 89960 4 > jbd 55060 4 [ext3] > ips 45252 5 > sd_mod 13712 10 > scsi_mod 114344 3 [sg ips sd_mod] > > /sbin/mii-tool > eth0: 100 Mbit, full duplex, link ok > eth1: 100 Mbit, full duplex, link ok > > > -- > redhat-list mailing list > unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe > https://www.redhat.com/mailman/listinfo/redhat-list -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
