Jimmy, This looks like a tough question and you have done a lot of troubleshooting already. One thing that I noticed you did not mention was checking the physical cable. Is it possible your CAT 5, 5e, or 6 cable is misrated or damaged? Also, if this is going through the wall you might try an alternative cable run. I know this seems very basic, but to me it sounds like you are running in half duplex mode even though you are set to full duplex. If it is imposible to try a different cable run then call up someone with some good CAT 5 testing equipment, and have them check it out. The guys with HP or Agilent testers are usually the best, but maybe that is more so in my area. Job Cacka -----Original Message----- From: redhat-list-bounces@xxxxxxxxxx [mailto:redhat-list-bounces@xxxxxxxxxx]On Behalf Of Jimmy Stewpot Sent: Monday, February 13, 2006 7:45 AM To: redhat-list@xxxxxxxxxx Subject: Issues with Performance Hello, I am currently trying to diagnose a particularly strange issue with a Redhat Enterprise Linux v3.0 Update 5 system. The particular issue is as follows. I am attempting to run an ICMP monitoring package from the system to monitor all of the devices around a network. (circa 10,000). When the ICMP monitoring software is enabled the server becomes unusable via SSH. By saying unusable I mean the system is very laggy often taking several minutes to respond to a command. However if I am on the system console the system is still usable and responds with NO delay. Originally I suspected that the issue was related to system load or network congestion. However I started to monitor the switch ports on the catalyst 6509 ports with snmp and the utilisation never reaches above 1Mbps. The system is an IBM x336 with dual 2.8Ghz XEONs, 4gb ram and SCSI drives it has dual port broad com 10/100/1000 interfaces. If I monitor iostat/vmstat or top I see that the system averages around 90% idle. If I look at SAR for a long term performance report I see similar information. On the server it directly connects to the 6509 @ 100Mbps Full duplex there are no errors on the interface using the bcm5700 driver provided by broadcom. Since looking into the issue I initially thought that the problem was with the SSH but the problem appears to be more wide spread. If I do a rspan on the interface that the server is connected to I see that the Ethernet frames go both directions however If I do a tcpdump on the server I am not seeing the packets with the syntax 'tcpdump -vvv -n icmp'. The SSH sessions and the ICMP return packets not being seen I am sure are related. In all cases the ports have been forced to 100-FULL Duplex and I have tested at several other port link speeds and modes without any success. One thing that is slightly different to many configurations is that the gateway is running CatOS, it then routes in hybrid mode and has VRRP configured on the device's gateway. However I do not see why that would cause any problems as the routing is not flipping and there are no layer 2/3 issues that I can detect. What I am trying to diagnose/determine is why the Linux box would not be seeing the packets returning even though we see them on an rspan as leaving the switch interface. There are no iptables rules on the server at all and from what I can see the sysctl stuff is all setup correctly. When I turn on the debug syslog I still get no error messages or notifications. Where should I be looking now and is there any advice that you guys can give me? Regards, Jimmy. ===System Information=== /sbin/lspci -v 00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a) Subsystem: IBM: Unknown device 02dd Flags: bus master, fast devsel, latency 0 Memory at <ignored> (32-bit, non-prefetchable) [size=4K] Capabilities: <available only to root> 00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting Register (rev 0a) Subsystem: IBM: Unknown device 02dd Flags: fast devsel 00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port A0 (rev 0a) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=02, subordinate=04, sec-latency=0 Capabilities: <available only to root> 00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port B0 (rev 0a) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 Memory behind bridge: ce000000-cfffffff Capabilities: <available only to root> 00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port B1 (rev 0a) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 Memory behind bridge: cc000000-cdffffff Capabilities: <available only to root> 00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port C0 (rev 0a) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=07, subordinate=09, sec-latency=0 I/O behind bridge: 00004000-0000ffff Memory behind bridge: ca000000-cbffffff Capabilities: <available only to root> 00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended Configuration Registers (rev 0a) Subsystem: IBM: Unknown device 02dd Flags: fast devsel 00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 (rev 02) (prog-if 00 [UHCI]) Subsystem: IBM: Unknown device 02dd Flags: bus master, medium devsel, latency 0, IRQ 16 I/O ports at 2200 [size=32] 00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 (rev 02) (prog-if 00 [UHCI]) Subsystem: IBM: Unknown device 02dd Flags: bus master, medium devsel, latency 0, IRQ 19 I/O ports at 2600 [size=32] 00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) (prog-if 20 [EHCI]) Subsystem: IBM: Unknown device 02dd Flags: bus master, medium devsel, latency 0, IRQ 23 Memory at f0000000 (32-bit, non-prefetchable) [size=1K] Capabilities: <available only to root> 00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev c2) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=32 I/O behind bridge: 00003000-00003fff Memory behind bridge: de000000-dfffffff Prefetchable memory behind bridge: d0000000-ddffffff 00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02) Flags: bus master, medium devsel, latency 0 00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100 Storage Controller (rev 02) (prog-if 8a [Master SecP PriP]) Subsystem: IBM: Unknown device 02dd Flags: bus master, medium devsel, latency 0, IRQ 17 I/O ports at <ignored> I/O ports at <ignored> I/O ports at <ignored> I/O ports at <ignored> I/O ports at 0480 [size=16] Memory at c8000000 (32-bit, non-prefetchable) [size=1K] 00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02) Subsystem: IBM: Unknown device 02dd Flags: medium devsel, IRQ 17 I/O ports at 0440 [size=32] 01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE] (prog-if 00 [VGA]) Subsystem: IBM: Unknown device 02c8 Flags: bus master, stepping, medium devsel, latency 64, IRQ 20 Memory at d0000000 (32-bit, prefetchable) [size=128M] I/O ports at 3000 [size=256] Memory at dfff0000 (32-bit, non-prefetchable) [size=64K] Expansion ROM at <unassigned> [disabled] [size=128K] Capabilities: <available only to root> 02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=03, subordinate=03, sec-latency=64 Capabilities: <available only to root> 02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=04, subordinate=04, sec-latency=64 Capabilities: <available only to root> 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 01) Subsystem: IBM: Unknown device 02c6 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at cfff0000 (64-bit, non-prefetchable) [size=64K] Capabilities: <available only to root> 06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 01) Subsystem: IBM: Unknown device 02c6 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at cdff0000 (64-bit, non-prefetchable) [size=64K] Capabilities: <available only to root> 07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=07, secondary=08, subordinate=08, sec-latency=64 Memory behind bridge: ca000000-cbffffff Capabilities: <available only to root> 07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=07, secondary=09, subordinate=09, sec-latency=64 I/O behind bridge: 00004000-0000ffff Capabilities: <available only to root> 08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07) Subsystem: IBM: Unknown device 028e Flags: bus master, stepping, 66Mhz, medium devsel, latency 64, IRQ 38 Memory at cbfff000 (32-bit, non-prefetchable) [size=4K] Expansion ROM at <unassigned> [disabled] [size=512K] Capabilities: <available only to root> Module Size Used by Tainted: P bcm5700 105380 2 ide-cd 34016 0 (autoclean) cdrom 32608 0 (autoclean) [ide-cd] sg 37260 0 (autoclean) autofs4 16984 0 (autoclean) (unused) audit 90584 2 basp 62020 0 (unused) microcode 6912 0 (autoclean) keybdev 2976 0 (unused) mousedev 5656 0 (unused) hid 22308 0 (unused) input 6176 0 [keybdev mousedev hid] ehci-hcd 20776 0 (unused) usb-uhci 26988 0 (unused) usbcore 81088 1 [hid ehci-hcd usb-uhci] ext3 89960 4 jbd 55060 4 [ext3] ips 45252 5 sd_mod 13712 10 scsi_mod 114344 3 [sg ips sd_mod] /sbin/mii-tool eth0: 100 Mbit, full duplex, link ok eth1: 100 Mbit, full duplex, link ok -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
