RE: Issues with Performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Jimmy,
   This looks like a tough question and you have done a lot of
troubleshooting already. One thing that I noticed you did not mention was
checking the physical cable. Is it possible your CAT 5, 5e, or 6 cable is
misrated or damaged? Also, if this is going through the wall you might try
an alternative cable run. I know this seems very basic, but to me it sounds
like you are running in half duplex mode even though you are set to full
duplex. If it is imposible to try a different cable run then call up someone
with some good CAT 5 testing equipment, and have them check it out. The guys
with HP or Agilent testers are usually the best, but maybe that is more so
in my area.

Job Cacka

-----Original Message-----
From: redhat-list-bounces@xxxxxxxxxx
[mailto:redhat-list-bounces@xxxxxxxxxx]On Behalf Of Jimmy Stewpot
Sent: Monday, February 13, 2006 7:45 AM
To: redhat-list@xxxxxxxxxx
Subject: Issues with Performance


Hello,

I am currently trying to diagnose a particularly strange issue with a
Redhat Enterprise Linux v3.0 Update 5 system. The particular issue is as
follows.

I am attempting to run an ICMP monitoring package from the system to
monitor all of the devices around a network. (circa 10,000). When the
ICMP monitoring software is enabled the server becomes unusable via SSH.
By saying unusable I mean the system is very laggy often taking several
minutes to respond to a command. However if I am on the system console
the system is still usable and responds with NO delay.

Originally I suspected that the issue was related to system load or
network congestion. However I started to monitor the switch ports on the
catalyst 6509 ports with snmp and the utilisation never reaches above 1Mbps.

The system is an IBM x336 with dual 2.8Ghz XEONs, 4gb ram and SCSI
drives it has dual port broad com 10/100/1000 interfaces. If I monitor
iostat/vmstat or top I see that the system averages around 90% idle. If
I look at SAR for a long term performance report I see similar information.

On the server it directly connects to the 6509 @ 100Mbps Full duplex
there are no errors on the interface using the bcm5700 driver provided
by broadcom. Since looking into the issue I initially thought that the
problem was with the SSH but the problem appears to be more wide spread.

If I do a rspan on the interface that the server is connected to I see
that the Ethernet frames go both directions however If I do a tcpdump on
the server I am not seeing the packets with the syntax

  'tcpdump -vvv -n icmp'.

  The SSH sessions and the ICMP return packets not being seen I am sure
are related. In all cases the ports have been forced to 100-FULL Duplex
and I have tested at several other port link speeds and modes without
any success.

One thing that is slightly different to many configurations is that the
gateway is running CatOS, it then routes in hybrid mode and has VRRP
configured on the device's gateway. However I do not see why that would
cause any problems as the routing is not flipping and there are no layer
2/3 issues that I can detect.

What I am  trying to diagnose/determine is why the Linux box would not
be seeing the packets returning even though we see them on an rspan as
leaving the switch interface. There are no iptables rules on the server
at all and from what I can see the sysctl stuff is all setup correctly.
When I turn on the debug syslog I still get no error messages or
notifications.

Where should I be looking now and is there any advice that you guys can
give me?

Regards,

Jimmy.




===System Information===

/sbin/lspci -v
00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a)
         Subsystem: IBM: Unknown device 02dd
         Flags: bus master, fast devsel, latency 0
         Memory at <ignored> (32-bit, non-prefetchable) [size=4K]
         Capabilities: <available only to root>

00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting
Register (rev 0a)
         Subsystem: IBM: Unknown device 02dd
         Flags: fast devsel

00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
A0 (rev 0a) (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=00, secondary=02, subordinate=04, sec-latency=0
         Capabilities: <available only to root>

00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B0 (rev 0a) (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
         Memory behind bridge: ce000000-cfffffff
         Capabilities: <available only to root>

00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B1 (rev 0a) (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
         Memory behind bridge: cc000000-cdffffff
         Capabilities: <available only to root>

00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
C0 (rev 0a) (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
         I/O behind bridge: 00004000-0000ffff
         Memory behind bridge: ca000000-cbffffff
         Capabilities: <available only to root>

00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended
Configuration Registers (rev 0a)
         Subsystem: IBM: Unknown device 02dd
         Flags: fast devsel

00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
(rev 02) (prog-if 00 [UHCI])
         Subsystem: IBM: Unknown device 02dd
         Flags: bus master, medium devsel, latency 0, IRQ 16
         I/O ports at 2200 [size=32]

00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
(rev 02) (prog-if 00 [UHCI])
         Subsystem: IBM: Unknown device 02dd
         Flags: bus master, medium devsel, latency 0, IRQ 19
         I/O ports at 2600 [size=32]

00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI
Controller (rev 02) (prog-if 20 [EHCI])
         Subsystem: IBM: Unknown device 02dd
         Flags: bus master, medium devsel, latency 0, IRQ 23
         Memory at f0000000 (32-bit, non-prefetchable) [size=1K]
         Capabilities: <available only to root>

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI
Bridge (rev c2) (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
         I/O behind bridge: 00003000-00003fff
         Memory behind bridge: de000000-dfffffff
         Prefetchable memory behind bridge: d0000000-ddffffff

00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
         Flags: bus master, medium devsel, latency 0

00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100
Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
         Subsystem: IBM: Unknown device 02dd
         Flags: bus master, medium devsel, latency 0, IRQ 17
         I/O ports at <ignored>
         I/O ports at <ignored>
         I/O ports at <ignored>
         I/O ports at <ignored>
         I/O ports at 0480 [size=16]
         Memory at c8000000 (32-bit, non-prefetchable) [size=1K]

00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
         Subsystem: IBM: Unknown device 02dd
         Flags: medium devsel, IRQ 17
         I/O ports at 0440 [size=32]

01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY
[Radeon 7000/VE] (prog-if 00 [VGA])
         Subsystem: IBM: Unknown device 02c8
         Flags: bus master, stepping, medium devsel, latency 64, IRQ 20
         Memory at d0000000 (32-bit, prefetchable) [size=128M]
         I/O ports at 3000 [size=256]
         Memory at dfff0000 (32-bit, non-prefetchable) [size=64K]
         Expansion ROM at <unassigned> [disabled] [size=128K]
         Capabilities: <available only to root>

02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00
[Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
         Capabilities: <available only to root>

02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00
[Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=02, secondary=04, subordinate=04, sec-latency=64
         Capabilities: <available only to root>

05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
         Subsystem: IBM: Unknown device 02c6
         Flags: bus master, fast devsel, latency 0, IRQ 16
         Memory at cfff0000 (64-bit, non-prefetchable) [size=64K]
         Capabilities: <available only to root>

06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
         Subsystem: IBM: Unknown device 02c6
         Flags: bus master, fast devsel, latency 0, IRQ 16
         Memory at cdff0000 (64-bit, non-prefetchable) [size=64K]
         Capabilities: <available only to root>

07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
         Memory behind bridge: ca000000-cbffffff
         Capabilities: <available only to root>

07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary=07, secondary=09, subordinate=09, sec-latency=64
         I/O behind bridge: 00004000-0000ffff
         Capabilities: <available only to root>

08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07)
         Subsystem: IBM: Unknown device 028e
         Flags: bus master, stepping, 66Mhz, medium devsel, latency 64,
IRQ 38
         Memory at cbfff000 (32-bit, non-prefetchable) [size=4K]
         Expansion ROM at <unassigned> [disabled] [size=512K]
         Capabilities: <available only to root>

Module                  Size  Used by    Tainted: P
bcm5700               105380   2
ide-cd                 34016   0 (autoclean)
cdrom                  32608   0 (autoclean) [ide-cd]
sg                     37260   0 (autoclean)
autofs4                16984   0 (autoclean) (unused)
audit                  90584   2
basp                   62020   0 (unused)
microcode               6912   0 (autoclean)
keybdev                 2976   0 (unused)
mousedev                5656   0 (unused)
hid                    22308   0 (unused)
input                   6176   0 [keybdev mousedev hid]
ehci-hcd               20776   0 (unused)
usb-uhci               26988   0 (unused)
usbcore                81088   1 [hid ehci-hcd usb-uhci]
ext3                   89960   4
jbd                    55060   4 [ext3]
ips                    45252   5
sd_mod                 13712  10
scsi_mod              114344   3 [sg ips sd_mod]

/sbin/mii-tool
eth0: 100 Mbit, full duplex, link ok
eth1: 100 Mbit, full duplex, link ok


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

-- 
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

[Index of Archives]     [CentOS]     [Kernel Development]     [PAM]     [Fedora Users]     [Red Hat Development]     [Big List of Linux Books]     [Linux Admin]     [Gimp]     [Asterisk PBX]     [Yosemite News]     [Red Hat Crash Utility]


  Powered by Linux