(Sorry about the duplicated mails, my mailer stinks.)
Hi ext3-gurus,
I was doing some testing of our 100 Mbit ethernet at work, and found a
weird problem which appears to be ext3 releated as far as I can tell.
The test is very simple. I use ncftp to connect to another machine
running vsftpd and download a very large file over the network. The
network is switched, and I get 8-10 MB/s transfers usually.
Now, if I download to /dev/null ("get -z bigfile /dev/null"), the
transfer is fast and steady. Same goes if I download to an ext2
partition. When I mount the partition as ext3 (with the default
data=ordered or data=writeback), ncftp stalls badly (progress display
stops updating for several seconds) and transfer rate drops to 4-5 MB/s.
vmstat traces show that the number of interrupts per second drop from
8-9K to a couple of hundred when ext3 begins writing out data.
I have tested on several different machines and configuration, the
pattern seems to be the same. The distribution is either RH 7.3 or RH
8.0, and I've tried both the latest errata (2.4.18-27) and the latest
RawHide (2.4.20-2.54) kernels.
While writing this report, I realized that I had only run my test on
machines with IDE partitions and 3Com NICs. For kicks, I tried on a
completely different machine with a SCSI disk and Intel NIC. Curiously,
this machine did not exhibit any stalls, even with ext3 in data=ordered
mode.
I've attached a longish file with vmstat traces of all the different
test cases, as well as selected dmesg output for the two test machines.
Any ideas?
Thanks,
Troels Walsted Hansen
Machine A, dual CPU Celeron/433 MHz
Linux version 2.4.20-2.54smp (bhcompile@porky.devel.redhat.com) (gcc version 3.2.2 2003
0217 (Red Hat Linux 8.0 3.2.2-2)) #1 SMP Sat Feb 22 08:18:56 EST 2003
PIIX4: IDE controller at PCI slot 00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:pio
hda: Maxtor 4G120J6, ATA DISK drive
hda: 240121728 sectors (122942 MB) w/2048KiB Cache, CHS=14946/255/63, UDMA(33)
3c59x: Donald Becker and others. www.scyld.com/network/vortex.html
See Documentation/networking/vortex.txt
00:0d.0: 3Com PCI 3c905C Tornado at 0xa800. Vers LK1.1.18-ac
00:50:da:23:61:29, IRQ 17
product code 5957 rev 00.13 date 08-30-99
8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
MII transceiver found at address 24, status 782d.
Enabling bus-master transmits and whole-frame receives.
00:0d.0: scatter/gather enabled. h/w checksums enabled
eth0: negotiated 100baseTx-HD, link ok
product info: vendor 00:10:18, model 23 rev 6
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-HD 10baseT-HD
(writing to /dev/null)
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 239064 141232 62536 0 0 0 0 137 44 0 1 99
0 0 0 0 239064 141232 62536 0 0 0 0 105 10 0 0 100
0 0 0 0 239064 141240 62536 0 0 0 16 120 48 0 0 100
0 0 0 0 239064 141240 62536 0 0 0 0 115 31 0 0 100
0 0 0 0 239064 141240 62536 0 0 0 0 4116 5236 0 13 87
1 0 0 0 239064 141240 62536 0 0 0 0 9180 12046 2 29 68
0 0 0 0 239064 141240 62536 0 0 0 0 9395 12309 3 27 70
8 0 0 0 239016 141252 62536 0 0 0 32 8820 11329 2 23 75
4 0 0 0 236756 141252 62536 0 0 0 280 8475 5687 53 47 0
1 0 0 0 234900 141260 62536 0 0 0 172 8577 6478 47 46 7
1 0 0 0 234900 141260 62536 0 0 0 0 9330 8781 26 32 42
0 0 0 0 238524 141260 62536 0 0 0 0 9239 10636 12 31 57
0 0 0 0 238524 141260 62536 0 0 0 48 9238 12097 2 25 73
0 0 0 0 238524 141260 62536 0 0 0 0 9253 12150 0 25 75
0 0 0 0 238524 141260 62536 0 0 0 0 9227 12092 1 24 75
0 0 0 0 238524 141260 62536 0 0 0 0 9414 12337 2 29 69
1 0 0 0 238524 141260 62536 0 0 0 0 9228 12121 2 22 75
0 0 0 0 238528 141260 62536 0 0 0 0 9402 12345 1 31 68
0 0 0 0 238528 141260 62536 0 0 0 0 9261 12133 1 27 72
0 0 0 0 238528 141260 62536 0 0 0 0 9291 12183 3 26 71
0 0 0 0 238528 141260 62536 0 0 0 0 9347 12272 3 23 74
1 0 0 0 238528 141260 62536 0 0 0 0 9418 12362 0 25 74
/dev/hda5 on /home type ext3 (rw)
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 276564 141540 48732 0 0 0 0 103 10 0 0 100
0 0 0 0 276564 141540 48732 0 0 0 0 105 10 0 0 100
0 0 0 0 276564 141540 48732 0 0 0 0 103 12 0 2 98
0 0 0 0 276564 141540 48732 0 0 0 0 109 17 0 0 100
1 0 0 0 276564 141540 50288 0 0 0 0 1714 90 1 8 91
1 0 0 0 268184 141540 58812 0 0 0 0 8372 299 2 42 56
1 0 0 0 259668 141540 67328 0 0 0 0 8386 311 1 40 59
1 0 0 0 251188 141540 75808 0 0 0 0 8345 323 2 42 56
0 1 1 0 248032 141540 78964 0 0 0 8120 3275 130 1 16 83
0 1 1 0 248032 141540 78964 0 0 0 10240 251 17 0 1 99
1 0 1 0 245676 141540 81320 0 0 0 8128 2475 115 0 11 89
1 0 0 0 238948 141548 89820 0 0 0 3824 8413 348 1 43 56
0 0 0 0 230428 141548 98340 0 0 0 0 8432 331 2 39 58
0 1 1 0 227752 141548 101016 0 0 0 8288 2916 113 2 12 86
0 1 1 0 227752 141548 101016 0 0 0 10240 261 17 0 2 98
1 0 0 0 224688 141556 104072 0 0 0 3708 3166 136 1 16 83
0 0 1 0 216024 141556 112736 0 0 0 0 8640 340 1 42 57
1 0 0 0 207256 141556 121504 0 0 0 0 8722 357 0 50 50
0 1 1 0 205204 141556 123556 0 0 0 8444 2296 100 0 11 89
0 0 1 0 196836 141556 131924 0 0 0 6592 8254 322 2 42 56
1 0 1 0 188044 141556 140720 0 0 0 6088 8709 352 1 49 50
0 1 1 0 183884 141564 144868 0 0 0 7860 4258 143 0 25 75
0 1 1 0 183884 141564 144868 0 0 0 10244 265 21 0 1 99
0 0 0 0 182992 141580 145744 0 0 0 5116 1004 72 0 7 93
0 0 0 0 174440 141580 154296 0 0 0 0 8480 320 3 44 53
1 0 0 0 165888 141580 162848 0 0 0 0 8514 332 1 45 54
0 0 0 0 157312 141580 171424 0 0 0 0 8525 315 0 42 58
1 0 0 0 148964 141580 179776 0 0 0 0 8319 336 1 45 53
0 1 1 0 141844 141580 186896 0 0 0 6104 7201 287 0 41 59
1 1 1 0 141844 141580 186896 0 0 0 8196 252 19 0 1 99
0 1 1 0 141844 141580 186896 0 0 0 8192 241 19 0 0 99
/dev/hda5 on /home type ext3 (rw,data=writeback)
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 309448 104952 53976 0 0 0 0 135 38 0 0 100
0 0 0 0 309448 104960 53976 0 0 0 12 105 15 0 0 100
0 0 0 0 309448 104960 53976 0 0 0 0 103 7 0 0 100
0 0 0 0 309448 104960 53976 0 0 0 0 109 17 0 0 100
1 0 0 0 304124 104980 59792 0 0 8 0 5884 275 0 27 73
1 0 0 0 295848 104988 68060 0 0 0 0 8195 337 3 38 59
1 0 0 0 287780 105004 76120 0 0 0 60 7915 328 1 42 57
1 0 0 0 279776 105012 84112 0 0 0 0 7943 343 0 38 62
1 0 0 0 273240 105020 92416 0 0 0 0 8234 336 2 35 62
2 0 1 0 262852 105028 100892 0 0 0 40 8070 333 17 48 36
0 0 0 0 254880 105036 108956 0 0 0 196 7928 341 9 38 53
1 0 0 0 246832 105052 116988 0 0 0 72 7924 350 2 34 63
1 0 0 0 238632 105064 125172 0 0 4 0 8100 314 2 39 59
1 0 0 0 230372 105072 133424 0 0 0 0 8172 327 2 41 57
1 0 0 0 224256 105080 141312 0 0 0 0 7815 344 1 42 56
1 0 1 0 215708 105088 149856 0 0 0 4088 8496 344 1 47 52
0 1 2 0 214668 105092 150896 0 0 0 1480 1173 54 0 6 94
0 1 2 0 214668 105092 150896 0 0 0 4096 198 11 0 1 99
0 1 2 0 214668 105092 150896 0 0 0 7828 233 18 0 1 99
1 0 2 0 207568 105104 157980 0 0 0 8516 7046 284 0 35 65
0 0 2 0 199272 105112 166268 0 0 0 8376 8261 316 1 43 55
0 1 2 0 197952 105116 167588 0 0 0 8180 1519 70 0 8 92
0 1 2 0 197952 105116 167588 0 0 0 8192 250 15 0 0 100
1 0 0 0 194852 105124 170676 0 0 0 2616 3131 158 0 17 83
1 0 0 0 186416 105132 179108 0 0 0 0 8326 343 1 45 54
1 0 0 0 178204 105148 187304 0 0 0 52 8131 322 1 41 57
1 0 0 0 169848 105156 195648 0 0 0 0 8279 343 0 44 56
1 0 1 0 161356 105164 204132 0 0 0 7604 8423 323 1 42 56
1 0 1 0 153000 105172 212480 0 0 0 10240 8328 321 1 43 55
1 0 1 0 145420 105180 220052 0 0 0 8448 7556 296 0 38 62
0 1 2 0 138456 105192 227008 0 0 0 8260 6992 279 2 37 61
1 1 2 0 138456 105192 227008 0 0 0 8192 262 15 0 1 99
1 0 2 0 137916 105196 227544 0 0 0 10180 717 39 0 4 95
0 0 2 0 129708 105204 235740 0 0 0 8088 8153 320 2 40 58
0 0 2 0 122272 105212 243168 0 0 0 7396 7383 333 1 40 59
0 1 2 0 115728 105220 249704 0 0 0 8176 6598 273 0 35 65
0 0 0 0 114716 105228 250708 0 0 0 5460 1162 57 0 8 92
1 0 0 0 106460 105240 258952 0 0 4 0 8133 319 0 40 59
1 0 0 0 98328 105248 267072 0 0 0 0 8019 304 1 41 57
/dev/hda5 on /home type ext2 (rw)
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 309376 105180 53988 0 0 0 0 104 10 0 0 100
0 0 0 0 309376 105180 53988 0 0 0 0 103 8 0 0 100
0 0 0 0 309376 105180 53988 0 0 0 0 103 8 0 0 100
0 0 0 0 305868 105180 58092 0 0 0 0 4494 3680 0 22 78
1 0 0 0 299084 105180 66400 0 0 0 0 8934 7188 1 40 59
1 0 0 0 290840 105180 74644 0 0 0 0 8858 7314 0 36 63
1 0 0 0 282532 105180 82952 0 0 0 0 8934 7298 2 32 66
1 0 0 0 274356 105180 91128 0 0 0 0 8782 6965 1 36 63
1 0 0 0 265844 105180 99640 0 0 0 0 9130 7164 2 26 72
1 0 0 0 257564 105180 107920 0 0 0 0 8908 7151 2 35 63
1 0 0 0 249808 105180 115680 0 0 0 0 8351 6766 3 33 64
0 0 0 0 242368 105180 123116 0 0 0 0 7935 6391 1 34 65
0 0 0 0 234176 105180 131308 0 0 0 0 8798 6940 3 27 70
1 0 0 0 225964 105180 139520 0 0 0 0 8826 7004 4 31 65
1 0 1 0 217608 105180 147876 0 0 0 3780 9057 6799 1 39 60
0 0 1 0 209116 105180 156368 0 0 0 1984 9094 7258 2 38 60
1 0 1 0 200788 105180 164696 0 0 0 2048 8986 6779 3 36 61
0 0 1 0 192108 105180 173376 0 0 0 10180 9358 6214 1 43 56
1 0 1 0 183488 105180 181996 0 0 0 8192 9335 6294 1 31 68
3 1 2 0 171164 105180 190356 0 0 0 8196 8348 2556 36 48 16
2 1 2 0 163356 105180 198800 0 0 0 8192 8559 2794 32 52 16
1 1 3 0 154104 105180 206976 0 0 0 8140 8674 4186 20 38 42
1 1 2 0 144756 105180 215428 0 0 0 8192 8967 4326 18 37 45
2 1 2 0 135472 105180 224024 0 0 0 10184 8936 3699 26 41 33
0 1 2 0 131196 105180 232488 0 0 0 8184 8965 4214 19 40 41
1 1 2 0 122672 105180 241020 0 0 0 8192 9198 6283 3 38 59
1 1 2 0 114208 105180 249484 0 0 0 8360 9128 6392 0 38 62
1 1 2 0 105772 105180 257916 0 0 0 10064 9134 6236 1 42 57
1 1 2 0 97276 105180 266412 0 0 0 8192 9148 6229 1 42 57
1 1 2 0 88556 105180 275136 0 0 0 8136 9410 6233 2 36 62
1 1 2 0 80076 105180 283612 0 0 0 10180 9141 6344 1 33 66
1 1 2 0 71292 105180 292396 0 0 0 8140 9515 6794 1 42 56
0 1 2 0 62784 105188 300896 0 0 0 8192 9199 6527 1 39 60
1 1 2 0 54220 105196 309452 0 0 0 10180 9213 6296 2 36 62
0 1 2 0 45544 105204 318120 0 0 0 8196 9361 6386 1 33 66
1 1 2 0 39100 105212 326340 0 0 0 10240 8940 6087 2 39 58
1 1 2 0 30488 105220 334944 0 0 0 8136 9275 6387 2 38 59
1 1 2 0 21768 105228 343660 0 0 0 8132 9341 6032 0 38 62
0 1 2 0 16552 105232 348900 0 0 0 10012 5679 3054 0 32 67
0 1 3 0 8400 105244 357428 0 0 0 8132 9196 5911 0 42 58
0 1 3 0 6420 105252 359516 0 0 0 9896 9200 4847 1 40 58
1 1 3 0 6388 105260 359612 0 0 0 7228 9148 4637 0 39 61
1 1 3 0 6388 105260 359612 0 0 0 5036 2061 855 1 7 93
1 1 2 0 6524 105268 359508 0 0 4 5316 4940 2752 0 25 75
0 1 2 0 6524 105276 359496 0 0 0 8296 9370 5397 1 40 58
1 1 4 0 6396 105288 359616 0 0 0 8132 9355 5343 2 34 64
1 1 3 0 6384 105300 359564 0 0 0 11736 9179 4738 1 31 68
1 1 3 0 6388 105304 359568 0 0 0 8996 4644 2073 1 20 78
0 1 3 0 6388 105308 359576 0 0 0 9088 3744 1815 0 13 86
1 1 3 0 6392 105312 359584 0 0 0 6896 4339 1952 0 20 80
0 0 2 0 6396 105320 359576 0 0 0 7440 9061 5290 1 38 61
Machine B, single CPU P3/600 MHz
Linux version 2.4.18-26.8.0 (bhcompile@daffy.perf.redhat.com) (gcc version 3.2 20020903
(Red Hat Linux 8.0 3.2-7)) #1 Mon Feb 24 10:21:42 EST 2003
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.8
<Adaptec aic7899 Ultra160 SCSI adapter>
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
blk: queue c1503a14, I/O limit 4095Mb (mask 0xffffffff)
Vendor: SEAGATE Model: ST318404LC Rev: 0002
Type: Direct-Access ANSI SCSI revision: 03
eth0: OEM i82557/i82558 10/100 Ethernet, 00:D0:B7:60:0E:5A, IRQ 10.
Receiver lock-up bug exists -- enabling work-around.
Board assembly 751110-002, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x24c9f043).
Receiver lock-up workaround activated.
eth0: negotiated 100baseTx-FD flow-control, link ok
product info: Intel 82555 rev 0
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
/dev/sda2 on / type ext3 (rw)
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 5300 92668 860 19732 0 0 0 0 12753 16083 2 63 35
1 0 0 5300 80964 872 31272 0 0 0 0 12738 16079 4 64 32
1 0 1 5300 69508 888 42568 0 0 4 34448 12441 15390 3 64 33
1 0 0 5300 57960 908 54096 0 0 0 84 12733 15676 4 64 32
1 0 0 5300 46404 920 65640 0 0 0 0 12746 16069 3 62 35
1 0 0 5300 34900 928 77132 0 0 0 12288 12683 15849 4 64 32
1 0 0 5300 23256 940 88616 0 0 0 12288 12680 15813 3 64 33
1 0 1 5300 11796 956 99916 0 0 4 32472 12448 15420 5 63 32
1 0 0 5300 2172 916 109292 0 0 0 92 12692 15607 3 67 30
1 0 0 5300 2172 928 109636 0 0 0 2044 12706 15820 4 63 33
1 0 0 5300 2296 936 109664 0 0 0 2044 12718 15915 3 65 32
1 0 0 5300 2172 948 109400 0 0 0 12288 12435 15333 3 62 35
1 0 1 5300 2888 960 108732 0 0 0 40356 12281 15078 5 65 30
1 0 0 5300 2588 980 109524 0 0 0 92 12318 15050 2 63 35
1 0 0 5300 2296 988 109832 0 0 0 0 12232 15306 4 60 36
1 0 0 5300 2268 1000 109752 0 0 0 14332 12656 15639 4 63 33
1 0 0 5300 2296 1012 109860 0 0 0 12288 12594 15599 2 68 30
1 0 1 5300 2172 1024 109552 0 0 0 29456 12453 15338 5 65 30
1 0 0 5300 2172 1040 109900 0 0 0 84 12257 15042 3 63 34
1 0 0 5300 2780 1056 109332 0 0 4 0 12361 15408 4 63 33
1 0 0 5300 2172 1064 109836 0 0 0 14332 12658 15645 4 62 34
1 0 0 5300 2928 1076 109188 0 0 0 12288 12592 15597 4 66 30
1 0 1 5300 2296 1084 109812 0 0 0 29660 12522 15417 2 66 32
1 0 0 5300 2172 1104 109536 0 0 0 92 12573 15339 4 65 31
1 0 0 5300 2172 1116 109344 0 0 0 4 12290 15356 2 61 37
1 0 0 5300 2172 1128 110044 0 0 0 14332 12674 15702 3 64 33
1 0 0 5300 2172 1140 109616 0 0 0 12288 12665 15581 5 64 31
1 0 1 5300 2652 1148 109436 0 0 0 29612 12200 15074 3 65 32
1 0 0 5300 2296 1168 109780 0 0 0 88 12550 15428 4 62 34
1 0 0 5300 2172 1180 109748 0 0 4 0 12584 15654 4 62 34
1 0 0 5300 2296 1192 109760 0 0 0 14332 12618 15607 3 64 33
1 0 0 5300 2172 1200 109596 0 0 0 12288 12662 15650 3 64 33
1 0 1 5300 2940 1212 109084 0 0 0 30056 12452 15406 3 67 30
1 0 0 5300 2296 1232 109700 0 0 0 92 12279 15021 4 63 33
2 0 0 5300 2180 1244 109124 0 0 0 0 12139 15131 5 60 35
1 0 0 5300 2712 1252 109268 0 0 0 14332 12696 15719 3 66 31
1 0 0 5300 2172 1264 109212 0 0 0 12288 12603 15566 3 63 34
1 0 1 5300 2944 1276 108608 0 0 0 29404 12441 15435 3 66 31
1 0 0 5300 2296 1300 109628 0 0 4 84 12641 15529 4 62 34
1 0 0 5300 2172 1312 108964 0 0 4 0 12512 15559 2 64 34
1 0 0 5300 2296 1320 109612 0 0 0 14332 12701 15711 3 65 32
1 0 0 5300 2172 1332 108956 0 0 0 12288 12588 15536 3 65 32
