Download to ext3 partition stalls

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Title: Message
Hi ext3-gurus,
 
I was doing some testing of our 100 Mbit ethernet at work, and found a weird problem which appears to be ext3 releated as far as I can tell.
 
The test is very simple. I use ncftp to connect to another machine running vsftpd and download a very large file over the network. The network is switched, and I get 8-10 MB/s transfers usually.
 
Now, if I download to /dev/null ("get -z bigfile /dev/null"), the transfer is fast and steady. Same goes if I download to an ext2 partition. When I mount the partition as ext3 (with the default data="" or data="" ncftp stalls badly (progress display stops updating for several seconds) and transfer rate drops to 4-5 MB/s.
 
vmstat traces show that the number of interrupts per second drop from 8-9K to a couple of hundred when ext3 begins writing out data.
 
I have tested on several different machines and configuration, the pattern seems to be the same. The distribution is either RH 7.3 or RH 8.0, and I've tried both the latest errata (2.4.18-27) and the latest RawHide (2.4.20-2.54) kernels.
 
While writing this report, I realized that I had only run my test on machines with IDE partitions and 3Com NICs. For kicks, I tried on a completely different machine with a SCSI disk and Intel NIC. Curiously, this machine did not exhibit any stalls, even with ext3 in data="" mode.
 
I've attached a longish file with vmstat traces of all the different test cases, as well as selected dmesg output for the two test machines.
 
Any ideas?
 
Thanks,
 
Troels Walsted Hansen
 
Machine A, dual CPU Celeron/433 MHz

Linux version 2.4.20-2.54smp (bhcompile@porky.devel.redhat.com) (gcc version 3.2.2 2003
0217 (Red Hat Linux 8.0 3.2.2-2)) #1 SMP Sat Feb 22 08:18:56 EST 2003

PIIX4: IDE controller at PCI slot 00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:pio

hda: Maxtor 4G120J6, ATA DISK drive
hda: 240121728 sectors (122942 MB) w/2048KiB Cache, CHS=14946/255/63, UDMA(33)

3c59x: Donald Becker and others. www.scyld.com/network/vortex.html
See Documentation/networking/vortex.txt
00:0d.0: 3Com PCI 3c905C Tornado at 0xa800. Vers LK1.1.18-ac 
   00:50:da:23:61:29, IRQ 17
   product code 5957 rev 00.13 date 08-30-99
   8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
   MII transceiver found at address 24, status 782d.
   Enabling bus-master transmits and whole-frame receives.
00:0d.0: scatter/gather enabled. h/w checksums enabled

eth0: negotiated 100baseTx-HD, link ok
  product info: vendor 00:10:18, model 23 rev 6
  basic mode:   autonegotiation enabled
  basic status: autonegotiation complete, link ok
  capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
  advertising:  100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
  link partner: 100baseTx-HD 10baseT-HD

(writing to /dev/null)
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0      0 239064 141232  62536   0   0     0     0  137    44   0   1  99
 0  0  0      0 239064 141232  62536   0   0     0     0  105    10   0   0 100
 0  0  0      0 239064 141240  62536   0   0     0    16  120    48   0   0 100
 0  0  0      0 239064 141240  62536   0   0     0     0  115    31   0   0 100
 0  0  0      0 239064 141240  62536   0   0     0     0 4116  5236   0  13  87
 1  0  0      0 239064 141240  62536   0   0     0     0 9180 12046   2  29  68
 0  0  0      0 239064 141240  62536   0   0     0     0 9395 12309   3  27  70
 8  0  0      0 239016 141252  62536   0   0     0    32 8820 11329   2  23  75
 4  0  0      0 236756 141252  62536   0   0     0   280 8475  5687  53  47   0
 1  0  0      0 234900 141260  62536   0   0     0   172 8577  6478  47  46   7
 1  0  0      0 234900 141260  62536   0   0     0     0 9330  8781  26  32  42
 0  0  0      0 238524 141260  62536   0   0     0     0 9239 10636  12  31  57
 0  0  0      0 238524 141260  62536   0   0     0    48 9238 12097   2  25  73
 0  0  0      0 238524 141260  62536   0   0     0     0 9253 12150   0  25  75
 0  0  0      0 238524 141260  62536   0   0     0     0 9227 12092   1  24  75
 0  0  0      0 238524 141260  62536   0   0     0     0 9414 12337   2  29  69
 1  0  0      0 238524 141260  62536   0   0     0     0 9228 12121   2  22  75
 0  0  0      0 238528 141260  62536   0   0     0     0 9402 12345   1  31  68
 0  0  0      0 238528 141260  62536   0   0     0     0 9261 12133   1  27  72
 0  0  0      0 238528 141260  62536   0   0     0     0 9291 12183   3  26  71
 0  0  0      0 238528 141260  62536   0   0     0     0 9347 12272   3  23  74
 1  0  0      0 238528 141260  62536   0   0     0     0 9418 12362   0  25  74

/dev/hda5 on /home type ext3 (rw)
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0      0 276564 141540  48732   0   0     0     0  103    10   0   0 100
 0  0  0      0 276564 141540  48732   0   0     0     0  105    10   0   0 100
 0  0  0      0 276564 141540  48732   0   0     0     0  103    12   0   2  98
 0  0  0      0 276564 141540  48732   0   0     0     0  109    17   0   0 100
 1  0  0      0 276564 141540  50288   0   0     0     0 1714    90   1   8  91
 1  0  0      0 268184 141540  58812   0   0     0     0 8372   299   2  42  56
 1  0  0      0 259668 141540  67328   0   0     0     0 8386   311   1  40  59
 1  0  0      0 251188 141540  75808   0   0     0     0 8345   323   2  42  56
 0  1  1      0 248032 141540  78964   0   0     0  8120 3275   130   1  16  83
 0  1  1      0 248032 141540  78964   0   0     0 10240  251    17   0   1  99
 1  0  1      0 245676 141540  81320   0   0     0  8128 2475   115   0  11  89
 1  0  0      0 238948 141548  89820   0   0     0  3824 8413   348   1  43  56
 0  0  0      0 230428 141548  98340   0   0     0     0 8432   331   2  39  58
 0  1  1      0 227752 141548 101016   0   0     0  8288 2916   113   2  12  86
 0  1  1      0 227752 141548 101016   0   0     0 10240  261    17   0   2  98
 1  0  0      0 224688 141556 104072   0   0     0  3708 3166   136   1  16  83
 0  0  1      0 216024 141556 112736   0   0     0     0 8640   340   1  42  57
 1  0  0      0 207256 141556 121504   0   0     0     0 8722   357   0  50  50
 0  1  1      0 205204 141556 123556   0   0     0  8444 2296   100   0  11  89
 0  0  1      0 196836 141556 131924   0   0     0  6592 8254   322   2  42  56
 1  0  1      0 188044 141556 140720   0   0     0  6088 8709   352   1  49  50
 0  1  1      0 183884 141564 144868   0   0     0  7860 4258   143   0  25  75
 0  1  1      0 183884 141564 144868   0   0     0 10244  265    21   0   1  99
 0  0  0      0 182992 141580 145744   0   0     0  5116 1004    72   0   7  93
 0  0  0      0 174440 141580 154296   0   0     0     0 8480   320   3  44  53
 1  0  0      0 165888 141580 162848   0   0     0     0 8514   332   1  45  54
 0  0  0      0 157312 141580 171424   0   0     0     0 8525   315   0  42  58
 1  0  0      0 148964 141580 179776   0   0     0     0 8319   336   1  45  53
 0  1  1      0 141844 141580 186896   0   0     0  6104 7201   287   0  41  59
 1  1  1      0 141844 141580 186896   0   0     0  8196  252    19   0   1  99
 0  1  1      0 141844 141580 186896   0   0     0  8192  241    19   0   0  99

/dev/hda5 on /home type ext3 (rw,data=writeback)
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0      0 309448 104952  53976   0   0     0     0  135    38   0   0 100
 0  0  0      0 309448 104960  53976   0   0     0    12  105    15   0   0 100
 0  0  0      0 309448 104960  53976   0   0     0     0  103     7   0   0 100
 0  0  0      0 309448 104960  53976   0   0     0     0  109    17   0   0 100
 1  0  0      0 304124 104980  59792   0   0     8     0 5884   275   0  27  73
 1  0  0      0 295848 104988  68060   0   0     0     0 8195   337   3  38  59
 1  0  0      0 287780 105004  76120   0   0     0    60 7915   328   1  42  57
 1  0  0      0 279776 105012  84112   0   0     0     0 7943   343   0  38  62
 1  0  0      0 273240 105020  92416   0   0     0     0 8234   336   2  35  62
 2  0  1      0 262852 105028 100892   0   0     0    40 8070   333  17  48  36
 0  0  0      0 254880 105036 108956   0   0     0   196 7928   341   9  38  53
 1  0  0      0 246832 105052 116988   0   0     0    72 7924   350   2  34  63
 1  0  0      0 238632 105064 125172   0   0     4     0 8100   314   2  39  59
 1  0  0      0 230372 105072 133424   0   0     0     0 8172   327   2  41  57
 1  0  0      0 224256 105080 141312   0   0     0     0 7815   344   1  42  56
 1  0  1      0 215708 105088 149856   0   0     0  4088 8496   344   1  47  52
 0  1  2      0 214668 105092 150896   0   0     0  1480 1173    54   0   6  94
 0  1  2      0 214668 105092 150896   0   0     0  4096  198    11   0   1  99
 0  1  2      0 214668 105092 150896   0   0     0  7828  233    18   0   1  99
 1  0  2      0 207568 105104 157980   0   0     0  8516 7046   284   0  35  65
 0  0  2      0 199272 105112 166268   0   0     0  8376 8261   316   1  43  55
 0  1  2      0 197952 105116 167588   0   0     0  8180 1519    70   0   8  92
 0  1  2      0 197952 105116 167588   0   0     0  8192  250    15   0   0 100
 1  0  0      0 194852 105124 170676   0   0     0  2616 3131   158   0  17  83
 1  0  0      0 186416 105132 179108   0   0     0     0 8326   343   1  45  54
 1  0  0      0 178204 105148 187304   0   0     0    52 8131   322   1  41  57
 1  0  0      0 169848 105156 195648   0   0     0     0 8279   343   0  44  56
 1  0  1      0 161356 105164 204132   0   0     0  7604 8423   323   1  42  56
 1  0  1      0 153000 105172 212480   0   0     0 10240 8328   321   1  43  55
 1  0  1      0 145420 105180 220052   0   0     0  8448 7556   296   0  38  62
 0  1  2      0 138456 105192 227008   0   0     0  8260 6992   279   2  37  61
 1  1  2      0 138456 105192 227008   0   0     0  8192  262    15   0   1  99
 1  0  2      0 137916 105196 227544   0   0     0 10180  717    39   0   4  95
 0  0  2      0 129708 105204 235740   0   0     0  8088 8153   320   2  40  58
 0  0  2      0 122272 105212 243168   0   0     0  7396 7383   333   1  40  59
 0  1  2      0 115728 105220 249704   0   0     0  8176 6598   273   0  35  65
 0  0  0      0 114716 105228 250708   0   0     0  5460 1162    57   0   8  92
 1  0  0      0 106460 105240 258952   0   0     4     0 8133   319   0  40  59
 1  0  0      0  98328 105248 267072   0   0     0     0 8019   304   1  41  57

/dev/hda5 on /home type ext2 (rw)
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0      0 309376 105180  53988   0   0     0     0  104    10   0   0 100
 0  0  0      0 309376 105180  53988   0   0     0     0  103     8   0   0 100
 0  0  0      0 309376 105180  53988   0   0     0     0  103     8   0   0 100
 0  0  0      0 305868 105180  58092   0   0     0     0 4494  3680   0  22  78
 1  0  0      0 299084 105180  66400   0   0     0     0 8934  7188   1  40  59
 1  0  0      0 290840 105180  74644   0   0     0     0 8858  7314   0  36  63
 1  0  0      0 282532 105180  82952   0   0     0     0 8934  7298   2  32  66
 1  0  0      0 274356 105180  91128   0   0     0     0 8782  6965   1  36  63
 1  0  0      0 265844 105180  99640   0   0     0     0 9130  7164   2  26  72
 1  0  0      0 257564 105180 107920   0   0     0     0 8908  7151   2  35  63
 1  0  0      0 249808 105180 115680   0   0     0     0 8351  6766   3  33  64
 0  0  0      0 242368 105180 123116   0   0     0     0 7935  6391   1  34  65
 0  0  0      0 234176 105180 131308   0   0     0     0 8798  6940   3  27  70
 1  0  0      0 225964 105180 139520   0   0     0     0 8826  7004   4  31  65
 1  0  1      0 217608 105180 147876   0   0     0  3780 9057  6799   1  39  60
 0  0  1      0 209116 105180 156368   0   0     0  1984 9094  7258   2  38  60
 1  0  1      0 200788 105180 164696   0   0     0  2048 8986  6779   3  36  61
 0  0  1      0 192108 105180 173376   0   0     0 10180 9358  6214   1  43  56
 1  0  1      0 183488 105180 181996   0   0     0  8192 9335  6294   1  31  68
 3  1  2      0 171164 105180 190356   0   0     0  8196 8348  2556  36  48  16
 2  1  2      0 163356 105180 198800   0   0     0  8192 8559  2794  32  52  16
 1  1  3      0 154104 105180 206976   0   0     0  8140 8674  4186  20  38  42
 1  1  2      0 144756 105180 215428   0   0     0  8192 8967  4326  18  37  45
 2  1  2      0 135472 105180 224024   0   0     0 10184 8936  3699  26  41  33
 0  1  2      0 131196 105180 232488   0   0     0  8184 8965  4214  19  40  41
 1  1  2      0 122672 105180 241020   0   0     0  8192 9198  6283   3  38  59
 1  1  2      0 114208 105180 249484   0   0     0  8360 9128  6392   0  38  62
 1  1  2      0 105772 105180 257916   0   0     0 10064 9134  6236   1  42  57
 1  1  2      0  97276 105180 266412   0   0     0  8192 9148  6229   1  42  57
 1  1  2      0  88556 105180 275136   0   0     0  8136 9410  6233   2  36  62
 1  1  2      0  80076 105180 283612   0   0     0 10180 9141  6344   1  33  66
 1  1  2      0  71292 105180 292396   0   0     0  8140 9515  6794   1  42  56
 0  1  2      0  62784 105188 300896   0   0     0  8192 9199  6527   1  39  60
 1  1  2      0  54220 105196 309452   0   0     0 10180 9213  6296   2  36  62
 0  1  2      0  45544 105204 318120   0   0     0  8196 9361  6386   1  33  66
 1  1  2      0  39100 105212 326340   0   0     0 10240 8940  6087   2  39  58
 1  1  2      0  30488 105220 334944   0   0     0  8136 9275  6387   2  38  59
 1  1  2      0  21768 105228 343660   0   0     0  8132 9341  6032   0  38  62
 0  1  2      0  16552 105232 348900   0   0     0 10012 5679  3054   0  32  67
 0  1  3      0   8400 105244 357428   0   0     0  8132 9196  5911   0  42  58
 0  1  3      0   6420 105252 359516   0   0     0  9896 9200  4847   1  40  58
 1  1  3      0   6388 105260 359612   0   0     0  7228 9148  4637   0  39  61
 1  1  3      0   6388 105260 359612   0   0     0  5036 2061   855   1   7  93
 1  1  2      0   6524 105268 359508   0   0     4  5316 4940  2752   0  25  75
 0  1  2      0   6524 105276 359496   0   0     0  8296 9370  5397   1  40  58
 1  1  4      0   6396 105288 359616   0   0     0  8132 9355  5343   2  34  64
 1  1  3      0   6384 105300 359564   0   0     0 11736 9179  4738   1  31  68
 1  1  3      0   6388 105304 359568   0   0     0  8996 4644  2073   1  20  78
 0  1  3      0   6388 105308 359576   0   0     0  9088 3744  1815   0  13  86
 1  1  3      0   6392 105312 359584   0   0     0  6896 4339  1952   0  20  80
 0  0  2      0   6396 105320 359576   0   0     0  7440 9061  5290   1  38  61

Machine B, single CPU P3/600 MHz

Linux version 2.4.18-26.8.0 (bhcompile@daffy.perf.redhat.com) (gcc version 3.2 20020903
 (Red Hat Linux 8.0 3.2-7)) #1 Mon Feb 24 10:21:42 EST 2003

scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.8
        <Adaptec aic7899 Ultra160 SCSI adapter>
        aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

blk: queue c1503a14, I/O limit 4095Mb (mask 0xffffffff)
  Vendor: SEAGATE   Model: ST318404LC        Rev: 0002
  Type:   Direct-Access                      ANSI SCSI revision: 03

eth0: OEM i82557/i82558 10/100 Ethernet, 00:D0:B7:60:0E:5A, IRQ 10.
  Receiver lock-up bug exists -- enabling work-around.
  Board assembly 751110-002, Physical connectors present: RJ45
  Primary interface chip i82555 PHY #1.
  General self-test: passed.
  Serial sub-system self-test: passed.
  Internal registers self-test: passed.
  ROM checksum self-test: passed (0x24c9f043).
  Receiver lock-up workaround activated.

eth0: negotiated 100baseTx-FD flow-control, link ok
  product info: Intel 82555 rev 0
  basic mode:   autonegotiation enabled
  basic status: autonegotiation complete, link ok
  capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
  advertising:  100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
  link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control

/dev/sda2 on / type ext3 (rw)
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 1  0  0   5300  92668    860  19732   0   0     0     0 12753 16083   2  63  35
 1  0  0   5300  80964    872  31272   0   0     0     0 12738 16079   4  64  32
 1  0  1   5300  69508    888  42568   0   0     4 34448 12441 15390   3  64  33
 1  0  0   5300  57960    908  54096   0   0     0    84 12733 15676   4  64  32
 1  0  0   5300  46404    920  65640   0   0     0     0 12746 16069   3  62  35
 1  0  0   5300  34900    928  77132   0   0     0 12288 12683 15849   4  64  32
 1  0  0   5300  23256    940  88616   0   0     0 12288 12680 15813   3  64  33
 1  0  1   5300  11796    956  99916   0   0     4 32472 12448 15420   5  63  32
 1  0  0   5300   2172    916 109292   0   0     0    92 12692 15607   3  67  30
 1  0  0   5300   2172    928 109636   0   0     0  2044 12706 15820   4  63  33
 1  0  0   5300   2296    936 109664   0   0     0  2044 12718 15915   3  65  32
 1  0  0   5300   2172    948 109400   0   0     0 12288 12435 15333   3  62  35
 1  0  1   5300   2888    960 108732   0   0     0 40356 12281 15078   5  65  30
 1  0  0   5300   2588    980 109524   0   0     0    92 12318 15050   2  63  35
 1  0  0   5300   2296    988 109832   0   0     0     0 12232 15306   4  60  36
 1  0  0   5300   2268   1000 109752   0   0     0 14332 12656 15639   4  63  33
 1  0  0   5300   2296   1012 109860   0   0     0 12288 12594 15599   2  68  30
 1  0  1   5300   2172   1024 109552   0   0     0 29456 12453 15338   5  65  30
 1  0  0   5300   2172   1040 109900   0   0     0    84 12257 15042   3  63  34
 1  0  0   5300   2780   1056 109332   0   0     4     0 12361 15408   4  63  33
 1  0  0   5300   2172   1064 109836   0   0     0 14332 12658 15645   4  62  34
 1  0  0   5300   2928   1076 109188   0   0     0 12288 12592 15597   4  66  30
 1  0  1   5300   2296   1084 109812   0   0     0 29660 12522 15417   2  66  32
 1  0  0   5300   2172   1104 109536   0   0     0    92 12573 15339   4  65  31
 1  0  0   5300   2172   1116 109344   0   0     0     4 12290 15356   2  61  37
 1  0  0   5300   2172   1128 110044   0   0     0 14332 12674 15702   3  64  33
 1  0  0   5300   2172   1140 109616   0   0     0 12288 12665 15581   5  64  31
 1  0  1   5300   2652   1148 109436   0   0     0 29612 12200 15074   3  65  32
 1  0  0   5300   2296   1168 109780   0   0     0    88 12550 15428   4  62  34
 1  0  0   5300   2172   1180 109748   0   0     4     0 12584 15654   4  62  34
 1  0  0   5300   2296   1192 109760   0   0     0 14332 12618 15607   3  64  33
 1  0  0   5300   2172   1200 109596   0   0     0 12288 12662 15650   3  64  33
 1  0  1   5300   2940   1212 109084   0   0     0 30056 12452 15406   3  67  30
 1  0  0   5300   2296   1232 109700   0   0     0    92 12279 15021   4  63  33
 2  0  0   5300   2180   1244 109124   0   0     0     0 12139 15131   5  60  35
 1  0  0   5300   2712   1252 109268   0   0     0 14332 12696 15719   3  66  31
 1  0  0   5300   2172   1264 109212   0   0     0 12288 12603 15566   3  63  34
 1  0  1   5300   2944   1276 108608   0   0     0 29404 12441 15435   3  66  31
 1  0  0   5300   2296   1300 109628   0   0     4    84 12641 15529   4  62  34
 1  0  0   5300   2172   1312 108964   0   0     4     0 12512 15559   2  64  34
 1  0  0   5300   2296   1320 109612   0   0     0 14332 12701 15711   3  65  32
 1  0  0   5300   2172   1332 108956   0   0     0 12288 12588 15536   3  65  32

[Index of Archives]         [Linux RAID]     [Kernel Development]     [Red Hat Install]     [Video 4 Linux]     [Postgresql]     [Fedora]     [Gimp]     [Yosemite News]

  Powered by Linux