OK, I think I have narrowed it down to aic7xxx driver, because when I specify CONFIG_SCSI_AIC7XXX_OLD, it does not regress. The regression occurs from 2.6.11 to 2.6.12 (from 17MB/s it goes down to 11MB/s) The regression is still there in 2.6.13-rc1 + the "speed fix" patch(discussed in the recent aic7xxx regression thread on linux-scsi) Recreate by running: aio-stress -O -o3 <1 GB testfile> Config options: CONFIG_SCSI_AIC7XXX=y CONFIG_AIC7XXX_CMDS_PER_DEVICE=32 CONFIG_AIC7XXX_RESET_DELAY_MS=15000 CONFIG_AIC7XXX_DEBUG_ENABLE=y CONFIG_AIC7XXX_DEBUG_MASK=0 CONFIG_AIC7XXX_REG_PRETTY_PRINT=y Comparing dmesg outputs On 2.6.11 scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36 <Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs (scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit) Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 Type: Direct-Access ANSI SCSI revision: 03 scsi0:A:0:0: Tagged Queuing enabled. Depth 32 (scsi0:A:1): 80.000MB/s transfers (40.000MHz, offset 63, 16bit) Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 ... On 2.6.12 scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36 <Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 Type: Direct-Access ANSI SCSI revision: 03 scsi0:A:0:0: Tagged Queuing enabled. Depth 32 target0:0:0: Beginning Domain Validation WIDTH IS 1 (scsi0:A:0): 6.600MB/s transfers (16bit) (scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit) target0:0:0: Ending Domain Validation Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 ... On 2.6.13-rc1 + "speed fix" scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36 <Adaptec aic7896/97 Ultra2 SCSI adapter> aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs target0:0:0: asynchronous. Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 Type: Direct-Access ANSI SCSI revision: 03 scsi0:A:0:0: Tagged Queuing enabled. Depth 32 target0:0:0: Beginning Domain Validation target0:0:0: wide asynchronous. target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 63) target0:0:0: Ending Domain Validation target0:0:1: asynchronous. Vendor: IBM-ESXS Model: ST318305LC !# Rev: B245 ... Regards Suparna On Wed, Jul 06, 2005 at 04:07:29PM +0530, Suparna Bhattacharya wrote: > On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote: > > On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote: > > > Has anyone else noticed major throughput regressions for random > > > reads/writes with aio-stress in 2.6.12 ? > > > Or have there been any other FS/IO regressions lately ? > > > > > > On one test system I see a degradation from around 17+ MB/s to 11MB/s > > > for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11 > > > to 2.6.12. It doesn't seem filesystem specific. Not good :( > > > > > > BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an > > > impact (I copied those back to my 2.6.11 tree and tried the runs with no > > > effect) So it is something else ... > > > > > > Ideas/thoughts/observations ? > > > > Lets try to narrow it down a bit: > > > > aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request > This doesn't regress - the problem really happens when we don't wait one > at a time. > > > at a time). This doesn't take the aio subsystem out of the picture, but it > > does make the block layer interaction more or less the same as non-aio > > benchmarks. If this is slow, I would suspect something in the block layer, > > and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression. > > > > If it doesn't regress, I would suspect something in the aio core. My first > > attempts at the context switch reduction patches caused this kind of > > regression. There was too much latency in sending the events up to userland. > > > > Other options: > > > > Try different elevators > > Still regresses (I tried noop instead of as) > > > Try O_SYNC aio random writes > > Try aio random reads > > Try buffers random reads > > Again all these end up waiting one at a time with mainline because it > forces buffered AIO to be synchronous, so we the regression doesn't > show up. But, when I apply my patches to make buffered fsAIO async, > so we aren't waiting one at a time -- there is a similar regression. > In fact it was this regression that made me go back and check if it > was happening with AIO-DIO as well. > > Regards > Suparna > > > > > -chris > > -- > Suparna Bhattacharya (suparna@xxxxxxxxxx) > Linux Technology Center > IBM Software Lab, India > > -- > To unsubscribe, send a message with 'unsubscribe linux-aio' in > the body to majordomo@xxxxxxxxxx For more info on Linux AIO, > see: http://www.kvack.org/aio/ > Don't email: <a href=mailto:"aart@xxxxxxxxx">aart@xxxxxxxxx</a> -- Suparna Bhattacharya (suparna@xxxxxxxxxx) Linux Technology Center IBM Software Lab, India - : send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html