hi, Niklas,
On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> On Tue, Jan 07, 2025 at 04:27:44PM +0800, Oliver Sang wrote:
> > hi, Niklas,
> >
> > On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> > > On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > > > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > > > please let us know. thanks a lot!
> > > > > >
> > > > > > Yes, this looks like ahci. Thanks a lot!
> > > > >
> > > > > Did this ever get resolved?
> > > > >
> > > > > I haven't seen a patch that seems to address this.
> > > > >
> > > > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > > > reordering when issuing a batch of commands, my guess is that the problem
> > > > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > > > or block layer.
> > > >
> > > > I started looking into this before the holidays. blktrace shows perfectly
> > > > sequential writes without any reordering using ahci, directly on the
> > > > block device or using xfs and btrfs when using dd. I also started
> > > > looking into what the test does and got as far as checking out the
> > > > stress-ng source tree and looking at stress-aiol.c. AFAICS the default
> > > > submission does simple reads and writes using increasing offsets.
> > > > So if the test result isn't a fluke either the aio code does some
> > > > weird reordering or btrfs does.
> > > >
> > > > Oliver, did the test also show any interesting results on non-btrfs
> > > > setups?
> > > >
> > >
> > > One thing that came to mind.
> > > Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> > > the I/O scheduler to BFQ for single-queue HDDs.
> > >
> > > It could very well be the I/O scheduler that reorders.
> > >
> > > Oliver, which I/O scheduler are you using?
> > > $ cat /sys/block/sdb/queue/scheduler
> > > none mq-deadline kyber [bfq]
> >
> > while our test running:
> >
> > # cat /sys/block/sdb/queue/scheduler
> > none [mq-deadline] kyber bfq
>
> The stddev numbers you showed is all over the place, so are we certain
> if this is a regression caused by commit e70c301faece ("block:
> don't reorder requests in blk_add_rq_to_plug") ?
>
> Do you know if the stddev has such big variation for this test even before
> the commit?
>
>
> If it is not too much to ask... It might be interesting to know if we see
> a regression when comparing before/after e70c301faece with scheduler none
> instead of mq-deadline.
we also finished the test for scheduler none, run v6.12-rc4, before/after
e70c301faece 15 times for each. it seems the data is not stable enough under
scheduler none, only can say there is a regression trend if comparing
e70c301faece to its parent.
=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/none-scheduler/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s
commit:
v6.12-rc4
a3396b9999 ("block: add a rq_list type")
e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")
v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
114.62 ± 19% -1.9% 112.49 ± 17% -32.4% 77.47 ± 21% stress-ng.aiol.ops_per_sec
raw data is as below:
v6.12-rc4/matrix.json: "stress-ng.aiol.ops_per_sec": [
v6.12-rc4/matrix.json- 108.03,
v6.12-rc4/matrix.json- 108.4,
v6.12-rc4/matrix.json- 109.11,
v6.12-rc4/matrix.json- 109.58,
v6.12-rc4/matrix.json- 194.21,
v6.12-rc4/matrix.json- 111.53,
v6.12-rc4/matrix.json- 107.99,
v6.12-rc4/matrix.json- 115.29,
v6.12-rc4/matrix.json- 105.75,
v6.12-rc4/matrix.json- 113.62,
v6.12-rc4/matrix.json- 96.51,
v6.12-rc4/matrix.json- 110.53,
v6.12-rc4/matrix.json- 108.71,
v6.12-rc4/matrix.json- 98.06,
v6.12-rc4/matrix.json- 121.95
v6.12-rc4/matrix.json- ],
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json: "stress-ng.aiol.ops_per_sec": [
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 116.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 106.51,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 119.23,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 108.91,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.79,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.81,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 114.94,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 99.49,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 106.13,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 124.99,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 174.15,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 92.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 113.05,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 75.97,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- 111.05
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json- ],
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json: "stress-ng.aiol.ops_per_sec": [
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 85.2,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 72.6,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 73.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 69.03,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 66.9,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 90.24,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 66.88,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 71.53,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 56.86,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 63.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 97.99,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 69.28,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 58.52,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 114.23,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- 105.79
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json- ],
since not sure if it's valuable, I didn't list the full comparison table here.
if you want it, please let us know. thanks!
>
>
> Kind regards,
> Niklas
>
