---- On Tue, 10 Sep 2024 10:30:18 +0100 Robert Beckett wrote --- > > > > > > ---- On Mon, 09 Sep 2024 21:31:41 +0100 Keith Busch wrote --- > > On Mon, Sep 09, 2024 at 02:29:14PM -0600, Keith Busch wrote: > > > As a test, could you try kernel parameter "nvme.io_queue_depth_set=2"? > > > > Err, I mean "nvme.io_queue_depth=2". > > > > Thanks, I'll give it a try along with your other questions and report back. > > For clarity, the repro steps dropped a step. They should have included the make command: > > > $ dd if=/dev/urandom of=test_file bs=1M count=10240 > $ desync make test_file.caibx test_file > $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" > $ desync verify-index test_file.caibx test_file > CONFIG_SLUB_DEBUG_ON showed no debug output. nvme.io_queue_depth=2 appears to fix it. Could you explain the implications of this? I assume it is limiting to 2 outstanding requests concurrently. Does it suggest an issue with the specific device's FW? I assume this would suggest that it is not actually anything wrong with the dmapool, it was just exposing the issue of the device/fw? Any advice for handling this and/or investigating further? My initial speculation was that maybe the disk fw is signalling completion of an access before it has actually finished making it's way to ram. I checked the code and saw that the dmapool appears to be used for storing the buffer page addresses, so I imagine that is not updated by the disk at all, which would rule out my assumption. I'd appreciate any insight you could give on the usage of the dmapools in the driver and whether you would expect them to be significant in this issue, or if they are just making a device/fw bug more observable. Thanks Bob p.s. Here is an transcript of the issue seen in testing. To my knowledge, if everything is working as it should, nothing should be able to produce this output, that dropping caches and re-priming the page cache via a linear read it fixes things. $ dd if=/dev/urandom of=test_file bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 111.609 s, 96.2 MB/s $ desync make test_file.caibx test_file Chunking [=======================================================================================================================================] 100.00% 18s $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" $ desync verify-index test_file.caibx test_file [=============>-----------------------------------------------------------------------------------------------------------------------------------] 9.00% 4s Error: seed index for test_file doesn't match its data $ md5sum test_file ce4f1cca0b3dfd63ea2adfd745e4bfc1 test_file $ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches" $ md5sum test_file 1edb3eaf5ae57b6187cc0be843ed2e5c test_file $ desync verify-index test_file.caibx test_file [=================================================================================================================================================] 100.00% 5s
