On 12/29/20 9:19 AM, Keith Busch wrote: > On Tue, Dec 29, 2020 at 02:40:57PM +0100, Stefan Lederer wrote: >> Hello dear list, >> >> (I hope I do not annoy you as a simple application programmer) >> >> for a seminar paper at my university we reproduced the 2009 paper >> "Pathologies of big data" by Jacobs, where he basically reads a >> 100GB file sequentially from a HDD with some light processing. >> >> We have a PCIE4.0 SSD with up to 7GB/s reading (Samsung 980) but >> nothing we programmed so far comes even close to that speed (regular >> read(), mmap() with optional threads, io_uring, multi-process) so we >> wonder if it is possible at all? >> >> According to iostat mmap is the fastest with 4GB/s and a queue depth >> of ~3. All other approaches do not go beyond 2.5GB/s. >> >> Also we get some strange effects like sequential read() with 16KB >> buffers being faster than one with 16MB and io_uring being alot >> slower than mmap (all tested on Manjaro with kernel 5.8/5.10 and ext4). >> >> So, now we are quite lost and would appreciate a hint into the right >> direction :) >> >> What is neccesary to simply read 100GB of data at 7GB/s? > > Is your device running at gen4 speed? Easiest way to tell with an nvme > ssd (assuming you're reading from /dev/nvme0n1) is something like: > > # cat /sys/block/nvme0n1/device/device/current_link_speed > > If it says less than 16GT/s, then it can't read at 7GB/s. Does sound like that a lot. Simple test here on a gen4 device: # cat /sys/block/nvme3n1/device/device/current_link_speed 16.0 GT/s PCIe # ~axboe/git/fio/fio --name=bw --filename=/dev/nvme3n1 --direct=1 --bs=32k --ioengine=io_uring --iodepth=16 --rw=randread --norandommap [snip] READ: bw=6630MiB/s (6952MB/s), 6630MiB/s-6630MiB/s (6952MB/s-6952MB/s), io=36.4GiB (39.1GB), run=5621-5621msec -- Jens Axboe