Hello dear list, (I hope I do not annoy you as a simple application programmer) for a seminar paper at my university we reproduced the 2009 paper "Pathologies of big data" by Jacobs, where he basically reads a 100GB file sequentially from a HDD with some light processing. We have a PCIE4.0 SSD with up to 7GB/s reading (Samsung 980) but nothing we programmed so far comes even close to that speed (regular read(), mmap() with optional threads, io_uring, multi-process) so we wonder if it is possible at all? According to iostat mmap is the fastest with 4GB/s and a queue depth of ~3. All other approaches do not go beyond 2.5GB/s. Also we get some strange effects like sequential read() with 16KB buffers being faster than one with 16MB and io_uring being alot slower than mmap (all tested on Manjaro with kernel 5.8/5.10 and ext4). So, now we are quite lost and would appreciate a hint into the right direction :) What is neccesary to simply read 100GB of data at 7GB/s? I wish everybody a happy new year! Stefan Lederer
