Hello guys, During the last few months, I felt a performance regression when using read() and write() on my high-speed Nvme SSD (about 7GB/s). To get more precise information about it I quickly developed benchmark tool basically running read() or write() in a loop to simulate a sequential file read or write. The tool also measures the real time consumed by the loop. Finally, the tool can call open() with or without O_DIRECT. I ran the tests on EXT4 and Exfat with following settings (buffer values have been set for best result): - Write settings: buffer 400mb * 100 - Read settings: buffer 200mb - Drop caches before non-direct read/write test With this hardware: - CPU AMD Ryzen 7600X - RAM DDR5 5200 32GB - SSD Kingston Fury Renegade 4TB with 4K LBA Here are some results I got with last upstream kernels (default config): +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | ~42GB | O_DIRECT | Linux 6.2.0 | Linux 6.3.0 | Linux 6.4.0 | Linux 6.5.0 | Linux 6.5.5 | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ | Ext4 (sector 4k) | | | | | | | | Read | no | 7.2s (5800MB/s) | 7.1s (5890MB/s) | 8.3s (5050MB/s) | 13.2s (3180MB/s) | 13.2s (3180MB/s) | | Write | no | 12.0s (3500MB/s) | 12.6s (3340MB/s) | 12.2s (3440MB/s) | 28.9s (1450MB/s) | 28.9s (1450MB/s) | | Read | yes | 6.0s (7000MB/s) | 6.0s (7020MB/s) | 5.9s (7170MB/s) | 5.9s (7100MB/s) | 5.9s (7100MB/s) | | Write | yes | 6.7s (6220MB/s) | 6.7s (6290MB/s) | 6.9s (6080MB/s) | 6.9s (6080MB/s) | 6.9s (6970MB/s) | | Exfat (sector ?) | | | | | | | | Read | no | 7.3s (5770MB/s) | 7.2s (5830MB/s) | 9s (4620MB/s) | 13.3s (3150MB/s) | 13.2s (3180MB/s) | | Write | no | 8.3s (5040MB/s) | 8.9s (4750MB/s) | 8.3s (5040MB/s) | 18.3s (2290MB/s) | 18.5s (2260MB/s) | | Read | yes | 6.2s (6760MB/s) | 6.1s (6870MB/s) | 6.0s (6980MB/s) | 6.5s (6440MB/s) | 6.6s (6320MB/s) | | Write | yes | 16.1s (2610MB/s) | 16.0s (2620MB/s) | 18.7s (2240MB/s) | 34.1s (1230MB/s) | 34.5s (1220MB/s) | +------------------+----------+------------------+------------------+-- ----------------+------------------+------------------+ Please note that I rounded some values to clarify readiness. Small variations can be considered as margin error. Ext4 results: cached reads/writes time have increased of almost 100% from 6.2.0 to 6.5.0 with a first increase with 6.4.0. Direct access times have stayed similar though. Exfat results: performance decrease too with and without direct access this time. I realize there are thousands of commits between, plus the issue can come from multiple kernel parts such as the page cache, the file system implementation (especially for Exfat), the IO engine, a driver, etc. The results also showed that there is not only a specific version impacted. Anyway, at the end the performance have highly decreased. If you want to verify my benchmark tool source code, please ask. PS: sending again as only text body is accepted Regards Florent DELAHAYE
