On Mon, Feb 26, 2024 at 08:05:58PM -0600, John Groves wrote: > On 24/02/26 04:58PM, Luis Chamberlain wrote: > > On Mon, Feb 26, 2024 at 1:16 PM John Groves <John@xxxxxxxxxx> wrote: > > > > > > On 24/02/26 07:53AM, Luis Chamberlain wrote: > > > > On Mon, Feb 26, 2024 at 07:27:18AM -0600, John Groves wrote: > > > > > Run status group 0 (all jobs): > > > > > WRITE: bw=29.6GiB/s (31.8GB/s), 29.6GiB/s-29.6GiB/s (31.8GB/s-31.8GB/s), io=44.7GiB (48.0GB), run=1511-1511msec > > > > > > > > > This is run on an xfs file system on a SATA ssd. > > > > > > > > To compare more closer apples to apples, wouldn't it make more sense > > > > to try this with XFS on pmem (with fio -direct=1)? > > > > > > > > Luis > > > > > > Makes sense. Here is the same command line I used with xfs before, but > > > now it's on /dev/pmem0 (the same 128G, but converted from devdax to pmem > > > because xfs requires that. > > > > > > fio -name=ten-256m-per-thread --nrfiles=10 -bs=2M --group_reporting=1 --alloc-size=1048576 --filesize=256MiB --readwrite=write --fallocate=none --numjobs=48 --create_on_open=0 --ioengine=io_uring --direct=1 --directory=/mnt/xfs > > > > Could you try with mkfs.xfs -d agcount=1024 Won't change anything for the better, may make things worse. > bw ( MiB/s): min= 5085, max=27367, per=100.00%, avg=14361.95, stdev=165.61, samples=719 > iops : min= 2516, max=13670, avg=7160.17, stdev=82.88, samples=719 > lat (usec) : 4=0.05%, 10=0.72%, 20=2.23%, 50=2.48%, 100=3.02% > lat (usec) : 250=1.54%, 500=2.37%, 750=1.34%, 1000=0.75% > lat (msec) : 2=3.20%, 4=43.10%, 10=23.05%, 20=14.81%, 50=1.25% Most of the IO latencies are up round the 4-20ms marks. That seems kinda high for a 2MB IO. With a memcpy speed of 10GB/s, the 2MB should only take a couple of hundred microseconds. For Famfs, the latencies appear to be around 1-4ms. So where's all that extra time coming from? > lat (msec) : 100=0.08% > cpu : usr=10.18%, sys=0.79%, ctx=67227, majf=0, minf=38511 And why is system time reporting at almost zero instead of almost all the remaining cpu time (i.e. up at 80-90%)? Can you run call-graph kernel profiles for XFS and famfs whilst running this workload so we have some insight into what is behaving differently here? -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
