Elliott, Robert (Persistent Memory) wrote on 03/02/2016 10:37 AM: >>>> Here is my full fio job: >>>> >>>> [global] >>>> ioengine=libaio >>>> buffered=0 >>>> sync=0 >>>> direct=1 >>>> randrepeat=1 >>>> softrandommap=1 >>>> rw=randread >>>> bs=4k >>>> filename=./nvram (it's a link to a block device) >>>> exitall=1 >>>> thread=1 >>>> disable_lat=1 >>>> disable_slat=1 >>>> disable_clat=1 >>>> loops=10 >>>> iodepth=16 >>> >>> You appear to be missing gtod_reduce >>> (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L1668 ) or >>> gettimeofday cpu pinning. You also aren't using batching >>> (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L815 ). >> > ... >> Jobs IOPS(M) %user %sys >> 1 4.3 78 22 >> 2 7.6 67 33 >> 3 10.5 65 35 >> 4 7.7 61 38 >> 5 4.8 78 22 >> 6 4.7 83 17 >> 7 4.8 84 15 > > Use cpus_allowed_policy=split to keep threads or processes > from wandering across CPUs. Tried, no help, which is easily explainable, because I'm always running num jobs < num CPUs. > If the system is NUMA, split the NVDIMMs based on that > and use cpus_allowed to just have threads access local > NVDIMMs. No, it's single socket. > As someone else mentioned, the random map overhead is > noticeable, especially for large capacity devices. Use > norandommap and randrepeat=0. Had done, see the job file in my message you didn't quote. > There was a problem in 2014 requiring invalidate=0 for > small capacities; I don't recall if that was fixed. > > If you're using the pmem driver, it is incapable of queuing > (submissions are immediately completed), so iodepth is > irrelevant. > > There are other system and OS settings to tune, like: > * use the performance governor. The userspace governor > spends a lot of time making acpi_os_write calls Had done, see not quoted part of my e-mail. > * turn off auditing (audit=0 on the kernel command line) > * turn off C states Tried too. Overall, I appreciate your help, but again, question is not how to improve my results. The question is how to _decrease fio overhead_ with libaio, see subject of this e-mail. It's very different question. Thanks, Vlad -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
