On 2 March 2016 at 04:25, Vladislav Bolkhovitin <vst@xxxxxxxx> wrote: > Hi, > > Sitsofe Wheeler wrote on 02/29/2016 10:01 PM: >> Hi, >> >> On 1 March 2016 at 05:17, Vladislav Bolkhovitin <vst@xxxxxxxx> wrote: >>> Hello, >>> >>> I'm currently looking at one NVRAM device, and during fio tests noticed that each fio >>> thread consumes 30% of user space CPU. I'm using ioengine=libaio, buffered=0, sync=0 >>> and direct=1, so user space CPU consumption should be virtually zero. >>> >>> That 30% user CPU consumption makes me suspect that this is overhead for internal fio >>> housekeeping, i.e., scientifically speaking, fio instrumental measurement mistake (I >>> hope, I'm using correct English terms). >>> >>> Can anybody comment it and suggest how to decrease this user space CPU consumption? >>> >>> Here is my full fio job: >>> >>> [global] >>> ioengine=libaio >>> buffered=0 >>> sync=0 >>> direct=1 >>> randrepeat=1 >>> softrandommap=1 >>> rw=randread >>> bs=4k >>> filename=./nvram (it's a link to a block device) >>> exitall=1 >>> thread=1 >>> disable_lat=1 >>> disable_slat=1 >>> disable_clat=1 >>> loops=10 >>> iodepth=16 >> >> You appear to be missing gtod_reduce >> (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L1668 ) or >> gettimeofday cpu pinning. You also aren't using batching >> (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L815 ). > > Thanks, I tried them, but they did not make any significant difference. The biggest There was no cpu reduction from setting both iodepth_batch and iodepth_batch_complete together? > difference I had was when I changed CPU governor to "performance". Now I have 20-25% > user space, measured by fio itself, it's coherent with top. Note, I'm considering > per-thread CPU consumption, to see it in top you need to press '1' (one line per each CPU). Have you applied the points mentioned in https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/chap-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Performance_Features_in_RednbspHat_EnterprisenbspLinuxnbsp7.html ? Things like the scheduler granularity (as changed by Red Hat's tuned-adm) can have a large impact... > The full job file was: > > [global] > ioengine=sync > buffered=0 > sync=0 > direct=1 [...] > iodepth=8 /* does not matter */ ^^^ It's worth noting that direct and iodepth don't really have an impact on synchronous ioengines - https://github.com/axboe/fio/blob/master/HOWTO#L804 . -- Sitsofe | http://sucs.org/~sits/ -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
