> -----Original Message----- > From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On > Behalf Of Vladislav Bolkhovitin > Sent: Tuesday, March 01, 2016 10:25 PM > To: Sitsofe Wheeler <sitsofe@xxxxxxxxx>; fio@xxxxxxxxxxxxxxx > Subject: Re: Fio high IOPS measurement mistake > > Hi, > ... > >> Here is my full fio job: > >> > >> [global] > >> ioengine=libaio > >> buffered=0 > >> sync=0 > >> direct=1 > >> randrepeat=1 > >> softrandommap=1 > >> rw=randread > >> bs=4k > >> filename=./nvram (it's a link to a block device) > >> exitall=1 > >> thread=1 > >> disable_lat=1 > >> disable_slat=1 > >> disable_clat=1 > >> loops=10 > >> iodepth=16 > > > > You appear to be missing gtod_reduce > > (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L1668 ) or > > gettimeofday cpu pinning. You also aren't using batching > > (https://github.com/axboe/fio/blob/fio-2.6/HOWTO#L815 ). > ... > Jobs IOPS(M) %user %sys > 1 4.3 78 22 > 2 7.6 67 33 > 3 10.5 65 35 > 4 7.7 61 38 > 5 4.8 78 22 > 6 4.7 83 17 > 7 4.8 84 15 Use cpus_allowed_policy=split to keep threads or processes from wandering across CPUs. If the system is NUMA, split the NVDIMMs based on that and use cpus_allowed to just have threads access local NVDIMMs. As someone else mentioned, the random map overhead is noticeable, especially for large capacity devices. Use norandommap and randrepeat=0. There was a problem in 2014 requiring invalidate=0 for small capacities; I don't recall if that was fixed. If you're using the pmem driver, it is incapable of queuing (submissions are immediately completed), so iodepth is irrelevant. There are other system and OS settings to tune, like: * use the performance governor. The userspace governor spends a lot of time making acpi_os_write calls * turn off auditing (audit=0 on the kernel command line) * turn off C states ��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
