> -----Original Message----- > From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On > Behalf Of Jens Axboe > Sent: Sunday, December 3, 2017 11:11 AM > To: Sitsofe Wheeler <sitsofe@xxxxxxxxx>; Robert Elliott (Persistent > Memory) <elliott@xxxxxxx> > Cc: Gavriliuk, Anton (HPS Ukraine) <anton.gavriliuk@xxxxxxx>; Rebecca Cran > <rebecca@xxxxxxxxxxxx>; fio@xxxxxxxxxxxxxxx; Kani, Toshimitsu > <toshi.kani@xxxxxxx> > Subject: Re: fio 3.2 > > On 12/03/2017 02:35 AM, Sitsofe Wheeler wrote: > > On 1 December 2017 at 07:15, Robert Elliott (Persistent Memory) > > <elliott@xxxxxxx> wrote: > >> While discussing NUMA, I'll mention something else I saw in Windows > >> while fixing the thread affinities there. > >> > >> At startup, fio spawns threads on all CPUs to measure the clocks > >> (fio_monotonic_clocktest). If you've constrained the CPU affinity > >> outside fio, some of those will fail. In Windows, something like > >> START /AFFINITY 0x55555555 fio ... > >> can cause half of the clock threads to fail. > > > > This is very weird and doesn't make any sense (but I believe you): if > > you have multiple threads crammed on to the same CPUs the TSC no > > longer looks like it monotonically increases? Surely it should be MORE > > likely to increase because a thread is likely to be on the same CPU as > > another and can't actually be running at the same time as the other? > > The threads fail to start, it's not a TSC failure. I'm guessing it's > because fio gets limited to a subset of the available CPUs, and that > causes fio to fail doing the clock check when fio_setaffinity() in > clock_thread_fn() fails. I'll work on a patch to limit all threads (including clocktest threads at startup and I/O threads) to the parent's affinity mask in Windows. In linux, processes and threads are not restricted to their parent's affinity mask. Although they inherit by default, they may call sched_setaffinity with any values. The taskset and numactl manpages don't mention that programs may override the settings requested by the user. Example: $ numactl -a -C 5 fio --debug=process -dax_mmap.fio including: cpus_allowed_policy=split cpus_allowed=0-17,36-53 thread fio is allowed to place threads on all the CPUs: ROB: process 20358 cpu=5 thread affinity_count=1 ROB: process 20353 cpu=0 thread affinity_count=1 ROB: process 20355 cpu=2 thread affinity_count=1 ROB: process 20354 cpu=1 thread affinity_count=1 ROB: process 20356 cpu=3 thread affinity_count=1 ... Should fio apply the same filtering in linux? If not, should it print an info or warning message when creating threads that violate the parent mask? --- Robert Elliott, HPE Persistent Memory ��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
