> -----Original Message----- > From: Jens Axboe [mailto:axboe@xxxxxxxxx] > Sent: Wednesday, 26 February, 2014 6:08 PM > To: Elliott, Robert (Server Storage); fio@xxxxxxxxxxxxxxx > Subject: Re: cpus_allowed per thread behavior > > On 2014-02-26 15:54, Elliott, Robert (Server Storage) wrote: > > fio seems to assign the same cpus_allowed/cpumask value to all threads. > > I think this allows the OS to move the threads around those CPUs. > > Correct. As long as the number of cpus in the mask is equal to (or > larger than) the number of jobs within that group, the OS is free to > place them wherever it wants. In practice, unless the CPU scheduling is > horribly broken, they tend to "stick" for most intents and purposes. > > > In comparison, iometer assigns its worker threads to specific CPUs > > within the cpumask in round-robin manner. Would that be worth adding > > to fio, perhaps with an option like cpus_allowed_policy=roundrobin? > > Sure, we could add that feature. You can get the same setup now, if you > "unroll" the job section, but that might not always be practical. How > about cpus_allowed_policy, with 'shared' being the existing (and > default) behavior and 'split' being each thread grabbing one of the CPUs? Perhaps NUMA and hyperthreading aware allocation policies would also be useful? I don't know how consistent hyperthread CPU numbering is across systems. On some servers I've tried, linux assigns 0-5 to the main cores and 6-11 to the hyperthreaded siblings, while Windows assigns 0,2,4,6,8,10 to the main cores and 1,3,5,7,9,11 to their hyperthreaded siblings. Intel's OpenMP library offers two thread affinity types that might be worth simulating: COMPACT: pack them tightly foreach (node) foreach (core in the node) foreach (hyperthreaded sibling) SCATTER: spread across all the cores foreach (hyperthreaded sibling) foreach (core sharing a node) foreach (node) We could try: cpus_allowed_policy=shared cpus_allowed_policy=split (round-robin, don't care how the CPU IDs were assigned) cpus_allowed_policy=compact (NUMA/HT aware) cpus_allowed_policy=scatter (NUMA/HT aware) > -- > Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
