Robert, Agreed split is better. In trying to trim down the parameter list I left the policy to shared, which is default. I haven't constrained the fio process, so not sure why the resulting clock threads fail. Is the mask shifted the correct number of bits? Sitsofe, On a x64 version of Win2016, with dual Intel(R) Xeon(R) CPU E5-2623 v3 @ 3.00GHz, running fio --name=test --ioengine=windowsaio --direct=1 --rw=write --overwrite=1 --filename=\\.\PHYSICALDRIVE1 --runtime=1s --thread --cpus_allowed=0-1 --debug=process process 4788 row_first_cpu=0 cpumask->row[0]=1 process 4788 first set cpu in mask is at index 0 process 4788 group=0 group_start=0 group_size=64 search_cpu=0 process 4788 row_last_cpu=-1 cpumask->row[7]=0 process 4788 row_last_cpu=-1 cpumask->row[6]=0 process 5868 row_first_cpu=1 cpumask->row[0]=2 process 4788 row_last_cpu=-1 cpumask->row[5]=0 process 4788 row_last_cpu=-1 cpumask->row[4]=0 process 5868 first set cpu in mask is at index 1 process 4788 row_last_cpu=-1 cpumask->row[3]=0 process 4788 row_last_cpu=-1 cpumask->row[2]=0 process 4788 row_last_cpu=-1 cpumask->row[1]=0 process 4788 row_last_cpu=0 cpumask->row[0]=1 process 5868 group=0 group_start=0 group_size=64 search_cpu=1 process 4788 last set cpu in mask is at index 0 process 5868 row_last_cpu=-1 cpumask->row[7]=0 process 4788 Returning group=0 group_mask=1 process 5868 row_last_cpu=-1 cpumask->row[6]=0 process 5868 row_last_cpu=-1 cpumask->row[5]=0 process 5868 row_last_cpu=-1 cpumask->row[4]=0 process 5868 row_last_cpu=-1 cpumask->row[3]=0 process 920 row_first_cpu=3 cpumask->row[0]=8 process 5868 row_last_cpu=-1 cpumask->row[2]=0 process 920 first set cpu in mask is at index 3 process 5868 row_last_cpu=-1 cpumask->row[1]=0 process 5868 row_last_cpu=1 cpumask->row[0]=2 process 5868 last set cpu in mask is at index 1 process 920 group=0 group_start=0 group_size=64 search_cpu=3 process 5868 Returning group=0 group_mask=2 process 920 row_last_cpu=-1 cpumask->row[7]=0 process 920 row_last_cpu=-1 cpumask->row[6]=0 process 920 row_last_cpu=-1 cpumask->row[5]=0 process 920 row_last_cpu=-1 cpumask->row[4]=0 process 920 row_last_cpu=-1 cpumask->row[3]=0 process 920 row_last_cpu=-1 cpumask->row[2]=0 process 920 row_last_cpu=-1 cpumask->row[1]=0 process 5888 row_first_cpu=2 cpumask->row[0]=4 process 920 row_last_cpu=3 cpumask->row[0]=8 process 5888 first set cpu in mask is at index 2 process 920 last set cpu in mask is at index 3 process 920 Returning group=0 group_mask=8 process 5888 group=0 group_start=0 group_size=64 search_cpu=2 process 5888 row_last_cpu=-1 cpumask->row[7]=0 process 5888 row_last_cpu=-1 cpumask->row[6]=0 process 5888 row_last_cpu=-1 cpumask->row[5]=0 process 5888 row_last_cpu=-1 cpumask->row[4]=0 process 5888 row_last_cpu=-1 cpumask->row[3]=0 process 5888 row_last_cpu=-1 cpumask->row[2]=0 process 5888 row_last_cpu=-1 cpumask->row[1]=0 process 5888 row_last_cpu=2 cpumask->row[0]=4 process 5880 row_first_cpu=4 cpumask->row[0]=16 process 5888 last set cpu in mask is at index 2 process 5888 Returning group=0 group_mask=4 process 5880 first set cpu in mask is at index 4 process 5880 group=0 group_start=0 group_size=64 search_cpu=4 process 5880 row_last_cpu=-1 cpumask->row[7]=0 process 5880 row_last_cpu=-1 cpumask->row[6]=0 process 5880 row_last_cpu=-1 cpumask->row[5]=0 process 5880 row_last_cpu=-1 cpumask->row[4]=0 process 5880 row_last_cpu=-1 cpumask->row[3]=0 process 5880 row_last_cpu=-1 cpumask->row[2]=0 process 5880 row_last_cpu=-1 cpumask->row[1]=0 process 5880 row_last_cpu=4 cpumask->row[0]=16 process 5880 last set cpu in mask is at index 4 process 5880 Returning group=0 group_mask=16 process 1064 row_first_cpu=5 cpumask->row[0]=32 process 1064 first set cpu in mask is at index 5 process 1064 group=0 group_start=0 group_size=64 search_cpu=5 process 1064 row_last_cpu=-1 cpumask->row[7]=0 process 1064 row_last_cpu=-1 cpumask->row[6]=0 process 1064 row_last_cpu=-1 cpumask->row[5]=0 process 1064 row_last_cpu=-1 cpumask->row[4]=0 process 5104 row_first_cpu=6 cpumask->row[0]=64 process 5104 first set cpu in mask is at index 6 process 1064 row_last_cpu=-1 cpumask->row[3]=0 process 5104 group=0 group_start=0 group_size=64 search_cpu=6 process 1064 row_last_cpu=-1 cpumask->row[2]=0 process 1064 row_last_cpu=-1 cpumask->row[1]=0 process 5104 row_last_cpu=-1 cpumask->row[7]=0 process 5640 row_first_cpu=14 cpumask->row[0]=16384 process 5640 first set cpu in mask is at index 14 process 1064 row_last_cpu=5 cpumask->row[0]=32 process 5104 row_last_cpu=-1 cpumask->row[6]=0 process 2108 row_first_cpu=7 cpumask->row[0]=128 process 1064 last set cpu in mask is at index 5 process 5104 row_last_cpu=-1 cpumask->row[5]=0 process 5640 group=0 group_start=0 group_size=64 search_cpu=14 process 3776 row_first_cpu=8 cpumask->row[0]=256 process 1064 Returning group=0 group_mask=32 process 5104 row_last_cpu=-1 cpumask->row[4]=0 process 5640 row_last_cpu=-1 cpumask->row[7]=0 process 2108 first set cpu in mask is at index 7 process 1176 row_first_cpu=9 cpumask->row[0]=512 process 5640 row_last_cpu=-1 cpumask->row[6]=0 process 1420 row_first_cpu=13 cpumask->row[0]=8192 process 3776 first set cpu in mask is at index 8 process 5640 row_last_cpu=-1 cpumask->row[5]=0 process 4000 row_first_cpu=10 cpumask->row[0]=1024 process 5640 row_last_cpu=-1 cpumask->row[4]=0 process 3776 group=0 group_start=0 group_size=64 search_cpu=8 process 6028 row_first_cpu=11 cpumask->row[0]=2048 process 5640 row_last_cpu=-1 cpumask->row[3]=0 process 5104 row_last_cpu=-1 cpumask->row[3]=0 process 4000 first set cpu in mask is at index 10 process 2108 group=0 group_start=0 group_size=64 search_cpu=7 process 3776 row_last_cpu=-1 cpumask->row[7]=0 process 1420 first set cpu in mask is at index 13 process 1176 first set cpu in mask is at index 9 process 5104 row_last_cpu=-1 cpumask->row[2]=0 process 5640 row_last_cpu=-1 cpumask->row[2]=0 process 1088 row_first_cpu=12 cpumask->row[0]=4096 process 3776 row_last_cpu=-1 cpumask->row[6]=0 process 2108 row_last_cpu=-1 cpumask->row[7]=0 process 5104 row_last_cpu=-1 cpumask->row[1]=0 process 6028 first set cpu in mask is at index 11 process 5640 row_last_cpu=-1 cpumask->row[1]=0 process 4000 group=0 group_start=0 group_size=64 search_cpu=10 process 3776 row_last_cpu=-1 cpumask->row[5]=0 process 2108 row_last_cpu=-1 cpumask->row[6]=0 process 1420 group=0 group_start=0 group_size=64 search_cpu=13 process 5104 row_last_cpu=6 cpumask->row[0]=64 process 1088 first set cpu in mask is at index 12 process 5640 row_last_cpu=14 cpumask->row[0]=16384 process 4000 row_last_cpu=-1 cpumask->row[7]=0 process 1176 group=0 group_start=0 group_size=64 search_cpu=9 process 1420 row_last_cpu=-1 cpumask->row[7]=0 process 5104 last set cpu in mask is at index 6 process 2108 row_last_cpu=-1 cpumask->row[5]=0 process 3776 row_last_cpu=-1 cpumask->row[4]=0 process 5640 last set cpu in mask is at index 14 process 6028 group=0 group_start=0 group_size=64 search_cpu=11 process 360 row_first_cpu=15 cpumask->row[0]=32768 process 5104 Returning group=0 group_mask=64 process 2108 row_last_cpu=-1 cpumask->row[4]=0 process 1088 group=0 group_start=0 group_size=64 search_cpu=12 process 5640 Returning group=0 group_mask=16384 process 360 first set cpu in mask is at index 15 process 2108 row_last_cpu=-1 cpumask->row[3]=0 process 6028 row_last_cpu=-1 cpumask->row[7]=0 process 3776 row_last_cpu=-1 cpumask->row[3]=0 process 1088 row_last_cpu=-1 cpumask->row[7]=0 process 2108 row_last_cpu=-1 cpumask->row[2]=0 process 6028 row_last_cpu=-1 cpumask->row[6]=0 process 3776 row_last_cpu=-1 cpumask->row[2]=0 process 2108 row_last_cpu=-1 cpumask->row[1]=0 process 4000 row_last_cpu=-1 cpumask->row[6]=0 process 360 group=0 group_start=0 group_size=64 search_cpu=15 process 6028 row_last_cpu=-1 cpumask->row[5]=0 process 1088 row_last_cpu=-1 cpumask->row[6]=0 process 3776 row_last_cpu=-1 cpumask->row[1]=0 process 1420 row_last_cpu=-1 cpumask->row[6]=0 process 2108 row_last_cpu=7 cpumask->row[0]=128 process 360 row_last_cpu=-1 cpumask->row[7]=0 process 1088 row_last_cpu=-1 cpumask->row[5]=0 process 1176 row_last_cpu=-1 cpumask->row[7]=0 process 6028 row_last_cpu=-1 cpumask->row[4]=0 process 1420 row_last_cpu=-1 cpumask->row[5]=0 process 1088 row_last_cpu=-1 cpumask->row[4]=0 process 360 row_last_cpu=-1 cpumask->row[6]=0 process 5948 row_first_cpu=17 cpumask->row[0]=131072 process 3776 row_last_cpu=8 cpumask->row[0]=256 process 5948 first set cpu in mask is at index 17 process 360 row_last_cpu=-1 cpumask->row[5]=0 process 3076 row_first_cpu=18 cpumask->row[0]=262144 process 1420 row_last_cpu=-1 cpumask->row[4]=0 process 360 row_last_cpu=-1 cpumask->row[4]=0 process 3776 last set cpu in mask is at index 8 process 360 row_last_cpu=-1 cpumask->row[3]=0 process 3076 first set cpu in mask is at index 18 process 1088 row_last_cpu=-1 cpumask->row[3]=0 process 5948 group=0 group_start=0 group_size=64 search_cpu=17 process 3776 Returning group=0 group_mask=256 process 360 row_last_cpu=-1 cpumask->row[2]=0 process 6028 row_last_cpu=-1 cpumask->row[3]=0 process 5948 row_last_cpu=-1 cpumask->row[7]=0 process 360 row_last_cpu=-1 cpumask->row[1]=0 process 3076 group=0 group_start=0 group_size=64 search_cpu=18 process 1088 row_last_cpu=-1 cpumask->row[2]=0 process 5948 row_last_cpu=-1 cpumask->row[6]=0 process 4000 row_last_cpu=-1 cpumask->row[5]=0 process 360 row_last_cpu=15 cpumask->row[0]=32768 process 3076 row_last_cpu=-1 cpumask->row[7]=0 process 1088 row_last_cpu=-1 cpumask->row[1]=0 process 6028 row_last_cpu=-1 cpumask->row[2]=0 process 5948 row_last_cpu=-1 cpumask->row[5]=0 process 1420 row_last_cpu=-1 cpumask->row[3]=0 process 360 last set cpu in mask is at index 15 process 2108 last set cpu in mask is at index 7 process 3076 row_last_cpu=-1 cpumask->row[6]=0 process 1088 row_last_cpu=12 cpumask->row[0]=4096 process 6028 row_last_cpu=-1 cpumask->row[1]=0 process 5948 row_last_cpu=-1 cpumask->row[4]=0 process 1420 row_last_cpu=-1 cpumask->row[2]=0 process 360 Returning group=0 group_mask=32768 process 4000 row_last_cpu=-1 cpumask->row[4]=0 process 3076 row_last_cpu=-1 cpumask->row[5]=0 process 1088 last set cpu in mask is at index 12 process 5948 row_last_cpu=-1 cpumask->row[3]=0 process 4000 row_last_cpu=-1 cpumask->row[3]=0 process 2108 Returning group=0 group_mask=128 process 5948 row_last_cpu=-1 cpumask->row[2]=0 process 4000 row_last_cpu=-1 cpumask->row[2]=0 process 1420 row_last_cpu=-1 cpumask->row[1]=0 process 3076 row_last_cpu=-1 cpumask->row[4]=0 process 6028 row_last_cpu=11 cpumask->row[0]=2048 process 4000 row_last_cpu=-1 cpumask->row[1]=0 process 5948 row_last_cpu=-1 cpumask->row[1]=0 process 1088 Returning group=0 group_mask=4096 process 1420 row_last_cpu=13 cpumask->row[0]=8192 process 1176 row_last_cpu=-1 cpumask->row[6]=0 process 3076 row_last_cpu=-1 cpumask->row[3]=0 process 6028 last set cpu in mask is at index 11 process 5948 row_last_cpu=17 cpumask->row[0]=131072 process 4000 row_last_cpu=10 cpumask->row[0]=1024 process 1420 last set cpu in mask is at index 13 process 4280 row_first_cpu=19 cpumask->row[0]=524288 process 3180 row_first_cpu=16 cpumask->row[0]=65536 process 4000 last set cpu in mask is at index 10 process 4280 first set cpu in mask is at index 19 process 1176 row_last_cpu=-1 cpumask->row[5]=0 process 3180 first set cpu in mask is at index 16 process 4000 Returning group=0 group_mask=1024 process 1420 Returning group=0 group_mask=8192 process 3076 row_last_cpu=-1 cpumask->row[2]=0 process 1176 row_last_cpu=-1 cpumask->row[4]=0 process 6028 Returning group=0 group_mask=2048 process 4280 group=0 group_start=0 group_size=64 search_cpu=19 process 5948 last set cpu in mask is at index 17 process 3180 group=0 group_start=0 group_size=64 search_cpu=16 process 1176 row_last_cpu=-1 cpumask->row[3]=0 process 3076 row_last_cpu=-1 cpumask->row[1]=0 process 4280 row_last_cpu=-1 cpumask->row[7]=0 process 3180 row_last_cpu=-1 cpumask->row[7]=0 process 4280 row_last_cpu=-1 cpumask->row[6]=0 process 5948 Returning group=0 group_mask=131072 process 3076 row_last_cpu=18 cpumask->row[0]=262144 process 1176 row_last_cpu=-1 cpumask->row[2]=0 process 4280 row_last_cpu=-1 cpumask->row[5]=0 process 3180 row_last_cpu=-1 cpumask->row[6]=0 process 3076 last set cpu in mask is at index 18 process 4412 row_first_cpu=20 cpumask->row[0]=1048576 process 4412 first set cpu in mask is at index 20 process 4280 row_last_cpu=-1 cpumask->row[4]=0 process 3076 Returning group=0 group_mask=262144 process 4412 group=0 group_start=0 group_size=64 search_cpu=20 process 1176 row_last_cpu=-1 cpumask->row[1]=0 process 4412 row_last_cpu=-1 cpumask->row[7]=0 process 4280 row_last_cpu=-1 cpumask->row[3]=0 process 3180 row_last_cpu=-1 cpumask->row[5]=0 process 4412 row_last_cpu=-1 cpumask->row[6]=0 process 4412 row_last_cpu=-1 cpumask->row[5]=0 process 4280 row_last_cpu=-1 cpumask->row[2]=0 process 3180 row_last_cpu=-1 cpumask->row[4]=0 process 1176 row_last_cpu=9 cpumask->row[0]=512 process 4412 row_last_cpu=-1 cpumask->row[4]=0 process 4280 row_last_cpu=-1 cpumask->row[1]=0 process 3180 row_last_cpu=-1 cpumask->row[3]=0 process 4412 row_last_cpu=-1 cpumask->row[3]=0 process 1176 last set cpu in mask is at index 9 process 4280 row_last_cpu=19 cpumask->row[0]=524288 process 4412 row_last_cpu=-1 cpumask->row[2]=0 process 3180 row_last_cpu=-1 cpumask->row[2]=0 process 4280 last set cpu in mask is at index 19 process 1176 Returning group=0 group_mask=512 process 4280 Returning group=0 group_mask=524288 process 4412 row_last_cpu=-1 cpumask->row[1]=0 process 3180 row_last_cpu=-1 cpumask->row[1]=0 process 4412 row_last_cpu=20 cpumask->row[0]=1048576 process 3180 row_last_cpu=16 cpumask->row[0]=65536 process 4412 last set cpu in mask is at index 20 process 3180 last set cpu in mask is at index 16 process 4412 Returning group=0 group_mask=1048576 process 3180 Returning group=0 group_mask=65536 fio_setaffinity: failed to set thread affinity (pid 5948, group 0, mask 20000, GetLastError=87) process 5364 row_first_cpu=21 cpumask->row[0]=2097152 process 5364 first set cpu in mask is at index 21 process 5364 group=0 group_start=0 group_size=64 search_cpu=21 process 5364 row_last_cpu=-1 cpumask->row[7]=0 process 5364 row_last_cpu=-1 cpumask->row[6]=0 process 5364 row_last_cpu=-1 cpumask->row[5]=0 process 5364 row_last_cpu=-1 cpumask->row[4]=0 process 5364 row_last_cpu=-1 cpumask->row[3]=0 process 5364 row_last_cpu=-1 cpumask->row[2]=0 process 5364 row_last_cpu=-1 cpumask->row[1]=0 process 5364 row_last_cpu=21 cpumask->row[0]=2097152 process 5364 last set cpu in mask is at index 21 process 5364 Returning group=0 group_mask=2097152 process 3236 row_first_cpu=22 cpumask->row[0]=4194304 process 3236 first set cpu in mask is at index 22 process 3236 group=0 group_start=0 group_size=64 search_cpu=22 process 3236 row_last_cpu=-1 cpumask->row[7]=0 process 3236 row_last_cpu=-1 cpumask->row[6]=0 process 3236 row_last_cpu=-1 cpumask->row[5]=0 process 3236 row_last_cpu=-1 cpumask->row[4]=0 process 3236 row_last_cpu=-1 cpumask->row[3]=0 process 3236 row_last_cpu=-1 cpumask->row[2]=0 process 3236 row_last_cpu=-1 cpumask->row[1]=0 process 3236 row_last_cpu=22 cpumask->row[0]=4194304 process 3236 last set cpu in mask is at index 22 fio_setaffinity: failed to set thread affinity (pid 3076, group 0, mask 40000, GetLastError=87) process 3236 Returning group=0 group_mask=4194304 process 4340 row_first_cpu=59 cpumask->row[0]=0 process 4340 first set cpu in mask is at index 59 process 4340 group=0 group_start=0 group_size=64 search_cpu=59 process 4340 row_last_cpu=-1 cpumask->row[7]=0 process 4340 row_last_cpu=-1 cpumask->row[6]=0 process 4340 row_last_cpu=-1 cpumask->row[5]=0 process 4340 row_last_cpu=-1 cpumask->row[4]=0 process 4340 row_last_cpu=-1 cpumask->row[3]=0 process 4340 row_last_cpu=-1 cpumask->row[2]=0 process 4340 row_last_cpu=-1 cpumask->row[1]=0 process 4340 row_last_cpu=59 cpumask->row[0]=0 process 4340 last set cpu in mask is at index 59 process 4340 Returning group=0 group_mask=0 process 1084 row_first_cpu=25 cpumask->row[0]=33554432 process 1084 first set cpu in mask is at index 25 process 1084 group=0 group_start=0 group_size=64 search_cpu=25 process 1084 row_last_cpu=-1 cpumask->row[7]=0 fio_setaffinity: failed to set thread affinity (pid 4280, group 0, mask 80000, GetLastError=87) process 1084 row_last_cpu=-1 cpumask->row[6]=0 process 1128 row_first_cpu=26 cpumask->row[0]=67108864 process 1084 row_last_cpu=-1 cpumask->row[5]=0 process 1128 first set cpu in mask is at index 26 process 1084 row_last_cpu=-1 cpumask->row[4]=0 process 1084 row_last_cpu=-1 cpumask->row[3]=0 process 1084 row_last_cpu=-1 cpumask->row[2]=0 process 1128 group=0 group_start=0 group_size=64 search_cpu=26 process 1084 row_last_cpu=-1 cpumask->row[1]=0 process 1128 row_last_cpu=-1 cpumask->row[7]=0 process 1084 row_last_cpu=25 cpumask->row[0]=33554432 process 1128 row_last_cpu=-1 cpumask->row[6]=0 process 1084 last set cpu in mask is at index 25 process 1128 row_last_cpu=-1 cpumask->row[5]=0 process 1084 Returning group=0 group_mask=33554432 process 1128 row_last_cpu=-1 cpumask->row[4]=0 process 1128 row_last_cpu=-1 cpumask->row[3]=0 process 1128 row_last_cpu=-1 cpumask->row[2]=0 process 1128 row_last_cpu=-1 cpumask->row[1]=0 process 1128 row_last_cpu=26 cpumask->row[0]=67108864 process 1128 last set cpu in mask is at index 26 process 1128 Returning group=0 group_mask=67108864 process 4852 row_first_cpu=27 cpumask->row[0]=134217728 process 4852 first set cpu in mask is at index 27 process 4852 group=0 group_start=0 group_size=64 search_cpu=27 process 4852 row_last_cpu=-1 cpumask->row[7]=0 process 4852 row_last_cpu=-1 cpumask->row[6]=0 process 4852 row_last_cpu=-1 cpumask->row[5]=0 process 4852 row_last_cpu=-1 cpumask->row[4]=0 process 4852 row_last_cpu=-1 cpumask->row[3]=0 process 4852 row_last_cpu=-1 cpumask->row[2]=0 process 4852 row_last_cpu=-1 cpumask->row[1]=0 process 4852 row_last_cpu=27 cpumask->row[0]=134217728 process 4852 last set cpu in mask is at index 27 process 4852 Returning group=0 group_mask=134217728 fio_setaffinity: failed to set thread affinity (pid 4412, group 0, mask 100000, GetLastError=87) process 2332 row_first_cpu=28 cpumask->row[0]=268435456 process 2332 first set cpu in mask is at index 28 process 2332 group=0 group_start=0 group_size=64 search_cpu=28 process 2332 row_last_cpu=-1 cpumask->row[7]=0 process 2332 row_last_cpu=-1 cpumask->row[6]=0 process 2332 row_last_cpu=-1 cpumask->row[5]=0 process 2332 row_last_cpu=-1 cpumask->row[4]=0 process 2332 row_last_cpu=-1 cpumask->row[3]=0 process 2332 row_last_cpu=-1 cpumask->row[2]=0 process 2332 row_last_cpu=-1 cpumask->row[1]=0 process 2332 row_last_cpu=28 cpumask->row[0]=268435456 process 2332 last set cpu in mask is at index 28 process 2332 Returning group=0 group_mask=268435456 process 5156 row_first_cpu=29 cpumask->row[0]=536870912 process 5156 first set cpu in mask is at index 29 process 5156 group=0 group_start=0 group_size=64 search_cpu=29 process 5156 row_last_cpu=-1 cpumask->row[7]=0 process 5156 row_last_cpu=-1 cpumask->row[6]=0 process 5156 row_last_cpu=-1 cpumask->row[5]=0 process 5156 row_last_cpu=-1 cpumask->row[4]=0 fio_setaffinity: failed to set thread affinity (pid 4852, group 0, mask 8000000, GetLastError=87) process 5156 row_last_cpu=-1 cpumask->row[3]=0 process 5156 row_last_cpu=-1 cpumask->row[2]=0 process 5156 row_last_cpu=-1 cpumask->row[1]=0 process 5156 row_last_cpu=29 cpumask->row[0]=536870912 process 5156 last set cpu in mask is at index 29 process 5156 Returning group=0 group_mask=536870912 process 548 row_first_cpu=30 cpumask->row[0]=1073741824 process 548 first set cpu in mask is at index 30 process 548 group=0 group_start=0 group_size=64 search_cpu=30 process 548 row_last_cpu=-1 cpumask->row[7]=0 process 548 row_last_cpu=-1 cpumask->row[6]=0 process 548 row_last_cpu=-1 cpumask->row[5]=0 process 548 row_last_cpu=-1 cpumask->row[4]=0 process 548 row_last_cpu=-1 cpumask->row[3]=0 process 548 row_last_cpu=-1 cpumask->row[2]=0 process 548 row_last_cpu=-1 cpumask->row[1]=0 process 548 row_last_cpu=30 cpumask->row[0]=1073741824 process 548 last set cpu in mask is at index 30 process 548 Returning group=0 group_mask=1073741824 process 6020 row_first_cpu=31 cpumask->row[0]=2147483648 fio_setaffinity: failed to set thread affinity (pid 2332, group 0, mask 10000000, GetLastError=87) process 6020 first set cpu in mask is at index 31 process 6020 group=0 group_start=0 group_size=64 search_cpu=31 process 6020 row_last_cpu=-1 cpumask->row[7]=0 process 6020 row_last_cpu=-1 cpumask->row[6]=0 process 6020 row_last_cpu=-1 cpumask->row[5]=0 process 6020 row_last_cpu=-1 cpumask->row[4]=0 process 6020 row_last_cpu=-1 cpumask->row[3]=0 process 1012 row_first_cpu=32 cpumask->row[0]=0 process 6020 row_last_cpu=-1 cpumask->row[2]=0 process 6020 row_last_cpu=-1 cpumask->row[1]=0 process 1012 first set cpu in mask is at index 32 process 6020 row_last_cpu=31 cpumask->row[0]=2147483648 process 6020 last set cpu in mask is at index 31 process 6020 Returning group=0 group_mask=2147483648 process 1012 group=0 group_start=0 group_size=64 search_cpu=32 process 1012 row_last_cpu=-1 cpumask->row[7]=0 process 1012 row_last_cpu=-1 cpumask->row[6]=0 process 1012 row_last_cpu=-1 cpumask->row[5]=0 process 1012 row_last_cpu=-1 cpumask->row[4]=0 process 1012 row_last_cpu=-1 cpumask->row[3]=0 process 1012 row_last_cpu=-1 cpumask->row[2]=0 process 1012 row_last_cpu=-1 cpumask->row[1]=0 process 1012 row_last_cpu=32 cpumask->row[0]=0 process 1012 last set cpu in mask is at index 32 process 1012 Returning group=0 group_mask=0 process 2272 row_first_cpu=33 cpumask->row[0]=0 process 2272 first set cpu in mask is at index 33 process 2272 group=0 group_start=0 group_size=64 search_cpu=33 process 2272 row_last_cpu=-1 cpumask->row[7]=0 process 2272 row_last_cpu=-1 cpumask->row[6]=0 process 2272 row_last_cpu=-1 cpumask->row[5]=0 process 2272 row_last_cpu=-1 cpumask->row[4]=0 process 2272 row_last_cpu=-1 cpumask->row[3]=0 process 2272 row_last_cpu=-1 cpumask->row[2]=0 process 2272 row_last_cpu=-1 cpumask->row[1]=0 process 2272 row_last_cpu=33 cpumask->row[0]=0 process 2272 last set cpu in mask is at index 33 process 2272 Returning group=0 group_mask=0 process 5508 row_first_cpu=34 cpumask->row[0]=0 process 5508 first set cpu in mask is at index 34 process 5508 group=0 group_start=0 group_size=64 search_cpu=34 process 5508 row_last_cpu=-1 cpumask->row[7]=0 process 5508 row_last_cpu=-1 cpumask->row[6]=0 process 5508 row_last_cpu=-1 cpumask->row[5]=0 process 5508 row_last_cpu=-1 cpumask->row[4]=0 process 5508 row_last_cpu=-1 cpumask->row[3]=0 process 5508 row_last_cpu=-1 cpumask->row[2]=0 fio_setaffinity: failed to set thread affinity (pid 5364, group 0, mask 200000, GetLastError=87) process 5508 row_last_cpu=-1 cpumask->row[1]=0 process 5508 row_last_cpu=34 cpumask->row[0]=0 process 5508 last set cpu in mask is at index 34 process 5508 Returning group=0 group_mask=0 process 1904 row_first_cpu=35 cpumask->row[0]=0 process 1904 first set cpu in mask is at index 35 process 1904 group=0 group_start=0 group_size=64 search_cpu=35 process 1904 row_last_cpu=-1 cpumask->row[7]=0 process 1904 row_last_cpu=-1 cpumask->row[6]=0 process 1904 row_last_cpu=-1 cpumask->row[5]=0 process 1904 row_last_cpu=-1 cpumask->row[4]=0 process 1904 row_last_cpu=-1 cpumask->row[3]=0 process 1904 row_last_cpu=-1 cpumask->row[2]=0 process 1904 row_last_cpu=-1 cpumask->row[1]=0 process 1904 row_last_cpu=35 cpumask->row[0]=0 process 1904 last set cpu in mask is at index 35 process 1904 Returning group=0 group_mask=0 process 5356 row_first_cpu=36 cpumask->row[0]=0 process 5356 first set cpu in mask is at index 36 process 5356 group=0 group_start=0 group_size=64 search_cpu=36 process 5356 row_last_cpu=-1 cpumask->row[7]=0 process 5356 row_last_cpu=-1 cpumask->row[6]=0 process 5356 row_last_cpu=-1 cpumask->row[5]=0 process 5356 row_last_cpu=-1 cpumask->row[4]=0 process 5356 row_last_cpu=-1 cpumask->row[3]=0 process 5356 row_last_cpu=-1 cpumask->row[2]=0 process 5356 row_last_cpu=-1 cpumask->row[1]=0 fio_setaffinity: failed to set thread affinity (pid 2272, group 0, mask 200000000, GetLastError=87) process 5356 row_last_cpu=36 cpumask->row[0]=0 process 5356 last set cpu in mask is at index 36 process 2676 row_first_cpu=37 cpumask->row[0]=0 process 5356 Returning group=0 group_mask=0 process 2676 first set cpu in mask is at index 37 process 2676 group=0 group_start=0 group_size=64 search_cpu=37 process 2676 row_last_cpu=-1 cpumask->row[7]=0 process 2676 row_last_cpu=-1 cpumask->row[6]=0 process 2676 row_last_cpu=-1 cpumask->row[5]=0 process 2676 row_last_cpu=-1 cpumask->row[4]=0 process 2676 row_last_cpu=-1 cpumask->row[3]=0 process 2676 row_last_cpu=-1 cpumask->row[2]=0 process 2676 row_last_cpu=-1 cpumask->row[1]=0 process 2676 row_last_cpu=37 cpumask->row[0]=0 process 2676 last set cpu in mask is at index 37 process 2676 Returning group=0 group_mask=0 process 4104 row_first_cpu=38 cpumask->row[0]=0 process 4104 first set cpu in mask is at index 38 process 4104 group=0 group_start=0 group_size=64 search_cpu=38 process 4104 row_last_cpu=-1 cpumask->row[7]=0 process 4104 row_last_cpu=-1 cpumask->row[6]=0 process 4104 row_last_cpu=-1 cpumask->row[5]=0 process 4104 row_last_cpu=-1 cpumask->row[4]=0 process 4104 row_last_cpu=-1 cpumask->row[3]=0 clock setaffinity failed: No error process 4104 row_last_cpu=-1 cpumask->row[2]=0 process 4104 row_last_cpu=-1 cpumask->row[1]=0 process 4104 row_last_cpu=38 cpumask->row[0]=0 process 4104 last set cpu in mask is at index 38 process 4104 Returning group=0 group_mask=0 process 2660 row_first_cpu=39 cpumask->row[0]=0 process 2660 first set cpu in mask is at index 39 process 2660 group=0 group_start=0 group_size=64 search_cpu=39 process 2660 row_last_cpu=-1 cpumask->row[7]=0 process 2660 row_last_cpu=-1 cpumask->row[6]=0 process 2660 row_last_cpu=-1 cpumask->row[5]=0 process 2660 row_last_cpu=-1 cpumask->row[4]=0 process 2660 row_last_cpu=-1 cpumask->row[3]=0 process 2660 row_last_cpu=-1 cpumask->row[2]=0 process 2660 row_last_cpu=-1 cpumask->row[1]=0 process 2660 row_last_cpu=39 cpumask->row[0]=0 process 2660 last set cpu in mask is at index 39 process 2660 Returning group=0 group_mask=0 process 5136 row_first_cpu=40 cpumask->row[0]=0 process 5136 first set cpu in mask is at index 40 process 5136 group=0 group_start=0 group_size=64 search_cpu=40 process 5136 row_last_cpu=-1 cpumask->row[7]=0 process 5136 row_last_cpu=-1 cpumask->row[6]=0 process 5136 row_last_cpu=-1 cpumask->row[5]=0 process 5136 row_last_cpu=-1 cpumask->row[4]=0 process 5136 row_last_cpu=-1 cpumask->row[3]=0 process 5136 row_last_cpu=-1 cpumask->row[2]=0 process 5136 row_last_cpu=-1 cpumask->row[1]=0 process 5136 row_last_cpu=40 cpumask->row[0]=0 process 5136 last set cpu in mask is at index 40 process 5136 Returning group=0 group_mask=0 process 5352 row_first_cpu=41 cpumask->row[0]=0 process 5352 first set cpu in mask is at index 41 process 5352 group=0 group_start=0 group_size=64 search_cpu=41 process 5352 row_last_cpu=-1 cpumask->row[7]=0 process 5352 row_last_cpu=-1 cpumask->row[6]=0 process 5352 row_last_cpu=-1 cpumask->row[5]=0 process 5352 row_last_cpu=-1 cpumask->row[4]=0 process 5352 row_last_cpu=-1 cpumask->row[3]=0 process 5352 row_last_cpu=-1 cpumask->row[2]=0 process 5352 row_last_cpu=-1 cpumask->row[1]=0 process 5352 row_last_cpu=41 cpumask->row[0]=0 fio_setaffinity: failed to set thread affinity (pid 4340, group 0, mask 800000000000000, GetLastError=87) process 5352 last set cpu in mask is at index 41 process 5352 Returning group=0 group_mask=0 process 5916 row_first_cpu=42 cpumask->row[0]=0 process 5916 first set cpu in mask is at index 42 process 5916 group=0 group_start=0 group_size=64 search_cpu=42 process 5916 row_last_cpu=-1 cpumask->row[7]=0 process 5916 row_last_cpu=-1 cpumask->row[6]=0 process 5916 row_last_cpu=-1 cpumask->row[5]=0 process 5916 row_last_cpu=-1 cpumask->row[4]=0 process 5916 row_last_cpu=-1 cpumask->row[3]=0 process 5916 row_last_cpu=-1 cpumask->row[2]=0 process 1492 row_first_cpu=43 cpumask->row[0]=0 fio_setaffinity: failed to set thread affinity (pid 5136, group 0, mask 10000000000, GetLastError=87) process 1492 first set cpu in mask is at index 43 process 5916 row_last_cpu=-1 cpumask->row[1]=0 process 5916 row_last_cpu=42 cpumask->row[0]=0 process 1492 group=0 group_start=0 group_size=64 search_cpu=43 process 1492 row_last_cpu=-1 cpumask->row[7]=0 process 5916 last set cpu in mask is at index 42 process 1492 row_last_cpu=-1 cpumask->row[6]=0 process 5916 Returning group=0 group_mask=0 process 1492 row_last_cpu=-1 cpumask->row[5]=0 process 1492 row_last_cpu=-1 cpumask->row[4]=0 process 4948 row_first_cpu=44 cpumask->row[0]=0 process 1492 row_last_cpu=-1 cpumask->row[3]=0 process 1492 row_last_cpu=-1 cpumask->row[2]=0 process 4948 first set cpu in mask is at index 44 process 1492 row_last_cpu=-1 cpumask->row[1]=0 process 1492 row_last_cpu=43 cpumask->row[0]=0 process 1492 last set cpu in mask is at index 43 process 1492 Returning group=0 group_mask=0 process 4948 group=0 group_start=0 group_size=64 search_cpu=44 process 4948 row_last_cpu=-1 cpumask->row[7]=0 process 4948 row_last_cpu=-1 cpumask->row[6]=0 process 4948 row_last_cpu=-1 cpumask->row[5]=0 process 4948 row_last_cpu=-1 cpumask->row[4]=0 process 4948 row_last_cpu=-1 cpumask->row[3]=0 process 4948 row_last_cpu=-1 cpumask->row[2]=0 process 4948 row_last_cpu=-1 cpumask->row[1]=0 process 4948 row_last_cpu=44 cpumask->row[0]=0 process 4948 last set cpu in mask is at index 44 process 4948 Returning group=0 group_mask=0 process 4120 row_first_cpu=45 cpumask->row[0]=0 process 4120 first set cpu in mask is at index 45 process 4120 group=0 group_start=0 group_size=64 search_cpu=45 process 4120 row_last_cpu=-1 cpumask->row[7]=0 process 4120 row_last_cpu=-1 cpumask->row[6]=0 process 4120 row_last_cpu=-1 cpumask->row[5]=0 process 4120 row_last_cpu=-1 cpumask->row[4]=0 process 4120 row_last_cpu=-1 cpumask->row[3]=0 process 4120 row_last_cpu=-1 cpumask->row[2]=0 fio_setaffinity: failed to set thread affinity (pid 1084, group 0, mask 2000000, GetLastError=87) process 4120 row_last_cpu=-1 cpumask->row[1]=0 process 4120 row_last_cpu=45 cpumask->row[0]=0 process 4120 last set cpu in mask is at index 45 process 4120 Returning group=0 group_mask=0 process 1924 row_first_cpu=46 cpumask->row[0]=0 process 1924 first set cpu in mask is at index 46 process 1924 group=0 group_start=0 group_size=64 search_cpu=46 process 1924 row_last_cpu=-1 cpumask->row[7]=0 process 1924 row_last_cpu=-1 cpumask->row[6]=0 process 1924 row_last_cpu=-1 cpumask->row[5]=0 process 1924 row_last_cpu=-1 cpumask->row[4]=0 process 1924 row_last_cpu=-1 cpumask->row[3]=0 process 1924 row_last_cpu=-1 cpumask->row[2]=0 process 1924 row_last_cpu=-1 cpumask->row[1]=0 process 1924 row_last_cpu=46 cpumask->row[0]=0 process 1924 last set cpu in mask is at index 46 process 1924 Returning group=0 group_mask=0 process 5848 row_first_cpu=47 cpumask->row[0]=0 process 5848 first set cpu in mask is at index 47 process 5848 group=0 group_start=0 group_size=64 search_cpu=47 process 5848 row_last_cpu=-1 cpumask->row[7]=0 process 5848 row_last_cpu=-1 cpumask->row[6]=0 process 5848 row_last_cpu=-1 cpumask->row[5]=0 process 5848 row_last_cpu=-1 cpumask->row[4]=0 process 5848 row_last_cpu=-1 cpumask->row[3]=0 process 5848 row_last_cpu=-1 cpumask->row[2]=0 process 5848 row_last_cpu=-1 cpumask->row[1]=0 process 5848 row_last_cpu=47 cpumask->row[0]=0 process 5848 last set cpu in mask is at index 47 process 5848 Returning group=0 group_mask=0 process 4156 row_first_cpu=48 cpumask->row[0]=0 process 4156 first set cpu in mask is at index 48 process 4156 group=0 group_start=0 group_size=64 search_cpu=48 process 4156 row_last_cpu=-1 cpumask->row[7]=0 process 4156 row_last_cpu=-1 cpumask->row[6]=0 process 4156 row_last_cpu=-1 cpumask->row[5]=0 process 4156 row_last_cpu=-1 cpumask->row[4]=0 fio_setaffinity: failed to set thread affinity (pid 1128, group 0, mask 4000000, GetLastError=87) process 4156 row_last_cpu=-1 cpumask->row[3]=0 process 2096 row_first_cpu=49 cpumask->row[0]=0 process 4156 row_last_cpu=-1 cpumask->row[2]=0 process 4156 row_last_cpu=-1 cpumask->row[1]=0 process 4156 row_last_cpu=48 cpumask->row[0]=0 process 2096 first set cpu in mask is at index 49 process 4156 last set cpu in mask is at index 48 process 4156 Returning group=0 group_mask=0 process 2096 group=0 group_start=0 group_size=64 search_cpu=49 process 2096 row_last_cpu=-1 cpumask->row[7]=0 process 2096 row_last_cpu=-1 cpumask->row[6]=0 process 2096 row_last_cpu=-1 cpumask->row[5]=0 process 2096 row_last_cpu=-1 cpumask->row[4]=0 process 2096 row_last_cpu=-1 cpumask->row[3]=0 process 2096 row_last_cpu=-1 cpumask->row[2]=0 process 2096 row_last_cpu=-1 cpumask->row[1]=0 process 2096 row_last_cpu=49 cpumask->row[0]=0 process 2096 last set cpu in mask is at index 49 process 2096 Returning group=0 group_mask=0 process 6084 row_first_cpu=50 cpumask->row[0]=0 process 6084 first set cpu in mask is at index 50 process 6084 group=0 group_start=0 group_size=64 search_cpu=50 process 6084 row_last_cpu=-1 cpumask->row[7]=0 process 6084 row_last_cpu=-1 cpumask->row[6]=0 process 6084 row_last_cpu=-1 cpumask->row[5]=0 process 6084 row_last_cpu=-1 cpumask->row[4]=0 process 6084 row_last_cpu=-1 cpumask->row[3]=0 process 6084 row_last_cpu=-1 cpumask->row[2]=0 process 6084 row_last_cpu=-1 cpumask->row[1]=0 process 2772 row_first_cpu=51 cpumask->row[0]=0 process 6084 row_last_cpu=50 cpumask->row[0]=0 fio_setaffinity: failed to set thread affinity (pid 5848, group 0, mask 800000000000, GetLastError=87) process 2772 first set cpu in mask is at index 51 process 6084 last set cpu in mask is at index 50 process 6084 Returning group=0 group_mask=0 process 2772 group=0 group_start=0 group_size=64 search_cpu=51 process 2772 row_last_cpu=-1 cpumask->row[7]=0 process 3964 row_first_cpu=52 cpumask->row[0]=0 process 2772 row_last_cpu=-1 cpumask->row[6]=0 process 3964 first set cpu in mask is at index 52 process 2772 row_last_cpu=-1 cpumask->row[5]=0 process 2772 row_last_cpu=-1 cpumask->row[4]=0 process 3964 group=0 group_start=0 group_size=64 search_cpu=52 process 3964 row_last_cpu=-1 cpumask->row[7]=0 process 2772 row_last_cpu=-1 cpumask->row[3]=0 process 3964 row_last_cpu=-1 cpumask->row[6]=0 process 3964 row_last_cpu=-1 cpumask->row[5]=0 process 2772 row_last_cpu=-1 cpumask->row[2]=0 process 3964 row_last_cpu=-1 cpumask->row[4]=0 process 3964 row_last_cpu=-1 cpumask->row[3]=0 process 2772 row_last_cpu=-1 cpumask->row[1]=0 process 3964 row_last_cpu=-1 cpumask->row[2]=0 process 3964 row_last_cpu=-1 cpumask->row[1]=0 process 2772 row_last_cpu=51 cpumask->row[0]=0 process 3964 row_last_cpu=52 cpumask->row[0]=0 process 2772 last set cpu in mask is at index 51 process 3964 last set cpu in mask is at index 52 process 2772 Returning group=0 group_mask=0 process 3964 Returning group=0 group_mask=0 process 4384 row_first_cpu=53 cpumask->row[0]=0 process 4384 first set cpu in mask is at index 53 process 4384 group=0 group_start=0 group_size=64 search_cpu=53 process 4384 row_last_cpu=-1 cpumask->row[7]=0 process 4384 row_last_cpu=-1 cpumask->row[6]=0 process 4384 row_last_cpu=-1 cpumask->row[5]=0 process 4384 row_last_cpu=-1 cpumask->row[4]=0 process 4384 row_last_cpu=-1 cpumask->row[3]=0 clock setaffinity failed: No error process 4384 row_last_cpu=-1 cpumask->row[2]=0 process 4384 row_last_cpu=-1 cpumask->row[1]=0 process 4384 row_last_cpu=53 cpumask->row[0]=0 process 4384 last set cpu in mask is at index 53 process 4384 Returning group=0 group_mask=0 process 5244 row_first_cpu=54 cpumask->row[0]=0 process 5244 first set cpu in mask is at index 54 process 5244 group=0 group_start=0 group_size=64 search_cpu=54 process 5244 row_last_cpu=-1 cpumask->row[7]=0 process 5244 row_last_cpu=-1 cpumask->row[6]=0 process 5244 row_last_cpu=-1 cpumask->row[5]=0 process 5244 row_last_cpu=-1 cpumask->row[4]=0 process 5244 row_last_cpu=-1 cpumask->row[3]=0 process 5244 row_last_cpu=-1 cpumask->row[2]=0 process 5244 row_last_cpu=-1 cpumask->row[1]=0 process 5244 row_last_cpu=54 cpumask->row[0]=0 process 5244 last set cpu in mask is at index 54 process 5244 Returning group=0 group_mask=0 process 5076 row_first_cpu=55 cpumask->row[0]=0 process 5076 first set cpu in mask is at index 55 process 5076 group=0 group_start=0 group_size=64 search_cpu=55 process 5076 row_last_cpu=-1 cpumask->row[7]=0 process 5076 row_last_cpu=-1 cpumask->row[6]=0 process 5076 row_last_cpu=-1 cpumask->row[5]=0 process 5076 row_last_cpu=-1 cpumask->row[4]=0 process 5076 row_last_cpu=-1 cpumask->row[3]=0 process 5076 row_last_cpu=-1 cpumask->row[2]=0 process 5076 row_last_cpu=-1 cpumask->row[1]=0 clock setaffinity failed: No error process 5076 row_last_cpu=55 cpumask->row[0]=0 process 5076 last set cpu in mask is at index 55 process 5076 Returning group=0 group_mask=0 process 5344 row_first_cpu=56 cpumask->row[0]=0 process 5344 first set cpu in mask is at index 56 process 5344 group=0 group_start=0 group_size=64 search_cpu=56 process 5344 row_last_cpu=-1 cpumask->row[7]=0 process 5344 row_last_cpu=-1 cpumask->row[6]=0 process 5344 row_last_cpu=-1 cpumask->row[5]=0 process 5344 row_last_cpu=-1 cpumask->row[4]=0 process 5344 row_last_cpu=-1 cpumask->row[3]=0 process 5344 row_last_cpu=-1 cpumask->row[2]=0 process 5344 row_last_cpu=-1 cpumask->row[1]=0 process 5344 row_last_cpu=56 cpumask->row[0]=0 process 5344 last set cpu in mask is at index 56 process 5344 Returning group=0 group_mask=0 process 3204 row_first_cpu=57 cpumask->row[0]=0 process 3204 first set cpu in mask is at index 57 process 3204 group=0 group_start=0 group_size=64 search_cpu=57 process 3204 row_last_cpu=-1 cpumask->row[7]=0 process 3204 row_last_cpu=-1 cpumask->row[6]=0 process 3204 row_last_cpu=-1 cpumask->row[5]=0 process 3204 row_last_cpu=-1 cpumask->row[4]=0 process 3204 row_last_cpu=-1 cpumask->row[3]=0 process 3204 row_last_cpu=-1 cpumask->row[2]=0 process 3204 row_last_cpu=-1 cpumask->row[1]=0 process 3204 row_last_cpu=57 cpumask->row[0]=0 process 3204 last set cpu in mask is at index 57 process 3204 Returning group=0 group_mask=0 clock setaffinity failed: No error process 3848 row_first_cpu=58 cpumask->row[0]=0 process 3848 first set cpu in mask is at index 58 process 3848 group=0 group_start=0 group_size=64 search_cpu=58 process 3848 row_last_cpu=-1 cpumask->row[7]=0 process 3848 row_last_cpu=-1 cpumask->row[6]=0 process 3848 row_last_cpu=-1 cpumask->row[5]=0 process 3848 row_last_cpu=-1 cpumask->row[4]=0 process 3848 row_last_cpu=-1 cpumask->row[3]=0 process 3848 row_last_cpu=-1 cpumask->row[2]=0 process 3848 row_last_cpu=-1 cpumask->row[1]=0 process 3848 row_last_cpu=58 cpumask->row[0]=0 process 3848 last set cpu in mask is at index 58 process 3848 Returning group=0 group_mask=0 process 440 row_first_cpu=23 cpumask->row[0]=8388608 process 440 first set cpu in mask is at index 23 process 440 group=0 group_start=0 group_size=64 search_cpu=23 process 440 row_last_cpu=-1 cpumask->row[7]=0 process 440 row_last_cpu=-1 cpumask->row[6]=0 process 440 row_last_cpu=-1 cpumask->row[5]=0 process 440 row_last_cpu=-1 cpumask->row[4]=0 process 440 row_last_cpu=-1 cpumask->row[3]=0 process 440 row_last_cpu=-1 cpumask->row[2]=0 process 440 row_last_cpu=-1 cpumask->row[1]=0 process 440 row_last_cpu=23 cpumask->row[0]=8388608 process 440 last set cpu in mask is at index 23 fio_setaffinity: failed to set thread affinity (pid 5156, group 0, mask 20000000, GetLastError=87) process 440 Returning group=0 group_mask=8388608 process 4420 row_first_cpu=60 cpumask->row[0]=0 process 4420 first set cpu in mask is at index 60 process 4420 group=0 group_start=0 group_size=64 search_cpu=60 process 4420 row_last_cpu=-1 cpumask->row[7]=0 process 4420 row_last_cpu=-1 cpumask->row[6]=0 process 4420 row_last_cpu=-1 cpumask->row[5]=0 process 4420 row_last_cpu=-1 cpumask->row[4]=0 process 4420 row_last_cpu=-1 cpumask->row[3]=0 process 4420 row_last_cpu=-1 cpumask->row[2]=0 process 4420 row_last_cpu=-1 cpumask->row[1]=0 process 4420 row_last_cpu=60 cpumask->row[0]=0 process 4420 last set cpu in mask is at index 60 process 4420 Returning group=0 group_mask=0 process 5164 row_first_cpu=61 cpumask->row[0]=0 process 5164 first set cpu in mask is at index 61 process 5164 group=0 group_start=0 group_size=64 search_cpu=61 process 5164 row_last_cpu=-1 cpumask->row[7]=0 process 5164 row_last_cpu=-1 cpumask->row[6]=0 process 5164 row_last_cpu=-1 cpumask->row[5]=0 process 5164 row_last_cpu=-1 cpumask->row[4]=0 process 5164 row_last_cpu=-1 cpumask->row[3]=0 process 5164 row_last_cpu=-1 cpumask->row[2]=0 process 5164 row_last_cpu=-1 cpumask->row[1]=0 fio_setaffinity: failed to set thread affinity (pid 3848, group 0, mask 400000000000000, GetLastError=87) process 5164 row_last_cpu=61 cpumask->row[0]=0 process 5164 last set cpu in mask is at index 61 process 5164 Returning group=0 group_mask=0 process 5176 row_first_cpu=62 cpumask->row[0]=0 process 5176 first set cpu in mask is at index 62 process 5176 group=0 group_start=0 group_size=64 search_cpu=62 process 5176 row_last_cpu=-1 cpumask->row[7]=0 process 5176 row_last_cpu=-1 cpumask->row[6]=0 process 5176 row_last_cpu=-1 cpumask->row[5]=0 process 5176 row_last_cpu=-1 cpumask->row[4]=0 process 5176 row_last_cpu=-1 cpumask->row[3]=0 process 5176 row_last_cpu=-1 cpumask->row[2]=0 process 5176 row_last_cpu=-1 cpumask->row[1]=0 process 5176 row_last_cpu=62 cpumask->row[0]=0 process 5176 last set cpu in mask is at index 62 process 5176 Returning group=0 group_mask=0 process 5152 row_first_cpu=63 cpumask->row[0]=0 process 5152 first set cpu in mask is at index 63 process 5152 group=0 group_start=0 group_size=64 search_cpu=63 process 5152 row_last_cpu=-1 cpumask->row[7]=0 process 5152 row_last_cpu=-1 cpumask->row[6]=0 process 5152 row_last_cpu=-1 cpumask->row[5]=0 process 5152 row_last_cpu=-1 cpumask->row[4]=0 process 5152 row_last_cpu=-1 cpumask->row[3]=0 process 5152 row_last_cpu=-1 cpumask->row[2]=0 process 5152 row_last_cpu=-1 cpumask->row[1]=0 process 5152 row_last_cpu=63 cpumask->row[0]=0 process 5152 last set cpu in mask is at index 63 process 5152 Returning group=0 group_mask=0 clock setaffinity failed: No error process 3852 row_first_cpu=24 cpumask->row[0]=16777216 process 3852 first set cpu in mask is at index 24 process 3852 group=0 group_start=0 group_size=64 search_cpu=24 process 3852 row_last_cpu=-1 cpumask->row[7]=0 process 3852 row_last_cpu=-1 cpumask->row[6]=0 process 3852 row_last_cpu=-1 cpumask->row[5]=0 process 3852 row_last_cpu=-1 cpumask->row[4]=0 process 3852 row_last_cpu=-1 cpumask->row[3]=0 process 3852 row_last_cpu=-1 cpumask->row[2]=0 process 3852 row_last_cpu=-1 cpumask->row[1]=0 process 3852 row_last_cpu=24 cpumask->row[0]=16777216 process 3852 last set cpu in mask is at index 24 process 3852 Returning group=0 group_mask=16777216 fio_setaffinity: failed to set thread affinity (pid 5152, group 0, mask 8000000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 1012, group 0, mask 100000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 3236, group 0, mask 400000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 5508, group 0, mask 400000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 1904, group 0, mask 800000000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 5356, group 0, mask 1000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 2676, group 0, mask 2000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 4104, group 0, mask 4000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 2660, group 0, mask 8000000000, GetLastError=87) clock setaffinity failed: No error clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 5352, group 0, mask 20000000000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 5916, group 0, mask 40000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 1492, group 0, mask 80000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 4948, group 0, mask 100000000000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 4120, group 0, mask 200000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 1924, group 0, mask 400000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 3180, group 0, mask 10000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 4156, group 0, mask 1000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 2096, group 0, mask 2000000000000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 6084, group 0, mask 4000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 2772, group 0, mask 8000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 3964, group 0, mask 10000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 4384, group 0, mask 20000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 5244, group 0, mask 40000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 5076, group 0, mask 80000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 5344, group 0, mask 100000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 3204, group 0, mask 200000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 548, group 0, mask 40000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 440, group 0, mask 800000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 4420, group 0, mask 1000000000000000, GetLastError=87) clock setaffinity failed: No error fio_setaffinity: failed to set thread affinity (pid 5164, group 0, mask 2000000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 5176, group 0, mask 4000000000000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 6020, group 0, mask 80000000, GetLastError=87) fio_setaffinity: failed to set thread affinity (pid 3852, group 0, mask 1000000, GetLastError=87) clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error clock setaffinity failed: No error Starting 1 thread process 3176 pid=0: runstate NOT_CREATED -> SETTING_UP process 3176 pid=0: runstate SETTING_UP -> NOT_CREATED process 3176 pid=0: runstate NOT_CREATED -> CREATED process 3176 will pthread_create process 4324 jobs pid=4324 started process 4324 pid=4324: runstate CREATED -> INITIALIZED process 3176 pid=4324: runstate INITIALIZED -> RUNNING process 4324 row_first_cpu=0 cpumask->row[0]=3 process 4324 first set cpu in mask is at index 0 process 4324 group=0 group_start=0 group_size=64 search_cpu=0 process 4324 row_last_cpu=-1 cpumask->row[7]=0 process 4324 row_last_cpu=-1 cpumask->row[6]=0 process 4324 row_last_cpu=-1 cpumask->row[5]=0 process 4324 row_last_cpu=-1 cpumask->row[4]=0 process 4324 row_last_cpu=-1 cpumask->row[3]=0 process 4324 row_last_cpu=-1 cpumask->row[2]=0 process 4324 row_last_cpu=-1 cpumask->row[1]=0 process 4324 row_last_cpu=1 cpumask->row[0]=3 process 4324 last set cpu in mask is at index 1 process 4324 Returning group=0 group_mask=3 process 4324 row_first_cpu=0 cpumask->row[0]=3 process 4324 first set cpu in mask is at index 0 process 4324 group=0 group_start=0 group_size=64 search_cpu=0 process 4324 row_last_cpu=-1 cpumask->row[7]=0 process 4324 row_last_cpu=-1 cpumask->row[6]=0 process 4324 row_last_cpu=-1 cpumask->row[5]=0 process 4324 row_last_cpu=-1 cpumask->row[4]=0 process 4324 row_last_cpu=-1 cpumask->row[3]=0 process 4324 row_last_cpu=-1 cpumask->row[2]=0 process 4324 row_last_cpu=-1 cpumask->row[1]=0 process 4324 row_last_cpu=1 cpumask->row[0]=3 process 4324 last set cpu in mask is at index 1 process 4324 Returning group=0 group_mask=3 process 4324 pid=4324: runstate RUNNING -> FINISHING process 4324 pid=4324: runstate FINISHING -> EXITED process 3176 pid=4324: runstate EXITED -> REAPED process 3176 terminate group_id=-1 process 3176 setting terminate on test/4324 process 2204 pid=4324: runstate REAPED -> FINISHING process 2204 pid=4324: runstate FINISHING -> REAPED Regards, Jeff -----Original Message----- From: Elliott, Robert (Persistent Memory) [mailto:elliott@xxxxxxx] Sent: Monday, October 29, 2018 2:23 PM To: 'Sitsofe Wheeler' <sitsofe@xxxxxxxxx>; Jeff Furlong <jeff.furlong@xxxxxxx> Cc: fio <fio@xxxxxxxxxxxxxxx> Subject: RE: fio win affinity broken > -----Original Message----- > From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On > Behalf Of Sitsofe Wheeler > Sent: Monday, October 29, 2018 3:12 PM > Subject: Re: fio win affinity broken > > On Mon, 29 Oct 2018 at 20:04, Jeff Furlong <jeff.furlong@xxxxxxx> wrote: > > > > Back in > > http://git.kernel.dk/cgit/fio/commit/?id=c479640d6208236744f0562b1e7 > > 9535eec290e2b several updates were made for Windows CPU affinity. However, using the cpus_allowed since fio 3.6 and latest git appears to throw error messages. Consider a job: > > > > fio --name=test --ioengine=windowsaio --direct=1 --rw=write > > --overwrite=1 -- > > filename=\\.\PHYSICALDRIVE1 --runtime=1s --thread --cpus_allowed=0-1 > > --cpus_allowed_policy=shared Usually split is a better choice than shared - do you really want threads hopping between CPUs? > > fio_setaffinity: failed to set thread affinity (pid 2768, group 0, > > mask 10000, GetLastError=87) > > fio_setaffinity: failed to set thread affinity (pid 2372, group 0, > > mask 20000, GetLastError=87) clock setaffinity failed: No error Those are the clock threads, not the job threads. At startup, fio spawns threads on all CPUs to measure the clocks (fio_monotonic_clocktest). A Windows difference from linux I mentioned last year was: If you've constrained the CPU affinity outside fio, some of those will fail. In Windows, something like START /AFFINITY 0x55555555 fio ... can cause half of the clock threads to fail. In linux, processes and threads are not restricted to their parent's affinity mask. Although they inherit by default, they may call sched_setaffinity with any values. I hoped to work on a patch to limit the clock threads to the CPUs within the parent affinity mask and the logical OR of all the CPUs that are going to be used by the jobs, but then lost access to the Windows system I was using and never went back to it. > > It seems err 87 is ERROR_INVALID_PARAMETER, but any idea why > > "GROUP_AFFINITY struct's Reserved > members are not initialised to 0?" That's just one possible reason for failure; more likely causes are an invalid Mask or Group for the system under test. --- Robert Elliott, HPE Persistent Memory
