I bisected the kernel and it looks like the performance of the Connect-IB card goes down and the performance of the ConnectX-3 card goes up with this commit (but I'm not sure why this would cause this): ab46db0a3325a064bb24e826b12995d157565efb is the first bad commit commit ab46db0a3325a064bb24e826b12995d157565efb Author: Jiri Olsa <jolsa@xxxxxxxxxx> Date: Thu Dec 3 10:06:43 2015 +0100 perf stat: Use perf_evlist__enable in handle_initial_delay No need to mimic the behaviour of perf_evlist__enable, we can use it directly. Signed-off-by: Jiri Olsa <jolsa@xxxxxxxxxx> Tested-by: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> Cc: Adrian Hunter <adrian.hunter@xxxxxxxxx> Cc: David Ahern <dsahern@xxxxxxxxx> Cc: Namhyung Kim <namhyung@xxxxxxxxxx> Cc: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx> Link: http://lkml.kernel.org/r/1449133606-14429-5-git-send-email-jolsa@xxxxxxxxxx Signed-off-by: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> :040000 040000 67e69893bf6d47b372e08d7089d37a7b9f602fa7 b63d9b366f078eabf86f4da3d1cc53ae7434a949 M tools 4.4.0_rc2_3e27c920 sdc;10.218.128.17;5291495;1322873;15853 sde;10.218.202.17;4966024;1241506;16892 sdh;10.218.203.17;4980471;1245117;16843 sdk;10.218.204.17;4966612;1241653;16890 sdd;10.219.128.17;5060084;1265021;16578 sdf;10.219.202.17;5065278;1266319;16561 sdi;10.219.203.17;5047600;1261900;16619 sdl;10.219.204.17;5036992;1259248;16654 sdn;10.220.128.17;3775081;943770;22221 sdg;10.220.202.17;3758336;939584;22320 sdj;10.220.203.17;3792832;948208;22117 sdm;10.220.204.17;3771516;942879;22242 4.4.0_rc2_ab46db0a sdc;10.218.128.17;3792146;948036;22121 sdf;10.218.202.17;3738405;934601;22439 sdj;10.218.203.17;3764239;941059;22285 sdl;10.218.204.17;3785302;946325;22161 sdd;10.219.128.17;3762382;940595;22296 sdg;10.219.202.17;3765760;941440;22276 sdi;10.219.203.17;3873751;968437;21655 sdm;10.219.204.17;3769483;942370;22254 sde;10.220.128.17;5022517;1255629;16702 sdh;10.220.202.17;5018911;1254727;16714 sdk;10.220.203.17;5037295;1259323;16653 sdn;10.220.204.17;5033064;1258266;16667 ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jun 8, 2016 at 9:33 AM, Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote: > With 4.1.15, the C-IB card gets about 1.15 MIOPs, while the CX3 gets > about 0.99 MIOPs. But starting with the 4.4.4 kernel, the C-IB card > drops to 0.96 MIOPs and the CX3 card jumps to 1.25 MIOPs. In the 4.6.0 > kernel, both cards drop, the C-IB to 0.82 MIOPs and the CX3 to 1.15 > MIOPs. I confirmed this morning that the card order was swapped on the > 4.6.0 kernel and it was not different ports of the C-IB performing > differently, but different cards. > > Given the limitations of the PCIe 8x port for the CX3, I think 1.25 > MIOPs is about the best we can do there. In summary, the performance > of the C-IB card drops after 4.1.15 and gets progressively worse as > the kernels increase. The CX3 card peaks at the 4.4.4 kernel and > degrades a bit on the 4.6.0 kernel. > > Increasing the IO depth by adding jobs does not improve performance, > it actually decreases performance. Based on an average of 4 runs at > each job number from 1-80, the Goldilocks zone is 31-57 jobs where the > difference in performance is less than 1%. > > Similarly, increasing block request size does not really change the > figures to reach line speed. > > Here is the output of the 4.6.0 kernel with 4M bs: > sdc;10.218.128.17;3354638;819;25006 > sdf;10.218.202.17;3376920;824;24841 > sdm;10.218.203.17;3367431;822;24911 > sdk;10.218.204.17;3378960;824;24826 > sde;10.219.128.17;3366350;821;24919 > sdl;10.219.202.17;3379641;825;24821 > sdg;10.219.203.17;3391254;827;24736 > sdn;10.219.204.17;3401706;830;24660 > sdd;10.220.128.17;4597505;1122;18246 > sdi;10.220.202.17;4594231;1121;18259 > sdj;10.220.203.17;4667598;1139;17972 > sdh;10.220.204.17;4628197;1129;18125 > > The CPU on the target is a kworker thread at 96%, but no single > processor over 15%. The initiator has low fio CPU utilization (<10%) > for each job and no single CPU over 22% utilized. > > I have tried manually spreading the IRQ affinity over the processors > of the respective NUMA nodes and there was no noticeable change in > performance when doing so. > > Loading ib_iser on the initiator shows maybe a slight increase in performance: > > sdc;10.218.128.17;3396885;849221;24695 > sdf;10.218.202.17;3429240;857310;24462 > sdi;10.218.203.17;3454234;863558;24285 > sdm;10.218.204.17;3391666;847916;24733 > sde;10.219.128.17;3403914;850978;24644 > sdh;10.219.202.17;3491034;872758;24029 > sdk;10.219.203.17;3390569;847642;24741 > sdl;10.219.204.17;3498898;874724;23975 > sdd;10.220.128.17;4664743;1166185;17983 > sdg;10.220.202.17;4624880;1156220;18138 > sdj;10.220.203.17;4616227;1154056;18172 > sdn;10.220.204.17;4619786;1154946;18158 > > I'd like to see the C-IB card at 1.25+ MIOPs (I know that the target > can do that performance and we were limited on the CX3 by the PCIe bus > which isn't an issue with the 16x C-IB card for a single port). > Although the loss of performance in the CX3 card is concerning, I'm > mostly focused on the C-IB card at the moment. I will probably start > bisecting 4.1.15 to 4.4.4 to see if I can identify when the > performance of the C-IB card degrades. > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > > On Wed, Jun 8, 2016 at 7:52 AM, Max Gurtovoy <maxg@xxxxxxxxxxxx> wrote: >> >> >> On 6/8/2016 1:37 AM, Robert LeBlanc wrote: >>> >>> On the 4.1.15 kernel: >>> sdc;10.218.128.17;3971878;992969;21120 >>> sdd;10.218.202.17;3967745;991936;21142 >>> sdg;10.218.203.17;3938128;984532;21301 >>> sdk;10.218.204.17;3952602;988150;21223 >>> sdn;10.219.128.17;4615719;1153929;18174 >>> sdf;10.219.202.17;4622331;1155582;18148 >>> sdi;10.219.203.17;4602297;1150574;18227 >>> sdl;10.219.204.17;4565477;1141369;18374 >>> sde;10.220.128.17;4594986;1148746;18256 >>> sdh;10.220.202.17;4590209;1147552;18275 >>> sdj;10.220.203.17;4599017;1149754;18240 >>> sdm;10.220.204.17;4610898;1152724;18193 >>> >>> On the 4.6.0 kernel: >>> sdc;10.218.128.17;3239219;809804;25897 >>> sdf;10.218.202.17;3321300;830325;25257 >>> sdm;10.218.203.17;3339015;834753;25123 >>> sdk;10.218.204.17;3637573;909393;23061 >>> sde;10.219.128.17;3325777;831444;25223 >>> sdl;10.219.202.17;3305464;826366;25378 >>> sdg;10.219.203.17;3304032;826008;25389 >>> sdn;10.219.204.17;3330001;832500;25191 >>> sdd;10.220.128.17;4624370;1156092;18140 >>> sdi;10.220.202.17;4619277;1154819;18160 >>> sdj;10.220.203.17;4610138;1152534;18196 >>> sdh;10.220.204.17;4586445;1146611;18290 >>> >>> It seems that there is a lot of changes between the kernels. I had >>> these kernels already on the box and I can bisect them if you think it >>> would help. It is really odd that port 2 on the Connect-IB card did >>> better than port 1 on the 4.6.0 kernel. >>> ---------------- >>> Robert LeBlanc >>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >> >> >> so in these kernels you get better performance with the C-IB than CX3 ? >> we need to find the bottleneck. >> Can you increase the iodepth and/or block size to see if we can reach the >> wire speed. >> another try is to load ib_iser with always_register=N. >> >> what is the cpu utilzation in both initiator/target ? >> did you spread the irq affinity ? >> >> >>> >>> >>> On Tue, Jun 7, 2016 at 10:48 AM, Robert LeBlanc <robert@xxxxxxxxxxxxx> >>> wrote: >>>> >>>> The target is LIO (same kernel) with a 200 GB RAM disk and I'm running >>>> fio as follows: >>>> >>>> fio --rw=read --bs=4K --size=2G --numjobs=40 --name=worker.matt >>>> --group_reporting --minimal | cut -d';' -f7,8,9 >>>> >>>> All of the paths are set the same with noop and nomerges to either 1 >>>> or 2 (doesn't make a big difference). >>>> >>>> I started looking into this when the 4.6 kernel wasn't performing as >>>> well as we were able to get the 4.4 kernel to work. I went back to the >>>> 4.4 kernel and I could not replicate the 4+ million IOPs. So I started >>>> breaking down the problem to smaller pieces and found this anomaly. >>>> Since there hasn't been any suggestions up to this point, I'll check >>>> other kernel version to see if it is specific to certain kernels. If >>>> you need more information, please let me know. >>>> >>>> Thanks, >>>> ---------------- >>>> Robert LeBlanc >>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >>>> >>>> >>>> On Tue, Jun 7, 2016 at 6:02 AM, Max Gurtovoy <maxg@xxxxxxxxxxxx> wrote: >>>>> >>>>> >>>>> >>>>> On 6/7/2016 1:36 AM, Robert LeBlanc wrote: >>>>>> >>>>>> >>>>>> I'm trying to understand why our Connect-IB card is not performing as >>>>>> well as our ConnectX-3 card. There are 3 ports between the two cards >>>>>> and 12 paths to the iSER target which is a RAM disk. >>>>> >>>>> >>>>> >>>>> <snip> >>>>> >>>>>> >>>>>> When I run fio against each path individually, I get: >>>>> >>>>> >>>>> >>>>> What is the scenario (bs, numjobs, iodepth) for each run ? >>>>> Which target do you use ? backing store ? >>>>> >>>>> >>>>>> >>>>>> disk;target IP;bandwidth,IOPs,Execution time >>>>>> sdn;10.218.128.17;5053682;1263420;16599 >>>>>> sde;10.218.202.17;5032158;1258039;16670 >>>>>> sdh;10.218.203.17;4993516;1248379;16799 >>>>>> sdk;10.218.204.17;5081848;1270462;16507 >>>>>> sdc;10.219.128.17;3750942;937735;22364 >>>>>> sdf;10.219.202.17;3746921;936730;22388 >>>>>> sdi;10.219.203.17;3873929;968482;21654 >>>>>> sdl;10.219.204.17;3841465;960366;21837 >>>>>> sdd;10.220.128.17;3760358;940089;22308 >>>>>> sdg;10.220.202.17;3866252;966563;21697 >>>>>> sdj;10.220.203.17;3757495;939373;22325 >>>>>> sdm;10.220.204.17;4064051;1016012;20641 >>>>>> >> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
