----- Original Message ----- > From: "Laurence Oberman" <loberman@xxxxxxxxxx> > To: "Bart Van Assche" <bart.vanassche@xxxxxxxxxxx> > Cc: leon@xxxxxxxxxx, "Doug Ledford" <dledford@xxxxxxxxxx>, "Christoph Hellwig" <hch@xxxxxx>, "Sagi Grimberg" > <sagi@xxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "Or Gerlitz" <ogerlitz@xxxxxxxxxxxx> > Sent: Thursday, May 12, 2016 12:28:34 PM > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg() > > > > ----- Original Message ----- > > From: "Bart Van Assche" <bart.vanassche@xxxxxxxxxxx> > > To: "Laurence Oberman" <loberman@xxxxxxxxxx> > > Cc: leon@xxxxxxxxxx, "Doug Ledford" <dledford@xxxxxxxxxx>, "Christoph > > Hellwig" <hch@xxxxxx>, "Sagi Grimberg" > > <sagi@xxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "Or Gerlitz" > > <ogerlitz@xxxxxxxxxxxx> > > Sent: Wednesday, May 11, 2016 11:41:39 AM > > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg() > > > > On 05/11/2016 08:31 AM, Laurence Oberman wrote: > > > I chased that for a while too.:) > > > Landed up pulling the latest next, applying all of Christoph's 11 RDMA > > > patches, then the first 11 of Barts and the latest 6. > > > I had to hand fix some stuff. > > > Kernel is building now for testing :) > > > > Hello Laurence, > > > > Please wait with starting your tests until I have made a kernel tree > > with this patch series available. > > > > Thanks, > > > > Bart. > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > For Barts latest set of patches see subject, using Barts's tree the mapping > failures are gone and its run for over 24 hours stable. > This is with multiple parallel reads of 4MB issued direct and multiple > parallel writes to the same mpath devices issued buffered. > The only variation from what I am used to seeing (when its not failing :) on > prior ib_srp) is that the I/O sizes reach 4MB often but are also often > smaller. > Some of this could be issues with my LIO target and next week will have an > enterprise array directly connected I will be testing with. > > Its a huge improvement and seems good to me as I know these failures very > well. > > Tested-by: Laurence Oberman <loberman@xxxxxxxxxx> > > Example with direct reads and buffered writes > > mpath view > > ### RECORD 556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 > 2016) ### > # DISK STATISTICS (/sec) > # > <---------reads---------><---------writes---------><--------averages--------> > Pct > #Time Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize > QLen Wait SvcTim Util > 12:22:06 dm-6 212992 128 247 862 130200 64 126 1033 920 > 7 20 2 99 > 12:22:06 dm-7 213180 52 885 241 118784 29 406 293 257 > 25 20 0 99 > 12:22:06 dm-8 217088 53 424 512 122880 60 270 455 489 > 14 21 1 99 > 12:22:06 dm-9 211968 52 465 456 119760 30 409 293 379 > 19 22 1 99 > 12:22:06 dm-10 212992 52 364 585 121340 60 146 831 655 > 10 20 1 99 > 12:22:06 dm-11 221184 162 54 4096 135168 33 297 455 1015 > 9 26 2 99 *** Reads reach 4MB here, often smaller though > 12:22:06 dm-12 229376 168 280 819 126976 93 62 2048 1041 > 6 17 2 99 > 12:22:06 dm-13 229376 168 112 2048 131072 64 128 1024 1501 > 5 23 4 99 > 12:22:06 dm-14 225280 110 385 585 122880 29 185 664 610 > 11 20 1 99 > 12:22:06 dm-15 203912 50 549 371 118792 29 379 313 347 > 20 21 1 99 > > individual path view > > ### RECORD 556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 > 2016) ### > # DISK STATISTICS (/sec) > # > <---------reads---------><---------writes---------><--------averages--------> > Pct > #Time Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize > QLen Wait SvcTim Util > 12:26:37 sdc 128092 0 348 368 44468 0 482 92 207 > 18 21 1 95 > 12:26:37 sdd 103476 0 461 224 62956 0 266 237 228 > 14 20 1 97 > 12:26:37 sde 118100 0 145 814 65488 0 127 516 674 > 5 19 3 97 > 12:26:37 sdf 121660 0 651 187 53960 0 277 195 189 > 17 18 1 97 > 12:26:37 sdg 87276 0 305 286 76020 0 191 398 329 > 10 20 1 97 > 12:26:37 sdh 98688 0 166 595 67400 0 554 122 230 > 17 23 1 95 > 12:26:37 sdi 197112 0 945 209 16208 0 81 200 207 > 16 15 0 99 > 12:26:37 sdj 1776 0 16 111 143360 0 35 4096 2845 > 1 24 18 95 > 12:26:37 sdk 0 0 0 0 139264 0 374 372 372 > 10 26 2 96 > 12:26:37 sdl 77600 0 572 136 76624 0 268 286 183 > 17 19 1 93 > 12:26:37 sdm 80804 0 264 306 74316 0 533 139 194 > 19 24 1 98 > 12:26:37 sdn 101324 0 439 231 64020 0 385 166 200 > 18 22 1 98 > 12:26:37 sdo 107180 0 130 824 73776 0 145 509 658 > 5 21 3 97 > 12:26:37 sdp 80788 0 389 208 69376 0 339 205 206 > 17 22 1 99 > 12:26:37 sdq 116704 0 390 299 55052 0 129 427 330 > 10 20 1 99 > 12:26:37 sdr 113280 0 249 455 53816 0 401 134 257 > 16 24 1 99 > 12:26:37 sds 13192 0 93 142 110768 0 291 381 322 > 10 26 2 95 > 12:26:37 sdt 165184 0 1197 138 0 0 0 0 137 > 23 18 0 99 > 12:26:37 sdu 241664 0 59 4096 0 0 0 0 4096 > 1 16 16 97 > Bart, I meant to mention that that was with ib_srp untuned. My next set of tests will be with indirect_sg_entries=512 and cmd_sg_entries=64 for a start. Then I will max them out and see how we do. Thanks!!! -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
