----- Original Message ----- > From: "Laurence Oberman" <loberman@xxxxxxxxxx> > To: "Bart Van Assche" <bart.vanassche@xxxxxxxxxxx> > Cc: leon@xxxxxxxxxx, "Doug Ledford" <dledford@xxxxxxxxxx>, "Christoph Hellwig" <hch@xxxxxx>, "Sagi Grimberg" > <sagi@xxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "Or Gerlitz" <ogerlitz@xxxxxxxxxxxx> > Sent: Thursday, May 12, 2016 12:38:13 PM > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg() > > > > ----- Original Message ----- > > From: "Laurence Oberman" <loberman@xxxxxxxxxx> > > To: "Bart Van Assche" <bart.vanassche@xxxxxxxxxxx> > > Cc: leon@xxxxxxxxxx, "Doug Ledford" <dledford@xxxxxxxxxx>, "Christoph > > Hellwig" <hch@xxxxxx>, "Sagi Grimberg" > > <sagi@xxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "Or Gerlitz" > > <ogerlitz@xxxxxxxxxxxx> > > Sent: Thursday, May 12, 2016 12:28:34 PM > > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg() > > > > > > > > ----- Original Message ----- > > > From: "Bart Van Assche" <bart.vanassche@xxxxxxxxxxx> > > > To: "Laurence Oberman" <loberman@xxxxxxxxxx> > > > Cc: leon@xxxxxxxxxx, "Doug Ledford" <dledford@xxxxxxxxxx>, "Christoph > > > Hellwig" <hch@xxxxxx>, "Sagi Grimberg" > > > <sagi@xxxxxxxxxxx>, linux-rdma@xxxxxxxxxxxxxxx, "Or Gerlitz" > > > <ogerlitz@xxxxxxxxxxxx> > > > Sent: Wednesday, May 11, 2016 11:41:39 AM > > > Subject: Re: [PATCH 4/6] IB/core: Enhance ib_map_mr_sg() > > > > > > On 05/11/2016 08:31 AM, Laurence Oberman wrote: > > > > I chased that for a while too.:) > > > > Landed up pulling the latest next, applying all of Christoph's 11 RDMA > > > > patches, then the first 11 of Barts and the latest 6. > > > > I had to hand fix some stuff. > > > > Kernel is building now for testing :) > > > > > > Hello Laurence, > > > > > > Please wait with starting your tests until I have made a kernel tree > > > with this patch series available. > > > > > > Thanks, > > > > > > Bart. > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > For Barts latest set of patches see subject, using Barts's tree the mapping > > failures are gone and its run for over 24 hours stable. > > This is with multiple parallel reads of 4MB issued direct and multiple > > parallel writes to the same mpath devices issued buffered. > > The only variation from what I am used to seeing (when its not failing :) > > on > > prior ib_srp) is that the I/O sizes reach 4MB often but are also often > > smaller. > > Some of this could be issues with my LIO target and next week will have an > > enterprise array directly connected I will be testing with. > > > > Its a huge improvement and seems good to me as I know these failures very > > well. > > > > Tested-by: Laurence Oberman <loberman@xxxxxxxxxx> > > > > Example with direct reads and buffered writes > > > > mpath view > > > > ### RECORD 556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 > > 2016) ### > > # DISK STATISTICS (/sec) > > # > > <---------reads---------><---------writes---------><--------averages--------> > > Pct > > #Time Name KBytes Merged IOs Size KBytes Merged IOs Size > > RWSize > > QLen Wait SvcTim Util > > 12:22:06 dm-6 212992 128 247 862 130200 64 126 1033 > > 920 > > 7 20 2 99 > > 12:22:06 dm-7 213180 52 885 241 118784 29 406 293 > > 257 > > 25 20 0 99 > > 12:22:06 dm-8 217088 53 424 512 122880 60 270 455 > > 489 > > 14 21 1 99 > > 12:22:06 dm-9 211968 52 465 456 119760 30 409 293 > > 379 > > 19 22 1 99 > > 12:22:06 dm-10 212992 52 364 585 121340 60 146 831 > > 655 > > 10 20 1 99 > > 12:22:06 dm-11 221184 162 54 4096 135168 33 297 455 > > 1015 > > 9 26 2 99 *** Reads reach 4MB here, often smaller though > > 12:22:06 dm-12 229376 168 280 819 126976 93 62 2048 > > 1041 > > 6 17 2 99 > > 12:22:06 dm-13 229376 168 112 2048 131072 64 128 1024 > > 1501 > > 5 23 4 99 > > 12:22:06 dm-14 225280 110 385 585 122880 29 185 664 > > 610 > > 11 20 1 99 > > 12:22:06 dm-15 203912 50 549 371 118792 29 379 313 > > 347 > > 20 21 1 99 > > > > individual path view > > > > ### RECORD 556 >>> jumpclient <<< (1463070126.001) (Thu May 12 12:22:06 > > 2016) ### > > # DISK STATISTICS (/sec) > > # > > <---------reads---------><---------writes---------><--------averages--------> > > Pct > > #Time Name KBytes Merged IOs Size KBytes Merged IOs Size > > RWSize > > QLen Wait SvcTim Util > > 12:26:37 sdc 128092 0 348 368 44468 0 482 92 > > 207 > > 18 21 1 95 > > 12:26:37 sdd 103476 0 461 224 62956 0 266 237 > > 228 > > 14 20 1 97 > > 12:26:37 sde 118100 0 145 814 65488 0 127 516 > > 674 > > 5 19 3 97 > > 12:26:37 sdf 121660 0 651 187 53960 0 277 195 > > 189 > > 17 18 1 97 > > 12:26:37 sdg 87276 0 305 286 76020 0 191 398 > > 329 > > 10 20 1 97 > > 12:26:37 sdh 98688 0 166 595 67400 0 554 122 > > 230 > > 17 23 1 95 > > 12:26:37 sdi 197112 0 945 209 16208 0 81 200 > > 207 > > 16 15 0 99 > > 12:26:37 sdj 1776 0 16 111 143360 0 35 4096 > > 2845 > > 1 24 18 95 > > 12:26:37 sdk 0 0 0 0 139264 0 374 372 > > 372 > > 10 26 2 96 > > 12:26:37 sdl 77600 0 572 136 76624 0 268 286 > > 183 > > 17 19 1 93 > > 12:26:37 sdm 80804 0 264 306 74316 0 533 139 > > 194 > > 19 24 1 98 > > 12:26:37 sdn 101324 0 439 231 64020 0 385 166 > > 200 > > 18 22 1 98 > > 12:26:37 sdo 107180 0 130 824 73776 0 145 509 > > 658 > > 5 21 3 97 > > 12:26:37 sdp 80788 0 389 208 69376 0 339 205 > > 206 > > 17 22 1 99 > > 12:26:37 sdq 116704 0 390 299 55052 0 129 427 > > 330 > > 10 20 1 99 > > 12:26:37 sdr 113280 0 249 455 53816 0 401 134 > > 257 > > 16 24 1 99 > > 12:26:37 sds 13192 0 93 142 110768 0 291 381 > > 322 > > 10 26 2 95 > > 12:26:37 sdt 165184 0 1197 138 0 0 0 0 > > 137 > > 23 18 0 99 > > 12:26:37 sdu 241664 0 59 4096 0 0 0 0 > > 4096 > > 1 16 16 97 > > > > Bart, > I meant to mention that that was with ib_srp untuned. > My next set of tests will be with indirect_sg_entries=512 and > cmd_sg_entries=64 for a start. > Then I will max them out and see how we do. > > Thanks!!! > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > Replying to my own message Tuning ib_srp and I am back to full 4MB as expected so now we should be all set. Bart, and all, thanks for all the assistance with this. Awesome work Bart on your part as always. ### RECORD 3 >>> jumpclient <<< (1463071707.001) (Thu May 12 12:48:27 2016) ### # DISK STATISTICS (/sec) # <---------reads---------><---------writes---------><--------averages--------> Pct #Time Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize QLen Wait SvcTim Util 12:48:27 dm-6 0 0 0 0 286720 284 70 4096 4096 1 12 12 89 12:48:27 dm-7 0 0 0 0 290816 284 71 4096 4096 1 12 12 89 12:48:27 dm-8 0 0 0 0 286720 280 70 4096 4096 1 12 12 89 12:48:27 dm-9 0 0 0 0 294912 288 72 4096 4096 1 13 12 93 12:48:27 dm-10 0 0 0 0 290816 284 71 4096 4096 1 13 13 93 12:48:27 dm-11 0 0 0 0 286720 284 70 4096 4096 1 12 12 88 12:48:27 dm-12 0 0 0 0 290816 284 71 4096 4096 1 12 12 89 12:48:27 dm-13 0 0 0 0 290816 288 71 4096 4096 1 12 12 90 12:48:27 dm-14 0 0 0 0 286720 280 70 4096 4096 1 12 12 89 12:48:27 dm-15 0 0 0 0 282624 280 69 4096 4096 1 12 12 88 12:48:27 sdm 0 0 0 0 143360 0 35 4096 4096 1 12 12 42 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
