On 6/8/2016 1:37 AM, Robert LeBlanc wrote:
On the 4.1.15 kernel:
sdc;10.218.128.17;3971878;992969;21120
sdd;10.218.202.17;3967745;991936;21142
sdg;10.218.203.17;3938128;984532;21301
sdk;10.218.204.17;3952602;988150;21223
sdn;10.219.128.17;4615719;1153929;18174
sdf;10.219.202.17;4622331;1155582;18148
sdi;10.219.203.17;4602297;1150574;18227
sdl;10.219.204.17;4565477;1141369;18374
sde;10.220.128.17;4594986;1148746;18256
sdh;10.220.202.17;4590209;1147552;18275
sdj;10.220.203.17;4599017;1149754;18240
sdm;10.220.204.17;4610898;1152724;18193
On the 4.6.0 kernel:
sdc;10.218.128.17;3239219;809804;25897
sdf;10.218.202.17;3321300;830325;25257
sdm;10.218.203.17;3339015;834753;25123
sdk;10.218.204.17;3637573;909393;23061
sde;10.219.128.17;3325777;831444;25223
sdl;10.219.202.17;3305464;826366;25378
sdg;10.219.203.17;3304032;826008;25389
sdn;10.219.204.17;3330001;832500;25191
sdd;10.220.128.17;4624370;1156092;18140
sdi;10.220.202.17;4619277;1154819;18160
sdj;10.220.203.17;4610138;1152534;18196
sdh;10.220.204.17;4586445;1146611;18290
It seems that there is a lot of changes between the kernels. I had
these kernels already on the box and I can bisect them if you think it
would help. It is really odd that port 2 on the Connect-IB card did
better than port 1 on the 4.6.0 kernel.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
so in these kernels you get better performance with the C-IB than CX3 ?
we need to find the bottleneck.
Can you increase the iodepth and/or block size to see if we can reach
the wire speed.
another try is to load ib_iser with always_register=N.
what is the cpu utilzation in both initiator/target ?
did you spread the irq affinity ?
On Tue, Jun 7, 2016 at 10:48 AM, Robert LeBlanc <robert@xxxxxxxxxxxxx> wrote:
The target is LIO (same kernel) with a 200 GB RAM disk and I'm running
fio as follows:
fio --rw=read --bs=4K --size=2G --numjobs=40 --name=worker.matt
--group_reporting --minimal | cut -d';' -f7,8,9
All of the paths are set the same with noop and nomerges to either 1
or 2 (doesn't make a big difference).
I started looking into this when the 4.6 kernel wasn't performing as
well as we were able to get the 4.4 kernel to work. I went back to the
4.4 kernel and I could not replicate the 4+ million IOPs. So I started
breaking down the problem to smaller pieces and found this anomaly.
Since there hasn't been any suggestions up to this point, I'll check
other kernel version to see if it is specific to certain kernels. If
you need more information, please let me know.
Thanks,
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Jun 7, 2016 at 6:02 AM, Max Gurtovoy <maxg@xxxxxxxxxxxx> wrote:
On 6/7/2016 1:36 AM, Robert LeBlanc wrote:
I'm trying to understand why our Connect-IB card is not performing as
well as our ConnectX-3 card. There are 3 ports between the two cards
and 12 paths to the iSER target which is a RAM disk.
<snip>
When I run fio against each path individually, I get:
What is the scenario (bs, numjobs, iodepth) for each run ?
Which target do you use ? backing store ?
disk;target IP;bandwidth,IOPs,Execution time
sdn;10.218.128.17;5053682;1263420;16599
sde;10.218.202.17;5032158;1258039;16670
sdh;10.218.203.17;4993516;1248379;16799
sdk;10.218.204.17;5081848;1270462;16507
sdc;10.219.128.17;3750942;937735;22364
sdf;10.219.202.17;3746921;936730;22388
sdi;10.219.203.17;3873929;968482;21654
sdl;10.219.204.17;3841465;960366;21837
sdd;10.220.128.17;3760358;940089;22308
sdg;10.220.202.17;3866252;966563;21697
sdj;10.220.203.17;3757495;939373;22325
sdm;10.220.204.17;4064051;1016012;20641
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
