W dniu 13.12.2018 o 22:55, Sagi Grimberg pisze:
What device are you using?
Its just that I have a patch set I've been meaning to send out soon that
get rid of fmrs altogether? Is it ConnectX-3? or an older generation?
It's ConnectX-3 HP InfiniBand FDR/Ethernet 10/40Gb 2-port 544FLR-QSFP
Adapter, but as I've said before - this machine is running XenServer 7.5
and I can't modify it's kernel too far (ABI can't be broken).
Anyways, regardless this is a generic problem that is not private to
iSER I think. iSER allocates a finite set of resources per session, each
of them has HW resources and memory associated with it.
Now given that the amount of memory varies between device providers its
rather impossible to calculate it in a deterministic way.
Its clear that iSER cannot establish an infinite number of sessions.. Do
you have a target number of sessions you want to reach?
Currently it would be up to 530 sessions, in future it might go even
further (up to ~650-700).
Perhaps you can lower your queue depth? that would introduce nice memory
savings. The default is:
node.session.cmds_max = 128
I assume you don't need 300 sessions with queue depth of 128... Perhaps
you can settle with 64 or 32 instead?
Yes, I'm currently evaluating with cmds_max set to 64.
Is there any way of knowing max FMR number for a card that shows "0" in
ibv_devinfo -v (or allocated and possible to allocate) or is it only a
driver/OFED issue?
Regards,
Michał
