Hi Danil and Jack,
Hallo Doug, Hallo Jason, Hallo Jens, Hallo Greg, Could you please provide some feedback to the IBNBD driver and the IBTRS library? So far we addressed all the requests provided by the community
That is not exactly correct AFAIR, My main issues which were raised before are: - IMO there isn't any justification to this ibtrs layering separation given that the only user of this is your ibnbd. Unless you are trying to submit another consumer, you should avoid adding another subsystem that is not really general purpose. - ibtrs in general is using almost no infrastructure from the existing kernel subsystems. Examples are: - tag allocation mechanism (which I'm not clear why its needed) - rdma rw abstraction similar to what we have in the core - list_next_or_null_rr_rcu ?? - few other examples sprinkled around.. Another question, from what I understand from the code, the client always rdma_writes data on writes (with imm) from a remote pool of server buffers dedicated to it. Essentially all writes are immediate (no rdma reads ever). How is that different than using send wrs to a set of pre-posted recv buffers (like all others are doing)? Is it faster? Also, given that the server pre-allocate a substantial amount of memory for each connection, is it documented the requirements from the server side? Usually kernel implementations (especially upstream ones) will avoid imposing such large longstanding memory requirements on the system by default. I don't have a firm stand on this, but wanted to highlight this as you are sending this for upstream inclusion. and
continue to maintain our code up-to-date with the upstream kernel while having an extra compatibility layer for older kernels in our out-of-tree repository.
Overall, while I absolutely support your cause to lower your maintenance overhead by having this sit upstream, I don't see why this can be helpful to anyone else in the rdma community. If instead you can crystallize why/how ibnbd is faster than anything else, and perhaps contribute a common infrastructure piece (or enhance an existing one) such that other existing ulps can leverage, it will be a lot more compelling to include it upstream.
I understand that SRP and NVMEoF which are in the kernel already do provide equivalent functionality for the majority of the use cases. IBNBD on the other hand is showing higher performance and more importantly includes the IBTRS - a general purpose library to establish connections and transport BIO-like read/write sg-lists over RDMA,
But who needs it? Can other ulps use it or pieces of it? I keep failing to understand why is this a benefit if its specific to your ibnbd?
