On Wed, Mar 13, 2019 at 04:17:22PM +0000, Bernard Metzler wrote: > -----"Dennis Dalessandro" <dennis.dalessandro@xxxxxxxxx> wrote: ----- > > >To: "Leon Romanovsky" <leonro@xxxxxxxxxxxx> > >From: "Dennis Dalessandro" <dennis.dalessandro@xxxxxxxxx> > >Date: 03/13/2019 04:31PM > >Cc: "Parav Pandit" <parav@xxxxxxxxxxxx>, "Chuck Lever" > ><chuck.lever@xxxxxxxxxx>, "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>, > >"Steve Wise" <larrystevenwise@xxxxxxxxx>, > >"linux-rdma@xxxxxxxxxxxxxxx" <linux-rdma@xxxxxxxxxxxxxxx>, "Yuval > >Shaia" <yuval.shaia@xxxxxxxxxx> > >Subject: Re: [PATCH v5 00/13] SIW: Request for Comments > > > >On 3/12/2019 12:42 PM, Leon Romanovsky wrote: > >> On Tue, Mar 12, 2019 at 10:39:04AM -0400, Dennis Dalessandro wrote: > >>> On 3/10/2019 7:44 PM, Parav Pandit wrote: > >>>> Hi Chuck, > >>>> > >>>>> -----Original Message----- > >>>>> From: Chuck Lever <chuck.lever@xxxxxxxxxx> > >>>>> Sent: Sunday, March 10, 2019 5:30 PM > >>>>> To: Parav Pandit <parav@xxxxxxxxxxxx> > >>>>> Cc: Bernard Metzler <BMT@xxxxxxxxxxxxxx>; Steve Wise > >>>>> <larrystevenwise@xxxxxxxxx>; linux-rdma@xxxxxxxxxxxxxxx; Leon > >>>>> Romanovsky <leonro@xxxxxxxxxxxx>; Yuval Shaia > >>>>> <yuval.shaia@xxxxxxxxxx> > >>>>> Subject: Re: [PATCH v5 00/13] SIW: Request for Comments > >>>>> > >>>>> Hey Parav- > >>>>> > >>>>>> On Mar 10, 2019, at 6:01 PM, Parav Pandit <parav@xxxxxxxxxxxx> > >wrote: > >>>>>> > >>>>>> Hi Bernard, > >>>>>> > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Bernard Metzler <BMT@xxxxxxxxxxxxxx> > >>>>>>> Sent: Sunday, March 10, 2019 4:41 PM > >>>>>>> To: Parav Pandit <parav@xxxxxxxxxxxx> > >>>>>>> Cc: Steve Wise <larrystevenwise@xxxxxxxxx>; > >>>>>>> linux-rdma@xxxxxxxxxxxxxxx; Leon Romanovsky > ><leonro@xxxxxxxxxxxx>; > >>>>>>> Yuval Shaia <yuval.shaia@xxxxxxxxxx> > >>>>>>> Subject: RE: [PATCH v5 00/13] SIW: Request for Comments > >>>>>>> > >>>>>>> Hi Parav, > >>>>>>> > >>>>>>> I understand your point of having some common driver > >infrastructure > >>>>>>> between rxe and siw. I remember I was even proposing rxe folks > >>>>>>> looking into that when rxe emerged after siw was already > >prototyped > >>>>>>> and open sourced (yes, siw was first out there). There was not > >much > >>>>>>> interest from rxe folks at that point in time, which is OK and > >>>>>>> understood (always cool to write something new and better from > >scratch, > >>>>> who wants iWarp, etc.). > >>>>>>> > >>>>>> I didn't follow that part of the history, but it clearly proves > >the point that it > >>>>> was started incorrectly. > >>>>>> With all the fancy code for doorbells, with mmap() call for QP, > >CQ, ground > >>>>> reality is, it doesn't even pass a developer's silly ib_write_bw > >today. > >>>>>> And at that test shows 2Gbps bw on 64Gb RAM with 48 core cpus! > >>>>>> > >>>>>>> At the current stage of the project, it is rather > >contra-productive > >>>>>>> to completely refactor siw to match with rxe, since I do not > >have the > >>>>>>> resources to rip apart the whole thing, and I am not yet > >convinced it > >>>>> makes sense. > >>>>>>> > >>>>>> But that seems to be right approach to have modular code and > >reuse > >>>>> existing pieces. > >>>>>> Leon, Yuval and others resonate that too. > >>>>> > >>>>> I agree that code sharing can be valuable. However... > >>>>> > >>>>> > >>>>>> If existing pieces are inferior or mis fit, their technical > >details has to be > >>>>> called out. > >>>>>> rdmavt improved and used library model to share code between > >two > >>>>> drivers. > >>>>>> > >>>>>>> If one asks for merging these days, one asks at first for a > >delay in > >>>>>>> getting siw accepted. > >>>>>>> > >>>>>> Yuval and Leon wants to improve rxe in general and have desire > >to reuse > >>>>> rxe. > >>>>>> They might be able to help you. Please check with them. > >>>>>> > >>>>>>> Over time, it might indeed be a good thing to merge parts. > >>>>>> It should happen from beginning. > >>>>> > >>>>> I disagree that merging siw should wait for code de-duplication > >with rxe. > >>>>> > >>>>> The current siw is coming close to being merge-ready. I would > >like to see it in > >>>>> the kernel as soon as possible. RoCE, hard or soft, requires DCE > >in order to > >>>>> be fully useful, and thus it will never be possible to have a > >software RoCE > >>>>> driver that can operate via any Ethernet driver on any network. > >That is not > >>>>> the case with iWARP, which relies on TCP's built-in congestion > >avoidance, > >>>>> which is truly routable and available everywhere. > >>>> > >>>> Soft roce driver faces soft cpu lockup in single system at 2Gbps > >bandwidth with just two QPs (send and receive). > >>>> :-) > >>>> Going across a system is brave world. > >>>> > >>>> But Leon and Yuval likes to see it enhanced/used in general > >instead of new driver. > >>>> And therefore, it extends to siw efforts too. > >>>> > >>>>> > >>>>> The ULPs need to have a robust generic in-tree software driver > >as soon as we > >>>>> can get it to enable development, testing, and deployment in > >environments > >>>>> that have no physical RDMA device. siw is the fastest way to get > >that. > >>>>> > >>>>> > >>>> I totally agree to your point. > >>>> In case if you didn't notice, a loopback driver [1] that > >outperforms soft_roce by 25x in performance and it is more stable. > >:-) > >>>> > >>>>>>> As said, I'd still like to investigate that. > >>>>>> That will be very helpful. > >>>>>> > >>>>>>> But let the sometimes intentionally different concepts of > >these two > >>>>>>> drivers co-exist for some time out there, and let's draw > >conclusions > >>>>>>> of what to make common later. > >>>>>>> > >>>>>> It won't happen later once it is in tree. :-) > >>>>> > >>>>> There is nothing stopping code de-duplication once siw is in the > >kernel. > >>>>> > >>>>> The most obvious counter-example of your claim is the in-kernel > >ULPs which > >>>>> have adopted new core APIs such as ib_alloc_cq long after the > >ULPs were > >>>>> accepted and merged. > >>>>> > >>>> It happened because of active maintainers like you, Sagi and many > >others. > >>>> > >>>>> > >>>>>>> And let's be clear, lots of things between RoCE and iWarp are > >very > >>>>> different. > >>>>>> All rest of the core components such as ib_core, addr, rdmacm > >are able to > >>>>> handle it two different transports. > >>>>>> Cavium or qedr driver is dual protocol driver that support RoCE > >and iWarp > >>>>> both in single driver. > >>>>>> > >>>>>>> Not only wire protocol and lower interfaces, but also > >semantically. > >>>>>>> These things will stay different by specification, and an > >efficient > >>>>>>> implementation of those things will likely still go it's own > >ways. > >>>>>>> > >>>>>> We would like to understand technically why rxe is mis fit for > >it, and why it > >>>>> cannot be improved. > >>>>>> Lack of resources is probably not good reason. > >>>>> > >>>>> Actually I find lack of resources to be a compelling argument. > >Before siw is > >>>>> merged, Bernard and his team are the only ones who can do this > >work, and > >>>>> they are not yet familiar with the rxe code base. > >>>>> > >>>> I want Leon to explicitly ACK this point and want to listen to > >him what he has to say. > >>>> Leon? > >>>> Is this compelling reason for siw? > >>>> And therefore for loopback driver too? > >>>> > >>>>> After merging, anyone in the Linux RDMA community can help with > >it, > >>>>> including you, the person who is asking for this extra effort. > >>>>> > >>>> I asked because Leon is asking that for another simplified driver > >proposal. > >>>> Most ULPs maintainers and users would agree that rxe driver is > >not able to satisfy their needs > >>>> (stability, performance). > >>>> And your feedback is obviously important here. > >>>> > >>>>> It is reasonable to request that common code be refactored. It > >is neither fair > >>>>> nor productive to gate merging siw on that work. > >>>> > >>>> I do not intent to gate merging siw work by any means. > >>>> I want community to reach consensus on - why should we merge a > >new driver even though technically its feasible to share a code with > >existing in-tree driver. > >>>> > >>>> And therefore, that same exact reason won't be limited to just > >siw driver. > >>>> > >>>>> Why force Bernard to bear the burden of fixing rxe? > >>>> > >>>> As per Leon, rxe seems stable enough that doesn't need fixing. > >>>> I infer that from his suggestion to enhance rxe instead of a new > >loopback driver for roce and IB. > >>>> So Bernard won't need to fix it based on Leon's and Yuval's > >comments. > >>>> > >>>> And if rxe is not good enough, it speaks for itself. > >>>> Leon? Yuval? > >>>> > >>>> If Leon and Yuval didn't ask to enhance soft_roce driver, my > >proposal is: > >>>> > >>>> 1. siw to provide a transport agnostic user verbs like [2] > >>>> And improve performance incrementally with core extensions. > >>>> This will be similar to Jen's io_uring work to provide vendor > >independent io cmd and cmpl queues for block ios. > >>>> Currently rxe and siw creates their own rings etc. > >>>> > >>>> 2. loopback driver for RoCE and IB to reuse [2]. > >>>> 3. soft_roce maintainer (?) to decide if they want to reuse code > >from siw or loopback > >>>> > >>>> [1] > >https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.kernel > >.org_patch_10831261_&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8Z > >O1PP1alNwU_QJcRRLfmYTAgd3QCvqSc&m=_08EePmMUq466MpLUdKBkqpiAQwZO6-v3vQ > >V8lJGy8Q&s=PNoLlcMS2y11ULloi6Tk8J1q2ccevH1tuvXbZ_gA16Y&e= > >>>> [2] > >https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.kernel > >.org_patch_10831251_&d=DwIC-g&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8Z > >O1PP1alNwU_QJcRRLfmYTAgd3QCvqSc&m=_08EePmMUq466MpLUdKBkqpiAQwZO6-v3vQ > >V8lJGy8Q&s=5q96F0D8jXv81QEhEfvz90wh6CxdS6TFbqOSz8YzPnw&e= > >>>> > >>>>> > >>>>> Let's get siw merged, and then _everyone_ can have a say where > >it is > >>>>> valuable to share code. > >>>>> > >>>> Let's apply that to other driver proposed too? > >>> > >>> My 2 cents... > >>> > >>> If things compile and move, why not allow it into the tree. > >> > >> Because upstream != trash can. We want to see involvement, > >commitment > >> and in some extent development roadmap. > > > >First off I never suggested that, not in the slightest. Secondly it's > > > >not fair to label either siw or loopback drivers as trash. They both > >are > >cleanly (arguably) done, they both function. > > > >Can they run NVMEoF? If so then according to Jason [1] that is > >meeting > >the unambiguous low bar. To me that means they are supporting enough > >to > >be considered a serious driver and brought into the sub system for > >further development. > > > > I speak for myself and for siw development only here. I do this for > years as a side project. We used that driver a lot in the past for > RDMA application development, it is used in classes at ETH Zurich > university, there are even some companies out there using it for their > business. > > I am committed to further development and maintenance of software > based RDMA. I know sometimes it takes to long before I response/fix. > Like these days where I am completely consumed by my daily business. > It is not really laziness, but constant lack of time to pet my > pet project... One reason I am very much interested in getting it > upstream is, to better align it with a moving target: the RDMA > mid-layer. I spent lots of time with fixing the stand alone siw open > source project for all that changes over the years. I want to be > better integrated with the RDMA development community. > Yes, and I hope for a code quality improvement as well. Based on community > feedback, patches. I think this humble piece of code is not to bad to > get improved by folks who want the functionality it provides. Guys, I didn't mean anything bad about siw or loopback. Just responded on the Dennis's sentence "If things compile and move, why not allow it into the tree." I'm pretty convinced that SIW will be part of next kernel release. Sorry if I was too rude. Thanks > > Thanks > Bernard. >
Attachment:
signature.asc
Description: PGP signature
