Re: Network Block device (NBD) on top of glusterfs

Prasanna Kalever <pkalever@xxxxxxxxxx> · Thu, 21 Mar 2019 19:53:19 +0530

On Thu, Mar 21, 2019 at 6:31 PM Xiubo Li <xiubli@xxxxxxxxxx> wrote:

    On 2019/3/21 18:09, Prasanna Kalever
      wrote:

            On Thu, Mar 21, 2019 at
              9:00 AM Xiubo Li <xiubli@xxxxxxxxxx> wrote:

                All,
                I am one of
                  the contributor for gluster-block[1] project,
                  and also I contribute to linux kernel and open-iscsi
                  project.[2]
                NBD was
                  around for some time, but in recent time, linux
                  kernel’s Network Block Device (NBD) is enhanced and
                  made to work with more devices and also the option to
                  integrate with netlink is added. So, I tried to
                  provide a glusterfs client based NBD driver recently.
                  Please refer github issue #633[3], and
                  good news is I have a working code, with most basic
                  things @ nbd-runner project[4].
                While this
                  email is about announcing the project, and asking for
                  more collaboration, I would also like to discuss more
                  about the placement of the project itself. Currently
                  nbd-runner project is expected to be shared by our
                  friends at Ceph project too, to provide NBD driver for
                  Ceph. I have personally worked with some of them
                  closely while contributing to open-iSCSI project, and
                  we would like to take this project to great success.
                Now few
                  questions:

                  Can I continue to use http://github.com/gluster/nbd-runner as
                    home for this project, even if its shared by other
                    filesystem projects?

                  I personally am fine with this.

                  Should there be a separate organization for this
                    repo?

                  While it may make sense in future, for now, I am
                    not planning to start any new thing?

                It would be
                  great if we have some consensus on this soon as
                  nbd-runner is a new repository. If there are no
                  concerns, I will continue to contribute to the
                  existing repository.

              Thanks Xiubo Li,
                for finally sending this email out. Since this email is
                out on gluster mailing list, I would like to take a
                stand from gluster community point of view *only* and
                share my views.

              My honest answer is "If we want
                to maintain this within gluster org, then 80% of the
                effort is common/duplicate of what we did all these days
                with gluster-block",

    The great idea came from Mike Christie days ago and the
      nbd-runner project's framework is initially emulated from
      tcmu-runner. This is why I name this project as nbd-runner, which
      will work for all the other Distributed Storages, such as
      Gluster/Ceph/Azure, as discussed with Mike before. 

    nbd-runner(NBD proto) and tcmu-runner(iSCSI proto) are almost the
      same and both are working as lower IO(READ/WRITE/...) stuff, not
      the management layer like ceph-iscsi-gateway and gluster-block
      currently do.

    Currently since I only implemented the Gluster handler and also
      using the RPC like glusterfs and gluster-block, most of the other
      code (about 70%) in nbd-runner are for the NBD proto and these are
      very different from tcmu-runner/glusterfs/gluster-block projects,
      and there are many new features in NBD module that not yet
      supported and then there will be more different in future.

      The framework coding has been done and the nbd-runner project is
      already stable and could already work well for me now.

              like:
              * rpc/socket code
              * cli/daemon parser/helper
                logics
              * gfapi util functions
              * logger framework
              * inotify & dyn-config
                threads

    Yeah, these features were initially from tcmu-runner project,
      Mike and I coded two years ago. Currently nbd-runner also has
      copied them from tcmu-runner.

I don't think tcmu-runner has any of,

-> cli/daemon approach routines
-> rpc low-level clnt/svc routines
-> gfapi level file create/delete util functions  
-> Json parser support
-> socket bound/listener related functionalities
-> autoMake build frame-work, and 
-> many other maintenance files

I actually can go in detail and furnish a long list of reference made here and you cannot deny the fact, but its **all okay** to take references from other alike projects. But my intention was not to point about the copy made here, but rather saying we are just wasting our efforts rewriting, copy-pasting, maintaining and fixing the same functionality framework.

Again all I'm trying to make is, if at all you want to maintain nbd client as part of gluster.org, why not use gluster-block itself ? which is well tested and stable enough.

Apart from all the examples I have mentioned in my previous thread, there are other great advantages from user perspective as-well, like:

* The top layers such as heketi consuming gluster's block storage really don't have to care whether the backend provider is tcmu-runner or nbd-runner or qemu-tcmu or kernel loopback or fileIO or something else ...
They simply call gluster-block and get a block device out there.

* We can reuse the existing gluster-block's rest api interface too.

** Believe me, over the years I have learned it from my experience and its a very fact that, we can save lot of energy and time by reusing existing stable framework rather than building a new one from scratch **

I will try to spend few hours over my weekends and send a nbd client application PR for gluster-block (IMO this shouldn't exceed ~200 lines), will request your review there.

Cheers!
--
Prasanna

    Very appreciated for you great ideas here Prasanna and hope
      nbd-runner could be more generically and successfully used in
      future.
    BRs
    Xiubo Li

              * configure/Makefile/specfiles
              * docsAboutGluster and etc ..

              The repository gluster-block is
                actually a home for all the block related stuff within
                gluster and its designed to accommodate alike
                functionalities, if I was you I would have simply copied
                nbd-runner.c into https://github.com/gluster/gluster-block/tree/master/daemon/
                just like ceph plays it here https://github.com/ceph/ceph/blob/master/src/tools/rbd_nbd/rbd-nbd.cc
                and be done.

              Advantages of keeping nbd
                client within gluster-block:
              -> No worry about
                maintenance code burdon
              -> No worry about monitoring
                a new component
              -> shipping packages to
                fedora/centos/rhel is handled
              -> This helps improve and
                stabilize the current gluster-block framework
              -> We can build a common CI
              -> We can use reuse common
                test framework and etc ..

              If you have an impression that
                gluster-block is for management, then I would really
                want to correct you at this point.

              Some of my near future plans
                for gluster-block:
              * Allow exporting blocks with
                FUSE access via fileIO backstore to improve large-file
                workloads, draft:  https://github.com/gluster/gluster-block/pull/58
              * Accommodate kernel loopback
                handling for local only applications
              * The same way we can
                accommodate nbd app/client, and IMHO this effort
                shouldn't take 1 or 2 days to get it merged with in
                gluster-block and ready for a go release.

              Hope that clarifies it.

              Best Regards,
              --
              Prasanna

                Regards,

                  Xiubo Li (@lxbsz)
                [1] - https://github.com/gluster/gluster-block

                  [2] - https://github.com/open-iscsi

                  [3] - https://github.com/gluster/glusterfs/issues/633

                  [4] - https://github.com/gluster/nbd-runner

              _______________________________________________

              Gluster-devel mailing list

              Gluster-devel@xxxxxxxxxxx

              https://lists.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel