Awesome, Thanks guys. - Luis ----- Original Message ----- From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> To: "Niels de Vos" <ndevos@xxxxxxxxxx> Cc: "Luis Pabón" <lpabon@xxxxxxxxxx>, "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, "Ramakrishna Yekulla" <rreddy@xxxxxxxxxx>, "Humble Chirammal" <hchiramm@xxxxxxxxxx> Sent: Tuesday, September 20, 2016 5:53:30 AM Subject: Re: [Heketi] Block store related API design discussion On Mon, Sep 19, 2016 at 9:22 PM, Niels de Vos <ndevos@xxxxxxxxxx> wrote: > On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote: > > Using qemu is interesting, but the I/O should be using the IO path of > QEMU block API. If not, > > TCMU would not know how to work with QEMU dynamic QCOW2 files. > > > > Now, if TCMU already has this, then that would be great! > > It has a qcow2 header, maybe you guys are lucky! > https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h Sent the earlier mail before seeing this mail :-). So yes, what we discussed is to see if this qemu in tcmu can internally use gfapi for doing the operations or not is something we are trying to find out. > > > Niels > > > > > - Luis > > > > ----- Original Message ----- > > From: "Prasanna Kalever" <pkalever@xxxxxxxxxx> > > To: "Niels de Vos" <ndevos@xxxxxxxxxx> > > Cc: "Luis Pabón" <lpabon@xxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, > "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Ramakrishna Yekulla" < > rreddy@xxxxxxxxxx>, "Humble Chirammal" <hchiramm@xxxxxxxxxx> > > Sent: Monday, September 19, 2016 7:13:36 AM > > Subject: Re: [Heketi] Block store related API design > discussion > > > > On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos <ndevos@xxxxxxxxxx> wrote: > > > > > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos <ndevos@xxxxxxxxxx> > wrote: > > > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > > > >> Very good points. Thanks Prasanna for putting this together. I > agree with > > > > >> your comments in that Heketi is the high level abstraction API > and it should have > > > > >> an API similar of what is described by Prasanna. > > > > >> > > > > >> I definitely do not think any File Api should be available in > Heketi, > > > > >> because that is an implementation of the Block API. The Heketi > API should > > > > >> be similar to something like OpenStack Cinder. > > > > >> > > > > >> I think that the actual management of the Volumes used for Block > storage > > > > >> and the files in them should be all managed by Heketi. How they > are > > > > >> actually created is still to be determined, but we could have > Heketi > > > > >> create them, or have helper programs do that. > > > > > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand > the > > > > > format (at the very least 'raw'), you could get functionality like > > > > > snapshots pretty simple. > > > > > > > > Niels, > > > > > > > > This is brilliant and subset of the Idea falls in one among my > > > > thoughts, only concern is about building dependencies of qemu with > > > > Heketi. > > > > But at an advantage of easy and cool snapshots solution. > > > > > > And well tested as I understand that oVirt is moving to use qemu-img as > > > well. Other tools are able to use the qcow2 format, maybe the iscsi > > > servce that gets used does so too. > > > > > > Has there already been a decision on what Heketi will configure as > iscsi > > > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. > > > > Niels, > > > > yes we will be using TCMU (Kernel Module) and TCMU-runner (user space > > service) to expose file in Gluster volume as an iSCSI target. > > more at [1], [2] & [3] > > > > [1] https://pkalever.wordpress.com/2016/06/23/gluster- > solution-for-non-shared-persistent-storage-in-docker-container/ > > [2] https://pkalever.wordpress.com/2016/06/29/non-shared- > persistent-gluster-storage-with-kubernetes/ > > [3] https://pkalever.wordpress.com/2016/08/16/read-write- > once-persistent-storage-for-openshift-origin-using-gluster/ > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > 1. http://stgt.sourceforge.net/ > > > 2. https://github.com/open-iscsi/tcmu-runner > > > http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > > > > > > > -- > > > > Prasanna > > > > > > > > > > > > > > Niels > > > > > > > > > > > > > > >> We also need to document the exact workflow to enable a file in > > > > >> a Gluster volume to be exposed as a block device. This will help > > > > >> determine where the creation of the file could take place. > > > > >> > > > > >> We can capture our decisions from these discussions in the > > > > >> following page: > > > > >> > > > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > > > > >> > > > > >> - Luis > > > > >> > > > > >> > > > > >> ----- Original Message ----- > > > > >> From: "Humble Chirammal" <hchiramm@xxxxxxxxxx> > > > > >> To: "Raghavendra Talur" <rtalur@xxxxxxxxxx> > > > > >> Cc: "Prasanna Kalever" <pkalever@xxxxxxxxxx>, "gluster-devel" < > gluster-devel@xxxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, "Luis > Pabon" <lpabon@xxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>, > "Ramakrishna Yekulla" <rreddy@xxxxxxxxxx>, "Mohamed Ashiq Liyazudeen" < > mliyazud@xxxxxxxxxx> > > > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM > > > > >> Subject: Re: [Heketi] Block store related API > design discussion > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> ----- Original Message ----- > > > > >> | From: "Raghavendra Talur" <rtalur@xxxxxxxxxx> > > > > >> | To: "Prasanna Kalever" <pkalever@xxxxxxxxxx> > > > > >> | Cc: "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Stephen > Watt" <swatt@xxxxxxxxxx>, "Luis Pabon" <lpabon@xxxxxxxxxx>, > > > > >> | "Michael Adam" <madam@xxxxxxxxxx>, "Humble Chirammal" < > hchiramm@xxxxxxxxxx>, "Ramakrishna Yekulla" > > > > >> | <rreddy@xxxxxxxxxx>, "Mohamed Ashiq Liyazudeen" < > mliyazud@xxxxxxxxxx> > > > > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM > > > > >> | Subject: Re: [Heketi] Block store related API > design discussion > > > > >> | > > > > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever < > pkalever@xxxxxxxxxx> > > > > >> | wrote: > > > > >> | > > > > >> | > Hi all, > > > > >> | > > > > > >> | > This mail is open for discussion on gluster block store > integration with > > > > >> | > heketi and its REST API interface design constraints. > > > > >> | > > > > > >> | > > > > > >> | > ___ Volume Request ... > > > > >> | > | > > > > >> | > | > > > > >> | > PVC claim -> Heketi --->| > > > > >> | > | > > > > >> | > | > > > > >> | > | > > > > >> | > | > > > > >> | > | __ > BlockCreate > > > > >> | > | | > > > > >> | > | |__ > BlockInfo > > > > >> | > | | > > > > >> | > |___ Block Request (APIS)-> |__ > BlockResize > > > > >> | > | > > > > >> | > |__ > BlockList > > > > >> | > | > > > > >> | > |__ > BlockDelete > > > > >> | > > > > > >> | > Heketi will have block API and volume API, when user submit a > Persistent > > > > >> | > volume claim, Kubernetes provisioner based on the storage > class(from PVC) > > > > >> | > talks to heketi for storage, heketi intern calls block or > volume API's > > > > >> | > based on request. > > > > >> | > > > > > >> | > > > > >> | This is probably wrong. It won't be Heketi calling block or > volume APIs. It > > > > >> | would be Kubernetes calling block or volume API *of* Heketi. > > > > >> | > > > > >> | > > > > >> | > With my limited understanding, heketi currently creates > clusters from > > > > >> | > provided nodes, creates volumes and handover them to the user. > > > > >> | > For block related API's, it has to deal with files right ? > > > > >> | > > > > > >> | > Here is how block API's look like in short- > > > > >> | > Create: heketi has to create file in the volume and export it > as a iscsi > > > > >> | > target device and hand it over to user. > > > > >> | > Info: show block stores information across all the clusters, > connection > > > > >> | > info, size etc. > > > > >> | > resize: resize the file in the volume, refresh connections > from initiator > > > > >> | > side > > > > >> | > List: List the connections > > > > >> | > Delete: logout the connections and delete the file in the > gluster volume > > > > >> | > > > > > >> | > Couple of questions: > > > > >> | > 1. Should Block API have sub API's such as FileCreate, > FileList, > > > > >> | > FileResize, File delete and etc then get it used in Block API > as they > > > > >> | > mostly deal with files. > > > > >> | > > > > > >> | > > > > >> | IMO, Heketi should not expose any File related API. It should > only have > > > > >> | APIs to service request for block devices; how the block > devices are > > > > >> | created and modified is an implementation detail. > > > > >> | > > > > >> | > > > > >> | > 2. How do we create the actual file in the volume, meaning > using FUSE > > > > >> | > mount (which may involve an extra process running) or gfapi, > again if gfapi > > > > >> | > should we go with c API's, python bindings or go bindings ? > > > > >> | > > > > > >> | 3. Should we get targetcli related (LUN exporting) setup done > from heketi > > > > >> | > or do we seek help from gdeploy for this ? > > > > >> | > > > > > >> | > > > > >> | I would prefer to either have it in Heketi or in Kubernetes. If > the API in > > > > >> | Heketi promises just the creation of block device, then the > rest of the > > > > >> | implementation should be in Kubernetes(the export part). If the > API in > > > > >> | Heketi promises create and export both, I would say Heketi > should have the > > > > >> | implementation within itself. > > > > >> | > > > > >> | > > > > >> > > > > >> IMO, we should not think about how the clients ( ex: k8s) use it, > because there may be different clients. > > > > >> We should concentrate mainly on 'in/out' of block API in Heketi. > Regardless of which client, the API should act the same way. > > > > >> > > > > >> --Humble > > > > >> _______________________________________________ > > > > >> Gluster-devel mailing list > > > > >> Gluster-devel@xxxxxxxxxxx > > > > >> http://www.gluster.org/mailman/listinfo/gluster-devel > > > > > > > > > > _______________________________________________ > > > > > Gluster-devel mailing list > > > > > Gluster-devel@xxxxxxxxxxx > > > > > http://www.gluster.org/mailman/listinfo/gluster-devel > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-devel > -- Pranith _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
