Using qemu is interesting, but the I/O should be using the IO path of QEMU block API. If not, TCMU would not know how to work with QEMU dynamic QCOW2 files. Now, if TCMU already has this, then that would be great! - Luis ----- Original Message ----- From: "Prasanna Kalever" <pkalever@xxxxxxxxxx> To: "Niels de Vos" <ndevos@xxxxxxxxxx> Cc: "Luis Pabón" <lpabon@xxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Ramakrishna Yekulla" <rreddy@xxxxxxxxxx>, "Humble Chirammal" <hchiramm@xxxxxxxxxx> Sent: Monday, September 19, 2016 7:13:36 AM Subject: Re: [Heketi] Block store related API design discussion On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos <ndevos@xxxxxxxxxx> wrote: > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote: > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos <ndevos@xxxxxxxxxx> wrote: > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote: > > >> Very good points. Thanks Prasanna for putting this together. I agree with > > >> your comments in that Heketi is the high level abstraction API and it should have > > >> an API similar of what is described by Prasanna. > > >> > > >> I definitely do not think any File Api should be available in Heketi, > > >> because that is an implementation of the Block API. The Heketi API should > > >> be similar to something like OpenStack Cinder. > > >> > > >> I think that the actual management of the Volumes used for Block storage > > >> and the files in them should be all managed by Heketi. How they are > > >> actually created is still to be determined, but we could have Heketi > > >> create them, or have helper programs do that. > > > > > > Maybe a tool like qemu-img? If whatever iscsi service understand the > > > format (at the very least 'raw'), you could get functionality like > > > snapshots pretty simple. > > > > Niels, > > > > This is brilliant and subset of the Idea falls in one among my > > thoughts, only concern is about building dependencies of qemu with > > Heketi. > > But at an advantage of easy and cool snapshots solution. > > And well tested as I understand that oVirt is moving to use qemu-img as > well. Other tools are able to use the qcow2 format, maybe the iscsi > servce that gets used does so too. > > Has there already been a decision on what Heketi will configure as iscsi > service? I am aware of the tgt [1] and LIO/TCMU [2] projects. Niels, yes we will be using TCMU (Kernel Module) and TCMU-runner (user space service) to expose file in Gluster volume as an iSCSI target. more at [1], [2] & [3] [1] https://pkalever.wordpress.com/2016/06/23/gluster-solution-for-non-shared-persistent-storage-in-docker-container/ [2] https://pkalever.wordpress.com/2016/06/29/non-shared-persistent-gluster-storage-with-kubernetes/ [3] https://pkalever.wordpress.com/2016/08/16/read-write-once-persistent-storage-for-openshift-origin-using-gluster/ -- Prasanna > > Niels > > 1. http://stgt.sourceforge.net/ > 2. https://github.com/open-iscsi/tcmu-runner > http://blog.gluster.org/2016/04/using-lio-with-gluster/ > > > > > -- > > Prasanna > > > > > > > > Niels > > > > > > > > >> We also need to document the exact workflow to enable a file in > > >> a Gluster volume to be exposed as a block device. This will help > > >> determine where the creation of the file could take place. > > >> > > >> We can capture our decisions from these discussions in the > > >> following page: > > >> > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes > > >> > > >> - Luis > > >> > > >> > > >> ----- Original Message ----- > > >> From: "Humble Chirammal" <hchiramm@xxxxxxxxxx> > > >> To: "Raghavendra Talur" <rtalur@xxxxxxxxxx> > > >> Cc: "Prasanna Kalever" <pkalever@xxxxxxxxxx>, "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, "Luis Pabon" <lpabon@xxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>, "Ramakrishna Yekulla" <rreddy@xxxxxxxxxx>, "Mohamed Ashiq Liyazudeen" <mliyazud@xxxxxxxxxx> > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM > > >> Subject: Re: [Heketi] Block store related API design discussion > > >> > > >> > > >> > > >> > > >> > > >> ----- Original Message ----- > > >> | From: "Raghavendra Talur" <rtalur@xxxxxxxxxx> > > >> | To: "Prasanna Kalever" <pkalever@xxxxxxxxxx> > > >> | Cc: "gluster-devel" <gluster-devel@xxxxxxxxxxx>, "Stephen Watt" <swatt@xxxxxxxxxx>, "Luis Pabon" <lpabon@xxxxxxxxxx>, > > >> | "Michael Adam" <madam@xxxxxxxxxx>, "Humble Chirammal" <hchiramm@xxxxxxxxxx>, "Ramakrishna Yekulla" > > >> | <rreddy@xxxxxxxxxx>, "Mohamed Ashiq Liyazudeen" <mliyazud@xxxxxxxxxx> > > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM > > >> | Subject: Re: [Heketi] Block store related API design discussion > > >> | > > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever <pkalever@xxxxxxxxxx> > > >> | wrote: > > >> | > > >> | > Hi all, > > >> | > > > >> | > This mail is open for discussion on gluster block store integration with > > >> | > heketi and its REST API interface design constraints. > > >> | > > > >> | > > > >> | > ___ Volume Request ... > > >> | > | > > >> | > | > > >> | > PVC claim -> Heketi --->| > > >> | > | > > >> | > | > > >> | > | > > >> | > | > > >> | > | __ BlockCreate > > >> | > | | > > >> | > | |__ BlockInfo > > >> | > | | > > >> | > |___ Block Request (APIS)-> |__ BlockResize > > >> | > | > > >> | > |__ BlockList > > >> | > | > > >> | > |__ BlockDelete > > >> | > > > >> | > Heketi will have block API and volume API, when user submit a Persistent > > >> | > volume claim, Kubernetes provisioner based on the storage class(from PVC) > > >> | > talks to heketi for storage, heketi intern calls block or volume API's > > >> | > based on request. > > >> | > > > >> | > > >> | This is probably wrong. It won't be Heketi calling block or volume APIs. It > > >> | would be Kubernetes calling block or volume API *of* Heketi. > > >> | > > >> | > > >> | > With my limited understanding, heketi currently creates clusters from > > >> | > provided nodes, creates volumes and handover them to the user. > > >> | > For block related API's, it has to deal with files right ? > > >> | > > > >> | > Here is how block API's look like in short- > > >> | > Create: heketi has to create file in the volume and export it as a iscsi > > >> | > target device and hand it over to user. > > >> | > Info: show block stores information across all the clusters, connection > > >> | > info, size etc. > > >> | > resize: resize the file in the volume, refresh connections from initiator > > >> | > side > > >> | > List: List the connections > > >> | > Delete: logout the connections and delete the file in the gluster volume > > >> | > > > >> | > Couple of questions: > > >> | > 1. Should Block API have sub API's such as FileCreate, FileList, > > >> | > FileResize, File delete and etc then get it used in Block API as they > > >> | > mostly deal with files. > > >> | > > > >> | > > >> | IMO, Heketi should not expose any File related API. It should only have > > >> | APIs to service request for block devices; how the block devices are > > >> | created and modified is an implementation detail. > > >> | > > >> | > > >> | > 2. How do we create the actual file in the volume, meaning using FUSE > > >> | > mount (which may involve an extra process running) or gfapi, again if gfapi > > >> | > should we go with c API's, python bindings or go bindings ? > > >> | > > > >> | 3. Should we get targetcli related (LUN exporting) setup done from heketi > > >> | > or do we seek help from gdeploy for this ? > > >> | > > > >> | > > >> | I would prefer to either have it in Heketi or in Kubernetes. If the API in > > >> | Heketi promises just the creation of block device, then the rest of the > > >> | implementation should be in Kubernetes(the export part). If the API in > > >> | Heketi promises create and export both, I would say Heketi should have the > > >> | implementation within itself. > > >> | > > >> | > > >> > > >> IMO, we should not think about how the clients ( ex: k8s) use it, because there may be different clients. > > >> We should concentrate mainly on 'in/out' of block API in Heketi. Regardless of which client, the API should act the same way. > > >> > > >> --Humble > > >> _______________________________________________ > > >> Gluster-devel mailing list > > >> Gluster-devel@xxxxxxxxxxx > > >> http://www.gluster.org/mailman/listinfo/gluster-devel > > > > > > _______________________________________________ > > > Gluster-devel mailing list > > > Gluster-devel@xxxxxxxxxxx > > > http://www.gluster.org/mailman/listinfo/gluster-devel _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
