Re: Gluster Coreutils

[Joe Julian <joe@xxxxxxxxxxxxxxxx>]

On 06/14/2015 11:43 AM, Raghavendra Talur wrote:

On Sun, Jun 14, 2015 at 11:02 PM, chris holcombe <chris.holcombe@xxxxxxxxxxxxx> wrote:

Welcome to the party Matthew!  Nice to see you're still keeping an eye on on the list.  I'm excited to see this collaboration.  This is going to turn out great :)

On 06/14/2015 01:58 AM, Matthew McKeen wrote:

Hey Craig and Chris:

I might be interested in collaborating on this as well.

Will be useful when I come back to FB in September.

Let me know where the public repository ends up being.

Thanks,
Matthew McKeen

P.S. Tell Richard I said hello

On Fri, Jun 12, 2015 at 11:29 AM chris holcombe <chris.holcombe@xxxxxxxxxxxxx <mailto:chris.holcombe@xxxxxxxxxxxxx>> wrote:

    Yeah I have this repo but it's basically empty:
    https://github.com/cholcombe973/GlusterUtils

    On 06/12/2015 11:27 AM, Craig Cabrey wrote:
    > Chris,
    >
    > That sounds good to me.
    >
    > I already have started on implementation, just to get familiar
    with the codebase and GFAPI.
    >
    > Is there a public repo that we can use for collaboration?
    >
    > Craig
    >
    >> On Jun 12, 2015, at 10:46 AM, chris holcombe
    <chris.holcombe@xxxxxxxxxxxxx

    <mailto:chris.holcombe@xxxxxxxxxxxxx>> wrote:
    >>
    >> Craig,
    >>
    >> I was actually planning on building the same tool set.  I would
    like to work with you also on this if that's ok.
    >>
    >> -Chris
    >>
    >> On 06/12/2015 10:43 AM, Jeff Darcy wrote:
    >>>> Hi everyone,
    >>>>
    >>>> This summer I am an intern at Facebook working on the Gluster
    team. Part of
    >>>> my project for the summer includes developing a set of
    coreutils that
    >>>> utilizes the Gluster C API natively.
    >>>>
    >>>> This project is similar in nature to the NFS coreutils that
    some of you may
    >>>> have heard about from the other Facebook engineers at the
    Gluster summit
    >>>> recently. I just wanted to reach out to the Gluster community
    to gather
    >>>> ideas, potential features, feedback, and direction.
    >>>>
    >>>> The initial set of utilities that I am developing includes
    the following:
    >>>>
    >>>> * cat
    >>>> * mkdir
    >>>> * put (read from stdin and write to a file)
    >>>> * mv
    >>>> * ls
    >>>> * rm
    >>>> * tail
    >>>>
    >>>> Again, any feedback will be welcome.
    >>> Hi, Craig, and welcome to the project.  :)
    >>>
    >>> There seems to be some overlap with a proposal Ragahavendra
    Talur sent out
    >>> a couple of days ago.
    >>>
    >>>
    https://urldefense.proofpoint.com/v1/url?u=https://docs.google.com/document/d/1yuRLRbdccx_0V0UDAxqWbz4g983q5inuINHgM1YO040/edit?usp%3Dsharing&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=ThH6JMKaB%2Fxgkh9d2jPjehcdps8B69L0q04jdBbZvX4%3D%0A&m=86la5Xg7nlxAzIR6E5c2v2SgQSd6VssYzB%2BklM3wf%2BI%3D%0A&s=8d55bb5770b8ed1d683a6908a05af32b79289735c537c660252fcaa7c690e162
    >>>
    >>> This seems like an excellent opportunity to collaborate.     Ideally, I think
    >>> it would be useful to have both an FTP-client-like "shell" and
    a set of
    >>> standalone "one shot" commands, based on as much common code
    as possible.
    >>>
    >>> _______________________________________________
    >>> Gluster-devel mailing list

    >>> Gluster-devel@xxxxxxxxxxx <mailto:Gluster-devel@xxxxxxxxxxx>
    >>>
    https://urldefense.proofpoint.com/v1/url?u=http://www.gluster.org/mailman/listinfo/gluster-devel&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=ThH6JMKaB%2Fxgkh9d2jPjehcdps8B69L0q04jdBbZvX4%3D%0A&m=86la5Xg7nlxAzIR6E5c2v2SgQSd6VssYzB%2BklM3wf%2BI%3D%0A&s=28546cdc6fdf6f75f4cfa4b8260abc595eee96601a5f849ebb230ddbd1faf8b3

    _______________________________________________
    Gluster-devel mailing list
    Gluster-devel@xxxxxxxxxxx <mailto:Gluster-devel@xxxxxxxxxxx>
    http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

Hi Craig, Chris and Matthew,

Welcome to the project  :)

We are really excited to see so many interested in getting core utils like feature in Gluster.

As Jeff mentioned earlier, we have put up a very similar proposal recently. He also shared

the design doc in the same mail.

Here is an illustration of how we imagined it:

​​

In gist, all the core utils commands would be functions in what we are calling

glusterclientd. The functions would return the output in a format like JSON or XML.

It would be the handlers which will send the data to a *real* client.

To support one-off commands, we were thinking of having a cli-direct handler,

which when invoked will behave like a one-off command and output directly to cli.

Also, we were of the opinion that python would be a good language to get this done.

As you can see, core-utils would be part of a bigger project and it would be really useful

for all of us to have same code base.

Please read the design doc and give your reviews either in doc or here in mail.

Thanks!

Raghavendra Talur 

One issue to keep in mind is (potentially) tens of thousands of clients connecting simultaneously. This would be a huge load on glusterd and potentially use up all the available ports (depending on tcp _fin_timeout).

See https://docs.google.com/document/d/15IiPVIPMzgGwkt1sKuIusRE2l3pQY6NnA4WmWaMsFJE/edit for a potential solution.

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel