Is here any progress on Cephfs Windows Client ? 2013-11-08 22:15 GMT+08:00 Alphe Salas Michels <asalas@xxxxxxxxx>: > Hello malcom and matt thank you for apporting some more information source. > OpenAFS is sure interesting httpfs too. > > I hope it will help us on deciding the best path to follow in our interface > with window. > Actually I still trying to isolate the needed client code in the shortest > way possible. > > Regards. > > Alphe Salas > > El nov 7, 2013 9:11 p.m., "Malcolm Haak" <malcolm@xxxxxxx > <mailto:malcolm@xxxxxxx>> escribió: > > I'm just going to throw these in there. > > http://www.acc.umu.se/~bosse/ <http://www.acc.umu.se/%7Ebosse/> > > > They are GPLv2 some already use sockets and such from inside the > kernel. Heck you might even be able to mod the HTTP one to use > rados gateway. I don't know as I havent sat down and pulled them > apart enough yet. > > They might help, but they might be useless. Not sure. > > On 08/11/13 06:47, Alphe Salas Michels wrote: > > Hello all I finally finished my first source code extraction > that starts > from ceph/src/client/fuse_ll.c > The result is accurate unlike previous provided results. > basically the > script start from a file extract all the private includes > definitions > #include "something.h" and recursively extract private includes > too. the > best way to know who is related with who. > > starting from fuse_ll.cc I optain 390 files retreived and 120 > 000 lines > of code ! > involved dirs are : in ceph/src > objclass/, common/, msg/, common/, osdc/, include/, client/, mds/, > global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ > > probably not a good way to analyse what amount of work it means > since > most of those directories are the implementation of servers > (osd, mon, > mds) and even if only a tiny bit of them is needed at client > level. you > need two structures from ./osd/OSD.h and my script by relation will > take into acount the whole directory... > > I ran the script with libcephfs.cc as start point and got almost the > same results. 131 000 lines of code and 386 files most of the > same dirs > involved. > > > > I think I will spend alot of time doing the manual source code > isolation > and understand way each #include is set in the files I read (what > purpose they have do they allow to integrate a crucial data type > or not. > > > The other way around will be to read src/libcephfs.cc. It seems > shorter > but without understanding what part is used for each included > header I > can t say anything... > > > > I will keep reading the source code and take notes. I think in > the case > of libcephfs I will gain alot of time. > > signature > > *Alphé Salas* > Ingeniero T.I > > asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx> > *www.kepler.cl <http://www.kepler.cl> <http://www.kepler.cl>* > > > On 11/07/13 15:02, Alphe Salas Michels wrote: > > Hello D.Ketor and Matt Benjamin, > You give me alot to think about and this is great! > I merged your previous post to make a single reply that > anyone can > report to easyly > > Windows NFS 4.1 is available here: > http://www.citi.umich.edu/projects/nfsv4/windows/readme.html > > pnfs is another name for NFS4.X. It is presented as > alternative to > ceph and we get known terminology as MDS and OSD but without > the self > healing part if I understand well my rapid look on the > topic. (when I > say rapid look I mean ... 5 minutes spent in that... which > is really > small amount of time to get an accurate view on something) > > > starting from mount.ceph ... I know that mount.ceph does > little but it > is a great hint to know what ceph needs and do things. > Basically mount.ceph modprobe the ceph driver in the linux > kernel then > call mount with the line command passed args and the cephfs > type as > argument. Then the kernel does the work I don t understand > yet what is > the start calls that are made to the ceph driver but it > seemed to me > that is was relatively light. (a first impression compared > to ceph-fuse.) > > I think I will do both isolate source code from ceph-client > kernel > (cephfs module for linux kernel) and the one pointed by Sage > starting > from client/fuse_ll.cc in ceph master branch. The common > files betwin > those 2 extractions will be our core set of mandatory features. > > Then we try to compile with cygwin a cephfs client library . > Then we > will try to interface with a modified windows nfs 4.1 client > or pnfs > or any other that will accept to be compiled with gcc for > win32... > > the fact that windows 8.1 is and windows 2012 are out of > reach at the > moment is not a problem to me. > > Our first concern is to understand what is ceph protocol. > Then adapt > it to something that can be used on windows prior windows > 8.1. Dokan > fs if I remember well use too the WDK (windows driver > dev-kit ) for it > s compilation so possibly we will see the same limitations. > > We need to multiply our source of information by example > regarding > ceph-client (kernel or fuse, radosgw is on a different layer > so I will > not try anything around it at first.) And we need to > multiply our > source of information by example regarding virtual file system > technologies on windoes OS. > Alot of work but all of those available source code everyone > point at > me will make our best solution. And in the end we will choose > technologies knowing what we do and what concequencies they > have. > > regards, > > > > > Regards > > signature > > *Alphé Salas* > Ingeniero T.I > > asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx> > > > > On 11/07/13 11:29, Ketor D wrote: > > Hi Alphe: > Yes Callback Filesystem is very expensive and > can't open source. > It's not a good choice for ceph4win. > Another way for ceph4win maybe develop a > kernel-mode fs like > pnfs. pnfs has a kernel-mode windows client. I think you > can read its > src code and maybe migrating from ceph kernel client to > windows kernel > fs is easier than from userspace ceph fuse client.And a > kernel-mode fs > client has greater performance than userspace fs like > ceph-fuse client > and ceph kernel client. > > Regards. > > On 11/07/13 11:50, Matt W. Benjamin wrote: > > Hi, > > The Window NFS v4.1 client is what we work on, so this > may be good for > code sharing. The license is lgplv2, like Ceph's. > > Something important to be aware of is that the client > uses rdbss, which > is a (partial) fsd abstraction that simplified > implementation > quite a bit, kind of like a mini driver. However, > Microsoft's support > for rdbss has been in limbo for a bit. For example, to > link with > the rdbss symbols you can't use the Windows 8 driver > kit--you'll need > to use the one for Windows 7. (There's a private rdbss2 > used internally > by Microsoft's SMB implemenation. A the moment, 3rd > party drivers > can't use that.) > > We've been in communication with Microsoft about this > issue, and know of > a few other fsds using it, but it could be a good thing > for that > lobbying > effort to have another user--or it could be a dead end :(. > > There are a couple of other choices if you're looking to > go this route, > that I'm aware of (and we may need to take them too, if > RDBSS has no > way forward), but the required work could be a lot larger. > > Matt > > ----- "Ketor D"<d.ketor@xxxxxxxxx > <mailto:d.ketor@xxxxxxxxx>> wrote: > > Hi Alphe: > Yes Callback Filesystem is very expensive > and can't open > source. > It's not a good choice for ceph4win. > Another way for ceph4win maybe develop a > kernel-mode fs like > pnfs. pnfs has a kernel-mode windows client. I think > you can read its > src code and maybe migrating from ceph kernel client > to windows > kernel > fs is easier than from userspace ceph fuse > client.And a kernel-mode > fs > client has greater performance than userspace fs > like ceph-fuse > client > and ceph kernel client. > > Regards. > > > > On Thu, Nov 7, 2013 at 8:13 PM, Alphe Salas > Michels<asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx>> > > wrote: > > Commercial libraries are a pain ... > > If we want the more permossive licence offered > by callback file > > system we > > have to buy it for 20.000 usd. Then we will have > to provide a > > backbox that > > we have no control upon and that will kill our > product anytime they > > want anf > > if they decide to stop their commercial activity > we will be in the > > same > > situation that with dokanfs but without having > the source code of > > the black > > box. If i have to spend 20 000 dollars i would > prefere paying > > someone to > > retake dokanfs or to write from scratch a > dokanfs fuselike software > > make it > > all shiny and pumpy fantastic and ready to plug > to ceph client. > > I would prefere if people have to pay something > to get access to > > ceph4win > > that this money goes in ceph main branch > pockets... Or as a gift you > > donante > > to ceph 10 dollars you get 2 free registration > codes for > > ceph4win... or > > something like that. > > If ceph4win as to be comercial then I would > prefer delegate the task > > to a > > company like south river technologies and their > great product > > webdrive. I > > would mininaly get involved in that project and > simply buy the final > > product > > to sell it together with my ceph based product > (which could be a > > calxeda > > ceph box or something like that). > > I m open anyway to any proposition. But I doubt > that callback > > filesystem > > offers us a suitable solution in the way I see > ceph4win to be spread > > and > > used... I m maybe wrong. And anything that will > be done around > > ceph4win will > > be public documented etc... And licensed the way > that if someone > > want to > > build a commercial solution on top of it, that > would be a > > possibility. > > My idea is to giveback somehow to ceph project > and at same time > > forge a > > better knowledge in ceph technologies. Because > like many in libre > > world I > > think the business is in the services around the > software more than > > on the > > software. That the ones writing code should be > financed and benefits > > from > > the one selling and giving support of the > software at all levels. I > > m > > probably too idealistic. And too optimistic > after all I m the one > > saying I > > will do this stuff I have no idea how but well > it is interesting and > > fun so > > lets do it. > > Regards, > > P.S: using commercial backend libraries appart > including their own > > cost will > > force you to use commercial IDE like MS > VisualStudio because their > > library > > has some kind of drm that only that IDE compiler > can use. So alot of > > cost > > and yet there is nothing done. If I had to open > a kickstarter > > project saying > > we need 60 000 USD to do ceph4win with that > monney we will buy the > > right to > > use and share a commercial copyrighted library > but abandonned > > punctually to > > us in public domaine and that we will > eventually produce something > > out of > > it. I doubt I will get a dollar. > > We still can suggest the idea to Edlos the > commercial company that > > has the > > copyright of Callback FS, Or to buy them their > product in a blender > > way > > (blender was bought with donation before being > put opensource and > > public > > domaine), Or to open source their library. But > in commercial minds > opensourcing = death of their technical > advantage and death of > > their > > marketing strategy. They will have to invent > something more to > > retrieve > > monney from it. > > El nov 6, 2013 11:22 p.m., "Ketor D" > <d.ketor@xxxxxxxxx <mailto:d.ketor@xxxxxxxxx> > <mailto:d.ketor@xxxxxxxxx > <mailto:d.ketor@xxxxxxxxx>>> escribió: > > > Hi Alphe, > I think you could try Callback > Filesystem dev > > framework. It > > is a commerical dev framework and is > maintained by Edlos today. > I have communicated with Edlos to > get a try code for > development. To dokan, Callback Filesystem > has vary document and > > maybe > > more stabilize. > > Regards. > > > > On Thu, Nov 7, 2013 at 10:00 AM, Alphe > Salas <asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx> > <mailto:asalas@xxxxxxxxx > > <mailto:asalas@xxxxxxxxx>>> wrote: > > Hello ketor thank you for your interest > un ceph4win. Since > > muy > > first mail I > > exposed the lacks of dokanfs and that I > m far from being a > specialist un > > filesystems. > > I exposed what i like un dokanfs bit I > not a fanátic of it. > > Muy > > goal is to > > have something working quickly. > > > > So I am up to any proposición sure the > one with the more docs > > and > > support > > will be the best choice. As for right > now what I need is > understand what are > > the files involved what are the > interfaces functions and what > > are > > the needed > > library dependencies and if they exist > ported to windows with > cygwin. And > > all that is retrieved from source code. > > > > Regards. > > > > El nov 6, 2013 10:34 p.m., "Ketor D" > <d.ketor@xxxxxxxxx <mailto:d.ketor@xxxxxxxxx> > <mailto:d.ketor@xxxxxxxxx > > <mailto:d.ketor@xxxxxxxxx>>> escribió: > > > > >> Hi Alphe, > >> We are taking an interest in your > work on Ceph Client > > for > > Windows > >> with Dokan.As we know, the performance > of Dokan is not very > good, and it's > >> abandoned 3 years ago. > >> I have learned and used > OpenDedup(SDFS) for a long > > time. > > OpenDedup > >> has a Dokan version. And the author of > OpenDedup said > >> > >> The Dokan library is quite flakey and > testing should be > performed before > >> putting into production > >> > >> So what do you think about this? > And if there is > > another > > solution of > >> fuse-like filesystem dev framwork on > Windows? > >> > >> Best Wish! > >> > >> > >> > >> On Thu, Nov 7, 2013 at 5:47 AM, Alphe > Salas Michels > <asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx> > <mailto:asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx>>> > > >> wrote: > >>> > >>> Hello I created the github repository > for this project > >>>https://github.com/alphe/Ceph4Win > >>> > >>> Regards, > >>> > >>> signature > >>> > >>> *Alphé Salas* > >>> Ingeniero T.I > >>> > >>>asalas@xxxxxxxxx > <mailto:asalas@xxxxxxxxx> > <mailto:asalas@xxxxxxxxx <mailto:asalas@xxxxxxxxx>> > > > >>> *<http://www.kepler.cl>* > >>> > >>> On 11/05/13 21:00, Sage Weil wrote: > >>>> > >>>> Hi Alphe, > >>>> > >>>> On Tue, 5 Nov 2013, Alphe Salas > Michels wrote: > >>>>> > >>>>> signature *Hi, Sage ! > >>>>> thank you for you enthousiast reply. > >>>>> I sure want to make the best use of > everything or > > anything > > previously > >>>>> done to > >>>>> tend to > >>>>> write ceph cliente for windows. > >>>>> > >>>>> Apart using libre tools for building > the future ceph > > cliente > > I am open > >>>>> to > >>>>> anything. > >>>>> I would recommand eclipse CDT or > Code::BLocks they are > > based > > on mingwin > >>>>> open > >>>>> and easyly enhanceable.** > >>>>> > >>>>> more free tools can be found here: > >>>>>>http://www.freebyte.com/programming/cpp/#cppcompilers > >>>>> > >>>>> > >>>>> I will read libcephfs source code > and take some notes > > about the > > >>>>> protocol. > >>>> > >>>> I think you don't need to worry about > hte protocol at all, > > since > > >>>> libcephs > >>>> implements it for you (and will > capture any future > > changes). > > >>>> > >>>>> I was more going from what I know > and trying to track down > > how > > >>>>> mount.ceph work > >>>>> with the parameters passed to it. > >>>>> since it point finally to > Kernel/fs/ceph and that I don t > > really > > >>>>> understand > >>>>> how that module work and that it > probably points to some > > other > > >>>>> dependencies > >>>>> Reading libcephfs source code could > be a big gain of > > time. > > >>>> > >>>> (I would also ignore mount.ceph as > everything it does it > specific to > >>>> how Linux mounts work.) > >>>> > >>>>> basically on the protocol what is > need are: > >>>>> > >>>>> 1) open and maintain a connection > (socket open, auth, etc > > ) > > >>>>> 2) retreive a map of directories and > disk Quota (disk > > sizing > > Y TB free, > >>>>> Z TB > >>>>> total) > >>>>> 3) procedure to send files / > directories in a maner that > > it > > will allow > >>>>> our > >>>>> client to fit ceph transmission > protocols > >>>>> (limit bandwith for stability?, > limit connection amount?, > limit cpu > >>>>> use?, > >>>>> Cache for preparing data transfer (a > FIFO cache)?) > >>>>> 4)Procedure to retreive files / > directory from ceph > > cluster > > >>>>> 5) Management copy/move files > /Directories, FS stats, > Connection Stats. > >>>>> logging. > >>>>> > >>>>> My idea to progress is to take those > main bulletpoint in > > ceph > > protocol > >>>>> based > >>>>> on general ideas of what ceph file > system does and start > identifying > >>>>> parts > >>>>> from libcephfs to match those "needs". > >>>> > >>>> Instead, I would look at > include/cephfs/libcephfs.h, the > interface that > >>>> libcephfs provides, and try to map > that to what the fuse > > layer > > expects. > >>>> There is both a path-based that I > suspsect lends itself > > well > > to the > >>>> Windows interface and (very soon now) > a handle based API > > that is > > >>>> targetted > >>>> at the Unix-style VFS layers. I'm > mostly guessing, > > though, > > since I've > >>>> never seen any low-level fs code in > windows before. > >>>> > >>>> In this case, the analogous code for > Linux should be > client/fuse_ll.cc > >>>> itself (and not much else), although > there will probably be > > a > > few tricks > >>>> necessary to map cleanly onto how the > windows interfaces > > work. > > >>>> > >>>> Does that make sense? > >>>> > >>>> Cheers! > >>>> sage > >>>> > >>>> > >>>>> Any suggestion and contributions are > welcome. > >>>>> > >>>>> > >>>>> * > >>>>> On 11/05/13 11:23, Sage Weil wrote: > >>>>>> > >>>>>> Hi Alphe, > >>>>>> > >>>>>> On Mon, 4 Nov 2013, Alphe Salas > Michels wrote: > >>>>>>> > >>>>>>> Good day developers! > >>>>>>> > >>>>>>> I would like to propose to the one > interested work with > > me to > > >>>>>>> develop a > >>>>>>> ceph > >>>>>>> cliente for MS windows world, > Basing us on dokanFS. > >>>>>>> > >>>>>>> My company is a ceph enthousiast > that use on a dayly > > basis > > ceph and > >>>>>>> that > >>>>>>> need > >>>>>>> both transfer speed and big > expendable and cheap > > storage. > > >>>>>>> My company is specialised in data > recovery and we want > > to > > participate > >>>>>>> to > >>>>>>> ceph > >>>>>>> effort by bringing a ceph cliente > for windows. > >>>>>> > >>>>>> Awesome! > >>>>>> > >>>>>>> Our experience shows us that the > best gateway is each > clientes being > >>>>>>> its > >>>>>>> own > >>>>>>> gateway, instead of having a > bottle neck server or a > > cluster of > > >>>>>>> bottle > >>>>>>> neck > >>>>>>> servers as gateway (FTP, samba, > SFTP,webdav, s3, > > etc..). > > >>>>>>> > >>>>>>> We already did some research in > that domain. > >>>>>>> > >>>>>>> Dokan FS is an intent to write an > opensource fuse like > cliente for > >>>>>>> MS > >>>>>>> windows. > >>>>>>> > >>>>>>> More information on DOKANFS can be > triggered here > >>>>>>>http://dokan-dev.net/en/download/ > >>>>>>> > >>>>>>> Positive points of using DOKANFS. > >>>>>>> > >>>>>>> - its opensourced and well > licenced mit licence, gpl > licence and lgpl > >>>>>>> licence. > >>>>>>> > >>>>>>> Negative point of using DOKAN FS. > >>>>>>> - unreachable author > >>>>>>> - Poor documentation . Dev > comments in japanese. > >>>>>>> - Work in progress so it is > unstable and needs to be > > updated, > > >>>>>>> debugged and > >>>>>>> maintained by a MS Windows file > system expert > > developper. > > >>>>>> > >>>>>> I am not very familiar with windows > storage APIs, but > somebody told me > >>>>>> at once point there were several > interfaces against which > > a > > new file > >>>>>> system could be implemented, > everything from a full > in-kernel driver > >>>>>> to > >>>>>> something that is explorer-based. > Are any of those > suitable? Using a > >>>>>> potentially abandoned fuse-like > layer makes me a bit > > nervous. > > >>>>>> > >>>>>> That said, > >>>>>> > >>>>>>> > >>>>>>> I try past year to do a merge from > ceph-fuse to dokanfs > >>>>>>> here are what I learnt. > >>>>>>> - Ceph-fuse and related source > code is around 60 000 > > lines > > of code. > >>>>>>> - Ceph protocol isn t documented > so it is like trying > > to > > draw a map > >>>>>>> of > >>>>>>> america > >>>>>>> using only a sextan and a compass. > >>>>>>> > >>>>>>> Those led me to those conclusions: > >>>>>>> - I can t do it alone. > >>>>>>> - It is easier to draw down the > ceph protocol way to > > work from > > >>>>>>> kernel/fs/ceph > >>>>>>> sources and mount.ceph > >>>>>>> - Ceph depending libraries may be > unexistant or not up > > to > > date in > >>>>>>> their > >>>>>>> port > >>>>>>> on MS Windows (cygwin) > >>>>>> > >>>>>> I think the most sane path should > be to make libcephfs > sufficiently > >>>>>> portable to build on windows (or > cygwin). For the bits > > used > > by the > >>>>>> client-side coe, I don't think > there should be much in > > the > > way of > >>>>>> dependencies, and the main > challenge would be untangling > > the > > build for > >>>>>> the necessary pieces out from the > rest of Ceph. > >>>>>> > >>>>>> Have you seen the wip-port > portability work that is > currently underway > >>>>>> by > >>>>>> Noah and Alan? That may solve many > of the cygwin > > problems > > you are > >>>>>> seeing > >>>>>> today. > >>>>>> > >>>>>>> - MS file system specialist are > hard do find in the > > "open > > source > >>>>>>> libre > >>>>>>> world" > >>>>>>> so I will try in the commercial world. > >>>>>>> > >>>>>>> The commercial world has some > problems too. They need > > ceph > > protocol > >>>>>>> draft > >>>>>>> to > >>>>>>> implemente it to their own product > They will have > > licencing > > >>>>>>> /commercial > >>>>>>> politics that infringe lgpl, and > hide that most of the > > work > > is done > >>>>>>> by > >>>>>>> people > >>>>>>> other than them. They will not > participate in a > > financial > > way to ceph > >>>>>>> enhancement and growth. > >>>>>> > >>>>>> I don't think reimplementing the > client code is an > > efficient way > > >>>>>> forward. > >>>>>> Unless the goal is a pure kernel > implementation...but a > significant > >>>>>> ongoing investment in development > resources would be > > needed > > for that > >>>>>> going > >>>>>> forward. I suspect that is a > challenge for a platform > > that > > does not > >>>>>> typically rally that sort of > community effort. > >>>>>> > >>>>>> The easiest thing is of course just > to use CIFS and > > Samba > > (which works > >>>>>> today). A fuse-like approach is > probably a reasonably > middle ground > >>>>>> (both > >>>>>> in initial effort and > maintainability going forward)... > >>>>>> > >>>>>> sage > >>>>>> > >>>>>> > >>>>> > >>> > >>> -- > >>> To unsubscribe from this list: send > the line "unsubscribe > ceph-devel" in > >>> the body of a message > tomajordomo@xxxxxxxxxxxxxxx > <mailto:tomajordomo@xxxxxxxxxxxxxxx> > <mailto:majordomo@xxxxxxxxxxxxxxx > <mailto:majordomo@xxxxxxxxxxxxxxx>> > > >>> More majordomo info at > > http://vger.kernel.org/majordomo-info.html > > >> > >> > > > > > > -- > To unsubscribe from this list: send the line > "unsubscribe ceph-devel" > in > the body of a message tomajordomo@xxxxxxxxxxxxxxx > <mailto:tomajordomo@xxxxxxxxxxxxxxx> > More majordomo info > athttp://vger.kernel.org/majordomo-info.html > <http://vger.kernel.org/majordomo-info.html> > > > > -- > To unsubscribe from this list: send the line "unsubscribe > ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > <mailto:majordomo@xxxxxxxxxxxxxxx> > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
