Supposedly cephfs-hadoop worked and/or works on hadoop 2. I am in the process of getting it working with cdh5.7.0 (based on hadoop 2.6.0). I'm under the impression that it is/was working with 2.4.0 at some point in time. At this very moment, I can use all of the DFS tools built into hadoop to create, list, delete, rename, and concat files. What I am not able to do (currently) is run any jobs. https://github.com/ceph/cephfs-hadoop It can be built using current (at least infernalis with my testing) cephfs-java and libcephfs. The only thing you'll for sure need to do is patch the file referenced here: https://github.com/ceph/cephfs-hadoop/issues/25 When building, you'll want to tell maven to skip tests (-Dmaven.test.skip=true). Like I said, I am digging into this still, and I am not entirely convinced my issues are ceph related at the moment. -- Adam On Sat, Apr 30, 2016 at 1:51 PM, Erik McCormick <emccormick@xxxxxxxxxxxxxxx> wrote: > I think what you are thinking of is the driver that was built to actually > replace hdfs with rbd. As far as I know that thing had a very short lifespan > on one version of hadoop. Very sad. > > As to what you proposed: > > 1) Don't use Cephfs in production pre-jewel. > > 2) running hdfs on top of ceph is a massive waste of disk and fairly > pointless as you make replicas of replicas. > > -Erik > > On Apr 29, 2016 9:20 PM, "Bill Sharer" <bsharer@xxxxxxxxxxxxxx> wrote: >> >> Actually this guy is already a fan of Hadoop. I was just wondering >> whether anyone has been playing around with it on top of cephfs lately. It >> seems like the last round of papers were from around cuttlefish. >> >> On 04/28/2016 06:21 AM, Oliver Dzombic wrote: >>> >>> Hi, >>> >>> bad idea :-) >>> >>> Its of course nice and important to drag developer towards a >>> new/promising technology/software. >>> >>> But if the technology under the individual required specifications does >>> not match, you will just risk to show this developer how worst this >>> new/promising technology is. >>> >>> So you will just reach the opposite of what you want. >>> >>> So before you are doing something, usually big, like hadoop on an >>> unstable software, maybe you should not use it. >>> >>> For the good of the developer, for your good and for the good of the >>> reputation of the new/promising technology/software you wish. >>> >>> To force a pinguin to somehow live in the sahara, might be possible ( at >>> least for some time ), but usually not a good idea ;-) >>> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |