On 03/19/2014 05:58 PM, Noah Watkins wrote: > That certainly is odd. Does it work if you list both old and new > properties (perhaps the CLI tools are looking at an older property.. > but that seems unlikely)? Sorry I don't have more answers, I haven't > yet deploying Hadoop 2.x.. Another strange thing but.. putting the both the properties on core-site.xml makes both tools works. - Gurvinder > > On Wed, Mar 19, 2014 at 9:30 AM, Gurvinder Singh > <gurvindersinghdahiya@xxxxxxxxx> wrote: >> On 03/19/2014 05:18 PM, Noah Watkins wrote: >>> Err, obviously switching things out for Ceph rather than Gluster. >>> >>> On Wed, Mar 19, 2014 at 9:18 AM, Noah Watkins <noah.watkins@xxxxxxxxxxx> wrote: >>>> Looks like this is a configuration issue that has popped up with other >>>> 3rd party file systems in Hadoop 2.x with YARN. >>>> >>>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3C1023550423.3137743.1371825668412.JavaMail.root@xxxxxxxxxx%3E >>>> >>>> Says use this: >>>> >>>> <property> >>>> <name>fs.AbstractFileSystem.glusterfs.impl</name> >>>> <value>org.apache.hadoop.fs.glusterfs.GlusterFS</value> >>>> </property> >> Interestingly this solves the mapred history server issue. It created >> the history folder and started. >> >> <property> >> <name>fs.AbstractFileSystem.ceph.impl</name> >> <value>org.apache.hadoop.fs.ceph.CephHadoop2FileSystem</value> >> <description> >> </description> >> </property> >> >> But now this fails as >> >> hadoop fs -ls / >> ls: No FileSystem for scheme: ceph >> >> - Gurvinder >>>> >>>> Apparently rather than the `fs.ceph.impl` property in 2.x >>>> >>>> >>>> On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh >>>> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>>>> On 03/19/2014 04:50 PM, Noah Watkins wrote: >>>>>> Since `hadoop -fs ls /` seems to work on your local node, can you >>>>>> verify that (1) it is in fact listing the contents of CephFS, and (2) >>>>>> that on your worker nodes where the error is occuring that the >>>>>> relevant dependencies (naming the Ceph hadoop bindings) are installed >>>>>> and in the classpath? >>>>> 1) yes it is listing the cephfs contents. >>>>> 2) Currently the setup is to start the yarn manager and mapreduce >>>>> history server on the same node and then setup the worker nodes. Yarn >>>>> starts up fine, but the mapreduce history server gives me the error. The >>>>> dependency lib are loaded in the mapreduce history server startup. Here >>>>> is the link to start up message which shows that the libcephfs and >>>>> cephfs-hadoop jar are loaded. >>>>> >>>>> http://pastebin.com/wED7nMMT >>>>> >>>>> I am wondering if this property value is correct >>>>> >>>>> <property> >>>>> <name>fs.ceph.impl</name> >>>>> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> >>>>> </property> >>>>> >>>>> or should I set it to something for hadoop 2.x I have tried setting it up as >>>>> >>>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as >>>>> >>>>> # hadoop fs -ls / >>>>> -ls: Fatal internal error >>>>> java.lang.RuntimeException: java.lang.NoSuchMethodException: >>>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) >>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) >>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) >>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) >>>>> at >>>>> org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) >>>>> at >>>>> org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) >>>>> at >>>>> org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) >>>>> at >>>>> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) >>>>> at org.apache.hadoop.fs.shell.Command.run(Command.java:154) >>>>> at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) >>>>> Caused by: java.lang.NoSuchMethodException: >>>>> org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() >>>>> at java.lang.Class.getConstructor0(Class.java:2800) >>>>> at java.lang.Class.getDeclaredConstructor(Class.java:2043) >>>>> at >>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) >>>>> ... 17 more >>>>> >>>>> Regards, >>>>> Gurvinder >>>>> >>>>>> >>>>>> The error itself looks like a missing dependency, but that exception >>>>>> being thrown might also be tirggered by other problems while loading >>>>>> the bindings. >>>>>> >>>>>> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh >>>>>> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>>>>>> On 03/19/2014 03:51 PM, Noah Watkins wrote: >>>>>>>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh >>>>>>>> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >>>>>>>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >>>>>>>>> support. I am able to run the command such as >>>>>>>> >>>>>>>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1? >>>>>>>> >>>>>>> Yes. >>>>>>>> Can you paste your core-site.xml? >>>>>>> >>>>>>> Below is the relevant part for ceph conf >>>>>>> >>>>>>> <property> >>>>>>> <name>fs.defaultFS</name> >>>>>>> <value>ceph://xxxxxxxxxxxx:6789/</value> >>>>>>> </property> >>>>>>> >>>>>>> <property> >>>>>>> <name>ceph.conf.file</name> >>>>>>> <value>/etc/ceph/ceph.conf</value> >>>>>>> </property> >>>>>>> >>>>>>> <property> >>>>>>> <name>ceph.auth.id</name> >>>>>>> <value>admin</value> >>>>>>> </property> >>>>>>> >>>>>>> <property> >>>>>>> <name>ceph.auth.keyfile</name> >>>>>>> <value>/etc/hadoop/conf/admin.secret</value> >>>>>>> </property> >>>>>>> >>>>>>> <property> >>>>>>> <name>fs.ceph.impl</name> >>>>>>> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> >>>>>>> </property> >>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>> >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |