Looks like this is a configuration issue that has popped up with other 3rd party file systems in Hadoop 2.x with YARN. http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201306.mbox/%3C1023550423.3137743.1371825668412.JavaMail.root@xxxxxxxxxx%3E Says use this: <property> <name>fs.AbstractFileSystem.glusterfs.impl</name> <value>org.apache.hadoop.fs.glusterfs.GlusterFS</value> </property> Apparently rather than the `fs.ceph.impl` property in 2.x On Wed, Mar 19, 2014 at 9:06 AM, Gurvinder Singh <gurvindersinghdahiya@xxxxxxxxx> wrote: > On 03/19/2014 04:50 PM, Noah Watkins wrote: >> Since `hadoop -fs ls /` seems to work on your local node, can you >> verify that (1) it is in fact listing the contents of CephFS, and (2) >> that on your worker nodes where the error is occuring that the >> relevant dependencies (naming the Ceph hadoop bindings) are installed >> and in the classpath? > 1) yes it is listing the cephfs contents. > 2) Currently the setup is to start the yarn manager and mapreduce > history server on the same node and then setup the worker nodes. Yarn > starts up fine, but the mapreduce history server gives me the error. The > dependency lib are loaded in the mapreduce history server startup. Here > is the link to start up message which shows that the libcephfs and > cephfs-hadoop jar are loaded. > > http://pastebin.com/wED7nMMT > > I am wondering if this property value is correct > > <property> > <name>fs.ceph.impl</name> > <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> > </property> > > or should I set it to something for hadoop 2.x I have tried setting it up as > > org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as > > # hadoop fs -ls / > -ls: Fatal internal error > java.lang.RuntimeException: java.lang.NoSuchMethodException: > org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) > at > org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) > at > org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) > at > org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) > at org.apache.hadoop.fs.shell.Command.run(Command.java:154) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) > Caused by: java.lang.NoSuchMethodException: > org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() > at java.lang.Class.getConstructor0(Class.java:2800) > at java.lang.Class.getDeclaredConstructor(Class.java:2043) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) > ... 17 more > > Regards, > Gurvinder > >> >> The error itself looks like a missing dependency, but that exception >> being thrown might also be tirggered by other problems while loading >> the bindings. >> >> On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh >> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>> On 03/19/2014 03:51 PM, Noah Watkins wrote: >>>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh >>>> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>>>> Hi, >>>>> >>>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >>>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >>>>> support. I am able to run the command such as >>>> >>>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1? >>>> >>> Yes. >>>> Can you paste your core-site.xml? >>> >>> Below is the relevant part for ceph conf >>> >>> <property> >>> <name>fs.defaultFS</name> >>> <value>ceph://xxxxxxxxxxxx:6789/</value> >>> </property> >>> >>> <property> >>> <name>ceph.conf.file</name> >>> <value>/etc/ceph/ceph.conf</value> >>> </property> >>> >>> <property> >>> <name>ceph.auth.id</name> >>> <value>admin</value> >>> </property> >>> >>> <property> >>> <name>ceph.auth.keyfile</name> >>> <value>/etc/hadoop/conf/admin.secret</value> >>> </property> >>> >>> <property> >>> <name>fs.ceph.impl</name> >>> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> >>> </property> >>> >>> >>>> >>> > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |