On 03/19/2014 04:50 PM, Noah Watkins wrote: > Since `hadoop -fs ls /` seems to work on your local node, can you > verify that (1) it is in fact listing the contents of CephFS, and (2) > that on your worker nodes where the error is occuring that the > relevant dependencies (naming the Ceph hadoop bindings) are installed > and in the classpath? 1) yes it is listing the cephfs contents. 2) Currently the setup is to start the yarn manager and mapreduce history server on the same node and then setup the worker nodes. Yarn starts up fine, but the mapreduce history server gives me the error. The dependency lib are loaded in the mapreduce history server startup. Here is the link to start up message which shows that the libcephfs and cephfs-hadoop jar are loaded. http://pastebin.com/wED7nMMT I am wondering if this property value is correct <property> <name>fs.ceph.impl</name> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> </property> or should I set it to something for hadoop 2.x I have tried setting it up as org.apache.hadoop.fs.ceph.CephHadoop2FileSystem, then I get this error as # hadoop fs -ls / -ls: Fatal internal error java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2315) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) at org.apache.hadoop.fs.shell.Command.run(Command.java:154) at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.ceph.CephHadoop2FileSystem.<init>() at java.lang.Class.getConstructor0(Class.java:2800) at java.lang.Class.getDeclaredConstructor(Class.java:2043) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 17 more Regards, Gurvinder > > The error itself looks like a missing dependency, but that exception > being thrown might also be tirggered by other problems while loading > the bindings. > > On Wed, Mar 19, 2014 at 8:43 AM, Gurvinder Singh > <gurvindersinghdahiya@xxxxxxxxx> wrote: >> On 03/19/2014 03:51 PM, Noah Watkins wrote: >>> On Wed, Mar 19, 2014 at 4:28 AM, Gurvinder Singh >>> <gurvindersinghdahiya@xxxxxxxxx> wrote: >>>> Hi, >>>> >>>> I have ceph 0.72.2 running on debian wheezy with cloudera 5.0 beta 2 >>>> hadoop. I have installed the ceph hadoop binding with hadoop 2.x >>>> support. I am able to run the command such as >>> >>> From github.com/noahdesu/cephfs-hadoop patched with the pull request PR#1? >>> >> Yes. >>> Can you paste your core-site.xml? >> >> Below is the relevant part for ceph conf >> >> <property> >> <name>fs.defaultFS</name> >> <value>ceph://xxxxxxxxxxxx:6789/</value> >> </property> >> >> <property> >> <name>ceph.conf.file</name> >> <value>/etc/ceph/ceph.conf</value> >> </property> >> >> <property> >> <name>ceph.auth.id</name> >> <value>admin</value> >> </property> >> >> <property> >> <name>ceph.auth.keyfile</name> >> <value>/etc/hadoop/conf/admin.secret</value> >> </property> >> >> <property> >> <name>fs.ceph.impl</name> >> <value>org.apache.hadoop.fs.ceph.CephFileSystem</value> >> </property> >> >> >>> >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |