Hi, >From the doc it looks like for the default cephfs-hadoop driver, Hadoop 2.x is not supported yet. You may need to get a newer hadoop-cephfs.jar if you need to use YARN? http://docs.ceph.com/docs/master/cephfs/hadoop/ https://github.com/GregBowyer/cephfs-hadoop Sincerely, Yuan On Mon, Oct 12, 2015 at 1:58 PM, Fulin Sun <sunfl@xxxxxxxxxxxxxxxx> wrote: > Thanks so much for kindly advice. This is my fault. > > I resolved the problem and the root cause is that I misconfigured > HADOOP_CLASSPATH, so sorry for confusing and troubling. > > But then I am trying to use hadoop yarn to do terasort benckmark test based > on cephfs. New exception message occurs as : > > Does this mean that I cannot use this ceph-hadoop plugin over the hadoop > version? Hadoop version is : 2.7.1 release, Ceph version is : 0.94.3 > > Thanks again for moving this thread. > > Best, > Sun. > > 15/10/12 11:08:35 INFO client.RMProxy: Connecting to ResourceManager at > /172.16.33.18:8032 > 15/10/12 11:08:35 INFO mapreduce.Cluster: Failed to use > org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: > java.lang.NoSuchMethodException: > org.apache.hadoop.fs.ceph.CephFS.<init>(java.net.URI, > org.apache.hadoop.conf.Configuration) > java.io.IOException: Cannot initialize Cluster. Please check your > configuration for mapreduce.framework.name and the correspond server > addresses. > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) > at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) > at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1260) > at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1256) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) > at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > ________________________________ > ________________________________ > > > From: Paul Evans > Date: 2015-10-12 11:10 > To: Fulin Sun > Subject: Re: cephfs replace hdfs problem > I don’t think there are many of us that have attempted what you are trying > to do… that’s the most likely reason the list is quiet. > You may need to be patient, and possibly provide updates (if you have any) > to keep the issue in front of people. > Best of luck... > -- > Paul > > On Oct 11, 2015, at 7:03 PM, Fulin Sun <sunfl@xxxxxxxxxxxxxxxx> wrote: > > sign... > I had to say that I have not received any reponse from this mailing list... > > ________________________________ > ________________________________ > > From: Fulin Sun > Date: 2015-10-10 17:27 > To: ceph-users > Subject: cephfs replace hdfs problem > Hi there, > > I configured hadoop-cephfs plugin and try to use cephfs as a replacement. I > had sucessfully configured > > hadoop-env.sh with setting the HADOOP_CLASSPATH for hadoop-cephfs.jar > > But when I run hadoop fs -ls /, I got the following exception. Looks like it > cannot find the actual jar for both > hadoop-cephfs.jar and libcephfs-java.jar I placed these two in the > /usr/local/hadoop/lib directory and edited > the hadoop classpath in hadoop-env.sh > > How could this issue be ? > > Thanks anyone for kind response. > > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.ceph.CephFileSystem not found > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) > at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) > at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) > at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.ceph.CephFileSystem not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) > ... 18 more > > ________________________________ > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |