Re: cephfs replace hdfs problem

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi, Yuan

Thanks a lot for your guide. I cloned the master branch and packaged a new jar and put it to the HADOOP_CLASSPATH 

and yarn.application.classpath. Now I can successfully run teragen and terasort benckmark test over cephfs . 

Thanks very much for advice.

Best,
Sun.


From: ZHOU Yuan
Date: 2015-10-12 15:30
To: Fulin Sun
CC: Paul Evans; ceph-users
Subject: Re: cephfs replace hdfs problem
Hi,
 
From the doc it looks like for the default cephfs-hadoop driver,
Hadoop 2.x is not supported yet. You may need to get a newer
hadoop-cephfs.jar if you need to use YARN?
 
http://docs.ceph.com/docs/master/cephfs/hadoop/
 
https://github.com/GregBowyer/cephfs-hadoop
 
Sincerely, Yuan
 
 
On Mon, Oct 12, 2015 at 1:58 PM, Fulin Sun <sunfl@xxxxxxxxxxxxxxxx> wrote:
> Thanks so much for kindly advice. This is my fault.
>
> I resolved the problem and the root cause is that I misconfigured
> HADOOP_CLASSPATH, so sorry for confusing and troubling.
>
> But then I am trying to use hadoop yarn to do terasort benckmark test based
> on cephfs. New exception message occurs as :
>
> Does this mean that I cannot use this ceph-hadoop plugin over the hadoop
> version? Hadoop version is : 2.7.1 release, Ceph version is : 0.94.3
>
> Thanks again for moving this thread.
>
> Best,
> Sun.
>
> 15/10/12 11:08:35 INFO client.RMProxy: Connecting to ResourceManager at
> /172.16.33.18:8032
> 15/10/12 11:08:35 INFO mapreduce.Cluster: Failed to use
> org.apache.hadoop.mapred.YarnClientProtocolProvider due to error:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ceph.CephFS.<init>(java.net.URI,
> org.apache.hadoop.conf.Configuration)
> java.io.IOException: Cannot initialize Cluster. Please check your
> configuration for mapreduce.framework.name and the correspond server
> addresses.
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1260)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1256)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
> at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> ________________________________
> ________________________________
>
>
> From: Paul Evans
> Date: 2015-10-12 11:10
> To: Fulin Sun
> Subject: Re: cephfs replace hdfs problem
> I don’t think there are many of us that have attempted what you are trying
> to do… that’s the most likely reason the list is quiet.
> You may need to be patient, and possibly provide updates (if you have any)
> to keep the issue in front of people.
> Best of luck...
> --
> Paul
>
> On Oct 11, 2015, at 7:03 PM, Fulin Sun <sunfl@xxxxxxxxxxxxxxxx> wrote:
>
> sign...
> I had to say that I have not received any reponse from this mailing list...
>
> ________________________________
> ________________________________
>
> From: Fulin Sun
> Date: 2015-10-10 17:27
> To: ceph-users
> Subject: cephfs replace hdfs problem
> Hi there,
>
> I configured hadoop-cephfs plugin and try to use cephfs as a replacement. I
> had sucessfully configured
>
> hadoop-env.sh with setting the HADOOP_CLASSPATH for hadoop-cephfs.jar
>
> But when I run hadoop fs -ls /, I got the following exception. Looks like it
> cannot find the actual jar for both
> hadoop-cephfs.jar  and  libcephfs-java.jar I placed these two in the
> /usr/local/hadoop/lib directory and edited
> the hadoop classpath in hadoop-env.sh
>
> How could this issue be ?
>
> Thanks anyone for kind response.
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.ceph.CephFileSystem not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
> at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.ceph.CephFileSystem not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
> ... 18 more
>
> ________________________________
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
 
 
 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux