Re: how to use hadoop when system is CephFS

log1024 <log1024@xxxxxxxx> · Sat, 19 Oct 2013 01:33:38 +0800 (CST)

Hi Peng, Noah
Thanks for your help. Now the hadoop & ceph are working now!
It takes 11m59.464s to finish MrBenchmark x 50 times.

Cheers!
Kai

At 2013-10-18 06:27:56,"Noah Watkins" <noah.watkins@xxxxxxxxxxx> wrote:
 Kai,
It looks like libcephfs-java (the CephFS Java bindings) are not in your classpath. Where did you install them?

-Noah

On Thu, Oct 17, 2013 at 11:30 PM, log1024 <log1024@xxxxxxxx> wrote:

Hi PengThe conf in my cluster is almost the same with yours, but when i run
#bin/hadoop fs -ls /
It failed with:
Exception in thread "main" java.lang.NoClassDefFoundError: com/ceph/fs/CephFileAlreadyExistsException
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:249)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1405)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1429)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
	at org.apache.hadoop.fs.FsShell.init(FsShell.java:82)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:1745)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: java.lang.ClassNotFoundException: com.ceph.fs.CephFileAlreadyExistsException
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 14 more

Is there anyone came across this exception before?  How to deal with it?

Thanks,
Kai

At 2013-10-17 19:17:05,"鹏" <wkp4666@xxxxxxx> wrote:

Hi all !
    I think I suceed using cephFS with hadoop, but I still face lost question for  that  I am a newer in hadoop and ceph!
    first, my ceph version is 0.62. java is 1.6.0_45 .hadoop is 1.1.2  I think suceed because:

      # hadoop fs -ls /
      drwxrwxrwx - root 0 2013-10-16 10:57 /libcephfs_junit
   but, when I use the commond  hadoop dfsadmin -report, nothing printed on the terminal。

  []   #hadoop dfsadmin -report
  []  #
   and then , I try  some mapreduce's work!
   #hadoop jar hadoop-examples-1.1.2.jar wordcount /tmp/wordcount/word.txt /tmp/wordcount/out
   .........INFO ceph.CephFileSystem:...........

...
   ..........WARN Snappy.Load Dnappy: Snappy native library not loaded.
....
            INFO:map 0% reduce 0%
 and  Nothing was print on!  I wait for it  more then 10 Minute and the word.txt content less the 10 words.

by the way, the jps restut is:
#jps
2913     secondaryNameNode
17888   TaskTracker
4581 JPS
2974  JobTracker

and my hadoop  conf is 
core-site.xml
hadoop.tmp.dir = /usr/hadoop/tmp
fs.ceph.impl =org.apache.hadoop.fs.ceph.CephFileSystem

fs.default.name = ceph://192.168.58.129/
ceph.conf.file = /etc/ceph/ceph.conf

mapres-site.xml
mapred.job.tracket=http://192.168.58.132:9001

ndfs-site.xml
dfs.replication=1

thanks
peng

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com