Using Hadoop With Cephfs

[log1024 <log1024@xxxxxxxx>]

Hi,

I have a 4-node Ceph cluster(2 mon, 1 mds, 2 osd) and a Hadoop node.

Currently, I'm trying to replace HDFS with CephFS. I followed the instructions in USING HADOOP WITH CEPHFS. But every time I run bin/start-all.sh to run Hadoop, it failed with:

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-namenode-ceph-srv1.out

localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-ceph-srv1.out

localhost: Exception in thread "IPC Client (47) connection to /172.29.84.56:6789 from hduser" java.lang.RuntimeException: readObject can't find class

localhost: at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:185)

localhost: at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)

localhost: at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:851)

localhost: at org.apache.hadoop.ipc.Client$Connection.run(Client.java:786)

localhost: Caused by: java.lang.ClassNotFoundException:

localhost: at java.lang.Class.forName0(Native Method)

localhost: at java.lang.Class.forName(Class.java:249)

localhost: at org.apache.hadoop.conf.Configu ration.getClassByName(Configuration.java:802)

localhost: at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)

My core-site.xml:

<configuration>

<property>

 <name>ceph.conf.file</name>

 <value>/etc/ceph/ceph.conf</value>

</property>

<property>

 <name>fs.default.name</name>

& nbsp;<value>ceph://172.29.84.56:6789/</value>

</property>

<property>

<name>ceph.mon.address</name>

<value>172.29.84.56:6789</value>

</property>

<property>

<name>ceph.auth.keyring</name>

<value>/etc/ceph/ceph.client.admin.keyring</value>

</property>

<property>

<name>ceph.data.pools</name>

<value>hadoop1</value>

</property>

</configuration>

Here is my ceph -s output:

ceph -s

  cluster 942afa43-9a92-434b-9dfa-e893d4e5d565

   health HEALTH_WARN 16 pgs degraded; 16 pgs stuck unclean; recovery 505/1719 degraded (29.378%); clock skew detected on mon.ceph-srv3

   monmap e1: 2 mons at {ceph-srv2=172.29.84.56:6789/0,ceph-srv3=172.29.84.57:6789/0}, election epoch 12, quorum 0,1 ceph-srv2,ceph-srv3

   osdmap e52: 2 osds: 2 up, 2 in

    pgmap v33521: 372 pgs: 356 active+clean, 16 active+degraded; 4097 MB data, 8277 MB used, 1384 GB / 1392 GB avail; 505/1719 degraded (29.378%)

   mdsmap e17: 1/1/1 up {0=ceph-srv2=up:active}

Can anyone show me how to use hadoop with cephfs correctly?

Thanks,

Kai