回复: About cephfs with hadoop

["Fulin Sun" <sunfl@xxxxxxxxxxxxxxxx>]

I changed the source code and comment the namenode schema check in NameNode.java 

then I rebuild hadoop 2.7.1 with maven and replace the hadoop-hdfs.jar 

When I am trying to restart hadoop, namenode still failed with the following exception : 

Does this mean that I cannot define the ceph host:port ？ Really confused about this.

Please offer some guide here.

java.net.BindException: Problem binding to [172.16.50.18:6789] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
at org.apache.hadoop.ipc.Server.bind(Server.java:425)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:574)
at org.apache.hadoop.ipc.Server.<init>(Server.java:2215)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:938)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:534)
at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:783)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:344)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:673)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:463)
at sun.nio.ch.Net.bind(Net.java:455)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:408)
... 13 more

---

---

 

发件人： Fulin Sun

发送时间： 2015-09-23 14:59

收件人： ceph-users

主题： About cephfs with hadoop

Hi, all

I am trying to use cephfs as a replacement drop-in for hadoop hdfs. Maily configure steps according to doc here : 

http://docs.ceph.com/docs/master/cephfs/hadoop/ 

I am using a 3 node hadoop 2.7.1 cluster. Noting the official doc recommend using 1.1.x stable release, I am not sure

if using 2.x release would cause some unexpected trouble. Anyway, I think I had configured right for cephfs and ceph

storage cluster goes well. 

When I am trying to start the hadoop cluster, namenode failed with the following exception: 

  2015-09-23 14:47:39,600 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

       java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): ceph://172.16.50.18:6789/ is not of scheme 'hdfs'.
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:477)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:461)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:512)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:612)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:632)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)

   

Quite weird here cause the Namenode check seems to NOT ALLOW any schema expect default hdfs. Then how would cephfs-hadoop work actually?

Really need your help here. 

Best,

Sun.

---

---

CertusNet

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com