Re: hadoop namenode not starting due to bindException while deploying hadoop with cephFS

[Ridwan Rashid Noel <ridwan064@xxxxxxxxx>]

Hi Greg,

Thank you for your response. I have understood that I should be starting only the mapred daemons when using cephFS instead of HDFS. I have fixed that and trying to run hadoop wordcount job using this instruction:

bin/hadoop jar hadoop*examples*.jar wordcount /tmp/wc-input /tmp/wc-output

but I am getting this error
 
15/03/26 02:54:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/03/26 02:54:35 INFO input.FileInputFormat: Total input paths to process : 1
15/03/26 02:54:35 WARN snappy.LoadSnappy: Snappy native library not loaded
15/03/26 02:54:35 INFO mapred.JobClient: Running job: job_201503260253_0001
15/03/26 02:54:36 INFO mapred.JobClient:  map 0% reduce 0%
15/03/26 02:54:36 INFO mapred.JobClient: Task Id : attempt_201503260253_0001_m_000021_0, Status : FAILED
Error initializing attempt_201503260253_0001_m_000021_0:
java.io.FileNotFoundException: File file:/tmp/hadoop-ceph/mapred/system/job_201503260253_0001/jobToken does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4445)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1272)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
        at java.lang.Thread.run(Thread.java:745)

.....   

I have used the core-site.xml configurations as mentioned in http://ceph.com/docs/master/cephfs/hadoop/

Please tell me how can this problem be solved?

Regards,

Ridwan Rashid Noel

Doctoral Student,

Department of Computer Science,

University of Texas at San Antonio

Contact# 210-773-9966

On Fri, Mar 20, 2015 at 4:04 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:

On Fri, Mar 20, 2015 at 1:05 PM, Ridwan Rashid <ridwan064@xxxxxxxxx> wrote:
> Gregory Farnum <greg@...> writes:

>
>>
>> On Thu, Mar 19, 2015 at 5:57 PM, Ridwan Rashid <ridwan064@...> wrote:
>> > Hi,
>> >
>> > I have a 5 node ceph(v0.87) cluster and am trying to deploy hadoop with
>> > cephFS. I have installed hadoop-1.1.1 in the nodes and changed the
>> > conf/core-site.xml file according to the ceph documentation
>> > http://ceph.com/docs/master/cephfs/hadoop/ but after changing the file the
>> > namenode is not starting (namenode can be formatted) but the other
>> > services(datanode, jobtracker, tasktracker) are running in hadoop.
>> >
>> > The default hadoop works fine but when I change the core-site.xml file as
>> > above I get the following bindException as can be seen from the namenode
> log:
>> >
>> >
>> > 2015-03-19 01:37:31,436 ERROR
>> > org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
>> > Problem binding to node1/10.242.144.225:6789 : Cannot assign requested
> address
>> >
>> >
>> > I have one monitor for the ceph cluster (node1/10.242.144.225) and I
>> > included in the core-site.xml file ceph://10.242.144.225:6789 as the value
>> > of fs.default.name. The 6789 port is the default port being used by the
>> > monitor node of ceph, so that may be the reason for the bindException but
>> > the ceph documentation mentions that it should be included like this in the
>> > core-site.xml file. It would be really helpful to get some pointers to where
>> > I am doing wrong in the setup.
>>
>> I'm a bit confused. The NameNode is only used by HDFS, and so
>> shouldn't be running at all if you're using CephFS. Nor do I have any
>> idea why you've changed anything in a way that tells the NameNode to
>> bind to the monitor's IP address; none of the instructions that I see
>> can do that, and they certainly shouldn't be.
>> -Greg
>>
>

> Hi Greg,
>
> I want to run a hadoop job (e.g. terasort) and want to use cephFS instead of
> HDFS. In "Using Hadoop with cephFS" documentation in
> http://ceph.com/docs/master/cephfs/hadoop/ if you look into the Hadoop
> configuration section, the first property fs.default.name has to be set as
> the ceph URI and in the notes it's mentioned as ceph://[monaddr:port]/. My
> core-site.xml of hadoop conf looks like this
>
> <configuration>
>
> <property>
>     <name>fs.default.name</name>
>     <value>ceph://10.242.144.225:6789</value>
> </property>

Yeah, that all makes sense. But I don't understand why or how you're
starting up a NameNode at all, nor what config values it's drawing
from to try and bind to that port. The NameNode is the problem because
it shouldn't even be invoked.
-Greg

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com