Re: hadoop gluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



You should submit jobs as a user other than yarn which is also a member of the hadoop group. We usually add a mapred user.

Also check your <Hadoop Home>/etc/hadoop/container-executor.cfg:

yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=yarn
min.user.id=1000
allowed.system.users=mapred

You'll want the mapred UID > 1000 or else adjust the setting in the file.
Regards,
Shubhendu

On 06/08/2015 07:44 AM, 中川智之 wrote:
Hi EveryOne.

I want to use Hadoop 2.x with GlusterFS. To testing, Prepare these
softwares.

<Cnfiguration>
- CentOS 7.1(on VMWare)
- Gluster FS 3.7
- Hadoop 2.x
- Glusterfs-Hadoop plugin 2.3.13

Test Process is this.

<Test Process>
1. Install CentOS 7.1. (3 VM Machine)
  - 1 Machine is GlusterFS Client, and hadoop Namenode
  - 2 Machine is GlusterFS BrickNode, and hadoop Datanode
2. Install GlusterFS 3.7, and cnfigure brick.
3. Mount Gluster volume from client Machine.
4. Install Hadoop 2.x, and HDFS configure.
  * Hadoop installed by root.
5. Starting and testing HDFS and Mapreduce.
6. Configure Hadoop 2.0 with GlusterFS

Finish and Succcess Test 5, and I started Test 6 with this Note.

<Note>
http://www.gluster.org/community/documentation/index.php/Hadoop
https://forge.gluster.org/hadoop/pages/Configuration

But I'm not entirely the way. 
- How to use glusterfs-hadoop-2.3.13.jar ?
- Only put glusterfs-hadoop-2.3.13.jar and edit core-site.xml, Hadoop can
use on GlusterFS?
- If Finished Configuration, Hadoop can access on GlusterFS Volume by hadoop
command ?
 ex) 
 $ bin/hadoop dfs -cat <GlusterVolume>/data
 $ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep
<GlusteVolume>/data <GlusteVolume>/dataout '[a-z.]+'

Please tll me the way of Configuring.
I'm a hadoop Beginner, and I'm not so good at English.





_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux