Re: Current state of the Ceph-Hadoop integration

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for all the help, guys!

Let me give back to community by listing all the minimal steps I needed to make it work on current versions of the software.

Ubuntu 12.04.2 LTS
ceph 0.61.2-1precise
Hadoop 1.1.2

1. Install additional packages:
libcephfs-java
libcephfs-jni

2. Download http://ceph.com/download/hadoop-cephfs.jar

3. In hadoop-env.sh
export HADOOP_CLASSPATH=/path/to/hadoop-cephfs.jar:/usr/share/java/libcephfs.jar
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"

4. Symink JNI library
cd $HADOOP_HOME/lib/native/Linux-amd64-64 (or Linux-i386-32 on 32-bit platform)
ln -s /usr/lib/jni/libcephfs_jni.so .

I have tried adding -Djava.library.path=/usr/lib/jni to HADOOP_OPTS in hadoop-env.sh and exporting LD_LIBRARY_PATH=/usr/lib/jni in hadoop-env.sh, but it didn't work for me.  I'd love to hear about a more elegant method of making Hadoop Ceph JNI aware w/o symlinking.

5. core-site.xml needed all 3 properties below
fs.default.name=ceph:///
fs.ceph.impl=org.apache.hadoop.fs.ceph.CephFileSystem
ceph.conf.file=/etc/ceph/ceph.conf

Cheers,
Ilja


This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, please note that any review, dissemination, disclosure, alteration, printing, circulation, retention or transmission of this e-mail and/or any file or attachment transmitted with it, is prohibited and may be unlawful. If you have received this e-mail or any file or attachment transmitted with it in error please notify postmaster@xxxxxxxxxx. Although Openet has taken reasonable precautions to ensure no viruses are present in this email, we cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux