Re: how to build libradosstriper

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Guys, 

The striping seems to be slightly better than non-striping write, given that my storage is configured with 4OSS, and 48OSD. each OSD is 8+2 Raid 6 of 24TB capacity. 
But still the performance is around 100MB/sec. On a single haswell core, I'm able to get 1GB/sec with dd buffered IO. 

My questions are: 
1. Does librados use direct IO? 
2. Is there option to leverage page cache?
3. What is the strategy to get the peak bandwidth on a rados object store?
4. Can I use MPI for parallel IO?

Best,
Jialin
NERSC

On Wed, May 30, 2018 at 11:08 AM, Jialin Liu <jalnliu@xxxxxxx> wrote:
Thanks Kefu. 

Best,
Jialin
NERSC

On Tue, May 29, 2018 at 11:52 PM, kefu chai <tchaikov@xxxxxxxxx> wrote:
On Wed, May 30, 2018 at 11:53 AM, Jialin Liu <jalnliu@xxxxxxx> wrote:
> Hi Brad,
>
> You are correct. the librados.so has the symbol but what I copied is a wrong
> file.
> Now I can test the striper api with the previous C example and this cpp
> example:
> http://mrbojangles3.github.io/ceph/systems/striping/alignment/2017/05/28/Ceph-Stripe/
> Both are working, but I haven't got any performance number yet.
> ps. I also found that the stripe unit must be set before object_size
> properly.
>
> Thanks much.
>
> Best,
> Jialin
> NERSC
>
>
> On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:
>>
>> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu <jalnliu@xxxxxxx> wrote:
>> > Thanks Brad,
>> > I run nm on those .so file, it prints 'no symbol'
>>
>> OK, well you need to link to a library that exports that symbol (has
>> it defined in its Text section). I suspect you'll find it is defined
>> in libceph-common.so so try linking to that explicitly.

please note, libceph-common does not exist in jewel (i.e. v10.2.10)
yet. it was introduce in luminous. and libceph-common is not supposed
to be linked against by rados application directly. it is an internal
library and is linked by librados, libradosstriper and other rados
tools/daemons.

>>
>> >
>> > Then with ldd librados.so, I don't see the libceph-common.so, instead:
>> >>
>> >>         jialin@cori12: ldd librados.so
>> >> linux-vdso.so.1 (0x00002aaaaaacf000)
>> >> libboost_thread-mt.so.1.53.0
>> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x00002aaaad24d000)
>> >> libboost_system-mt.so.1.53.0 =>
>> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x00002aaaad464000)
>> >> libssl3.so => /usr/lib64/libssl3.so (0x00002aaaad668000)
>> >> libsmime3.so => /usr/lib64/libsmime3.so (0x00002aaaad8b4000)
>> >> libnss3.so => /usr/lib64/libnss3.so (0x00002aaaadadb000)
>> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x00002aaaaddfe000)
>> >> libplds4.so => /usr/lib64/libplds4.so (0x00002aaaae02c000)
>> >> libplc4.so => /usr/lib64/libplc4.so (0x00002aaaae230000)
>> >> libnspr4.so => /usr/lib64/libnspr4.so (0x00002aaaae435000)
>> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaae673000)
>> >> libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaae890000)
>> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00002aaaaea94000)
>> >> librt.so.1 => /lib64/librt.so.1 (0x00002aaaaec99000)
>> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aaaaeea1000)
>> >> libm.so.6 => /lib64/libm.so.6 (0x00002aaaaf22a000)
>> >> libc.so.6 => /lib64/libc.so.6 (0x00002aaaaf527000)
>> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaaf8c8000)
>> >> libz.so.1 => /lib64/libz.so.1 (0x00002aaaafadf000)
>> >> /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
>>
>> Things were obviously different in jewel but the concept remains the
>> same. For each external symbol your binary requires you need to link
>> to a library that exports that symbol so finding the library that
>> exports the symbols in the error message should resolve the error.
>>
>> >
>> >
>> > I also found this thread: http://tracker.ceph.com/issues/14788
>> > which looks similar to the error I run into, and that thread mentioned
>> > the
>> > version between the headers file and the .so file, I think in my case
>> > both
>> > of them are v10.2.10, which should ok.
>> >
>> > Best,
>> > Jialin,
>> >
>> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard <bhubbard@xxxxxxxxxx>
>> > wrote:
>> >>
>> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu <jalnliu@xxxxxxx> wrote:
>> >> > Hi,
>> >> > I'm trying to use the libradosstriper api, but having some trouble in
>> >> > linking to lradosstriper. I copied only the `required' libraries from
>> >> > an
>> >> > pre-installed ceph (10.2.10), and put them under my local directory
>> >> > /rados_install/lib and rados_install/include, on a linux machine.
>> >> >
>> >> > /rados_install/lib:
>> >> >>
>> >> >> libboost_date_time.so.1.53.0      libboost_system.so.1.53.0
>> >> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
>> >> >> libboost_filesystem.so.1.53.0     libcephfs.so.1.0.0
>> >> >> libboost_iostreams-mt.so.1.53.0
>> >> >> libboost_iostreams.so.1.53.0
>> >> >> libboost_random-mt.so.1.53.0      librados.so.2.0.0
>> >> >> libboost_regex.so.1.53.0          librados_tp.so.2.0.0
>> >> >> libboost_system-mt.so.1.53.0      libradosstriper.so.1.0.0
>> >> >> libcephfs.so   librados.so  libradosstriper.so
>> >> >
>> >> >
>> >> > /rados_install/include:
>> >> >>
>> >> >> |-- rados
>> >> >> |   |-- buffer.h -> ../buffer.h
>> >> >> |   |-- buffer_fwd.h -> ../buffer_fwd.h
>> >> >> |   |-- crc32c.h -> ../crc32c.h
>> >> >> |   |-- inline_memory.h -> ../inline_memory.h
>> >> >> |   |-- librados.h
>> >> >> |   |-- librados.hpp
>> >> >> |   |-- librgw.h
>> >> >> |   |-- memory.h -> ../memory.h
>> >> >> |   |-- objclass.h
>> >> >> |   |-- page.h -> ../page.h
>> >> >> |   |-- rados_types.h
>> >> >> |   |-- rados_types.hpp
>> >> >> |   `-- rgw_file.h
>> >> >> `-- radosstriper
>> >> >>     |-- libradosstriper.h
>> >> >>     `-- libradosstriper.hpp
>> >> >
>> >> >
>> >> > The test code is simple, as attached, and I compiled it with dynamic
>> >> > linking:
>> >> >>
>> >> >> cc -L/rados_install/lib -I/rados_install/include -o librados_test
>> >> >> librados_test.c -lrados -lradosstriper
>> >> >
>> >> >
>> >> > But got the following message, can anyone plz advise?
>> >> >
>> >> >>
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `librados::IoCtx::remove(std::string const&, int)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*,
>> >> >> unsigned
>> >> >> int)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::list::is_provided_buffer(char const*) const'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::list::append(ceph::buffer::ptr&&)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*,
>> >> >> unsigned
>> >> >> int,
>> >> >> std::_List_iterator<ceph::buffer::ptr>, unsigned int)'
>> >> >> /rados_install/lib/libradosstriper.so: undefined reference to
>> >> >> `ceph::buffer::list::list(ceph::buffer::list&&)'
>> >>
>> >> Try doing the following in your lib directory.
>> >>
>> >> $ nm -A -C *.so|grep is_provided_buffer
>> >> libceph-common.so:0000000000223df0 T
>> >> ceph::buffer::list::is_provided_buffer(char const*) const
>> >> librados.so:00000000000bfef0 T
>> >> ceph::buffer::list::is_provided_buffer(char const*) const
>> >> libradosstriper.so:                 U
>> >> ceph::buffer::list::is_provided_buffer(char const*) const
>> >>
>> >> 'ldd librados.so' shows it links to libceph-common.so so I suspect you
>> >> may need that library.
>> >>
>> >> Don't forget that you need to tell the linker where to find these
>> >> libraries at runtime so you will need to pass something like
>> >> '-Wl,-rpath=/rados_install/lib' to the compiler so it finds the
>> >> correct versions of the libraries (you can check which ones it will
>> >> find with 'ldd librados_test'.
>> >>
>> >> HTH
>> >>
>> >> >
>> >> >
>> >> > Best,
>> >> > Jialin
>> >> > NERSC
>> >> >
>> >> > _______________________________________________
>> >> > ceph-users mailing list
>> >> > ceph-users@xxxxxxxxxxxxxx
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Cheers,
>> >> Brad
>> >
>> >
>>
>>
>>
>> --
>> Cheers,
>> Brad
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



--
Regards
Kefu Chai


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux