Hello ceph developers and users,
some time ago, I posted here a question regarding very different
performance for two volumes in one pool (backed by SSD drives).
After some examination, I probably got to the root of the problem..
When I create fresh volume (ie rbd create --image-format 2 --size 51200 ssd/test)
and run random io fio benchmark
fio --randrepeat=1 --ioengine=rbd --direct=1 --gtod_reduce=1 --name=test --pool=ssd3r --rbdname=${rbdname} --invalidate=1 --bs=4k --iodepth=64 --readwrite=randread
I get very nice performance of up to 200k IOPS. However once the volume is
written to (ie when I map it using rbd map and dd whole volume with some random data),
and repeat the benchmark, random performance drops to ~23k IOPS.
This leads me to conjecture that for unwritten (sparse) volumes, read
is just a noop, simply returning zeroes without really having to read
data from physical storage, and thus showing nice performance, but once
the volume is written, performance drops due to need to physically read the
data, right?
However I'm a bit unhappy about the performance drop, the pool is backed
by 3 SSD drives (each having random io performance of 100k iops) on three
nodes, and object size is set to 3. Cluster is completely idle, nodes
are quad core Xeons E3-1220 v3 @ 3.10GHz, 32GB RAM each, centos 6, kernel 3.18.12,
ceph 0.94.1. I'm using libtcmalloc (I even tried upgrading gperftools-libs to 2.4)
Nodes are connected using 10gb ethernet, with jumbo frames enabled.
I tried tuning following values:
osd_op_threads = 5
filestore_op_threads = 4
osd_op_num_threads_per_shard = 1
osd_op_num_shards = 25
filestore_fd_cache_size = 64
filestore_fd_cache_shards = 32
I don't see anything special in perf:
5.43% [kernel] [k] acpi_processor_ffh_cstate_enter
2.93% libtcmalloc.so.4.2.6 [.] 0x0000000000017d2c
2.45% libpthread-2.12.so [.] pthread_mutex_lock
2.37% libpthread-2.12.so [.] pthread_mutex_unlock
2.33% [kernel] [k] do_raw_spin_lock
2.00% libsoftokn3.so [.] 0x000000000001f455
1.96% [kernel] [k] __switch_to
1.32% [kernel] [k] __schedule
1.24% libstdc++.so.6.0.13 [.] std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char
1.24% libc-2.12.so [.] memcpy
1.19% libtcmalloc.so.4.2.6 [.] operator delete(void*)
1.16% [kernel] [k] __d_lookup_rcu
1.09% libstdc++.so.6.0.13 [.] 0x000000000007d6be
0.93% libstdc++.so.6.0.13 [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long)
0.93% ceph-osd [.] crush_hash32_3
0.85% libc-2.12.so [.] vfprintf
0.84% libc-2.12.so [.] __strlen_sse42
0.80% [kernel] [k] get_futex_key_refs
0.80% libpthread-2.12.so [.] pthread_mutex_trylock
0.78% libtcmalloc.so.4.2.6 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)
0.71% libstdc++.so.6.0.13 [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)
0.68% ceph-osd [.] ceph::log::Log::flush()
0.66% libtcmalloc.so.4.2.6 [.] tc_free
0.63% [kernel] [k] resched_curr
0.63% [kernel] [k] page_fault
0.62% libstdc++.so.6.0.13 [.] std::string::reserve(unsigned long)
I'm running benchmark directly on one of nodes, which I know is not optimal,
but it's still able to give those 200k iops for empty volume, so I guess it
shouldn't be problem..
Another story is random write performance, which is totally poor, but I't like
to deal with read performance first..
so my question is, are those numbers normal? If not, what should I check?
I'll be very grateful for all the hints I could get..
thanks a lot in advance
nik
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava
tel.: +420 591 166 214
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis@xxxxxxxxxxx
-------------------------------------
Attachment:
pgpSSDrw7xawo.pgp
Description: PGP signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
