Am 18.04.2015 um 07:24 schrieb Alexandre DERUMIER <aderumier@xxxxxxxxx>: >>> any idea whether this might be the tcmalloc bug? > > I still don't known if centos/redhat packages have also the bug or not. > gperftools.x86_64 2.1-1.el7 >From the version number it looks buggy. I'm really interested what fixed the issue for you. > > > > > ----- Mail original ----- > De: "Stefan Priebe" <s.priebe@xxxxxxxxxxxx> > À: "aderumier" <aderumier@xxxxxxxxx>, "Mark Nelson" <mnelson@xxxxxxxxxx>, "ceph-users" <ceph-users@xxxxxxxxxxxxxx> > Envoyé: Vendredi 17 Avril 2015 20:57:42 > Objet: Re: full ssd setup preliminary hammer bench > >> Am 17.04.2015 um 17:37 schrieb Alexandre DERUMIER: >> Hi Mark, >> >> I finally got my hardware for my production full ssd cluster. >> >> Here a first preliminary bench. (1osd). >> >> I got around 45K iops with randread 4K with a small 10GB rbd volume >> >> >> I'm pretty happy because I don't see anymore huge cpu difference between krbd && lirbd. >> In my previous bench I was using debian wheezy as client, >> now it's a centos 7.1, so maybe something is different (glibc,...). > > any idea whether this might be the tcmalloc bug? > >> >> I'm planning to do big benchmark centos vs ubuntu vs debian, client && server, to compare. >> I have 18 osd ssd for the benchmarks. >> >> >> >> >> >> >> >> results : rand 4K : 1 osd >> ------------------------- >> >> fio + librbd: >> ------------ >> iops: 45.1K >> >> clat percentiles (usec): >> | 1.00th=[ 358], 5.00th=[ 406], 10.00th=[ 446], 20.00th=[ 556], >> | 30.00th=[ 676], 40.00th=[ 1048], 50.00th=[ 1192], 60.00th=[ 1304], >> | 70.00th=[ 1400], 80.00th=[ 1496], 90.00th=[ 1624], 95.00th=[ 1720], >> | 99.00th=[ 1880], 99.50th=[ 1928], 99.90th=[ 2064], 99.95th=[ 2128], >> | 99.99th=[ 2512] >> >> cpu server : 89.1 iddle >> cpu client : 92,5 idle >> >> fio + krbd >> ---------- >> iops:47.5K >> >> clat percentiles (usec): >> | 1.00th=[ 620], 5.00th=[ 636], 10.00th=[ 644], 20.00th=[ 652], >> | 30.00th=[ 668], 40.00th=[ 676], 50.00th=[ 684], 60.00th=[ 692], >> | 70.00th=[ 708], 80.00th=[ 724], 90.00th=[ 756], 95.00th=[ 820], >> | 99.00th=[ 1004], 99.50th=[ 1032], 99.90th=[ 1144], 99.95th=[ 1448], >> | 99.99th=[ 2224] >> >> cpu server : 92.4 idle >> cpu client : 96,8 idle >> >> >> >> >> hardware (ceph node && client node): >> ----------------------------------- >> ceph : hammer >> os : centos 7.1 >> 2 x 10cores Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz >> 64GB ram >> 2 x intel s3700 100GB : raid1: os + monitor >> 6 x intel s3500 160GB : osds >> 2x10gb mellanox connect-x3 (lacp) >> >> network >> ------- >> mellanox sx1012 with breakout cables (10GB) >> >> >> centos tunning: >> --------------- >> -noop scheduler >> -tune-adm profile latency-performance >> >> ceph.conf >> --------- >> auth_cluster_required = cephx >> auth_service_required = cephx >> auth_client_required = cephx >> filestore_xattr_use_omap = true >> >> >> osd pool default min size = 1 >> >> debug lockdep = 0/0 >> debug context = 0/0 >> debug crush = 0/0 >> debug buffer = 0/0 >> debug timer = 0/0 >> debug journaler = 0/0 >> debug osd = 0/0 >> debug optracker = 0/0 >> debug objclass = 0/0 >> debug filestore = 0/0 >> debug journal = 0/0 >> debug ms = 0/0 >> debug monc = 0/0 >> debug tp = 0/0 >> debug auth = 0/0 >> debug finisher = 0/0 >> debug heartbeatmap = 0/0 >> debug perfcounter = 0/0 >> debug asok = 0/0 >> debug throttle = 0/0 >> >> osd_op_threads = 5 >> filestore_op_threads = 4 >> >> >> osd_op_num_threads_per_shard = 1 >> osd_op_num_shards = 10 >> filestore_fd_cache_size = 64 >> filestore_fd_cache_shards = 32 >> ms_nocrc = true >> ms_dispatch_throttle_bytes = 0 >> >> cephx sign messages = false >> cephx require signatures = false >> >> [client] >> rbd_cache = false >> >> >> >> >> >> rand 4K : rbd volume size: 10GB (data in osd node buffer - no access to disk) >> ------------------------------------------------------------------------------ >> fio + librbd >> ------------ >> [global] >> ioengine=rbd >> clientname=admin >> pool=pooltest >> rbdname=rbdtest >> invalidate=0 >> rw=randread >> direct=1 >> bs=4k >> numjobs=2 >> group_reporting=1 >> iodepth=32 >> >> >> >> fio + krbd >> ----------- >> [global] >> ioengine=aio >> invalidate=1 # mandatory >> rw=randread >> bs=4K >> direct=1 >> numjobs=2 >> group_reporting=1 >> size=10G >> >> iodepth=32 >> filename=/dev/rbd0 (noop scheduler) >> >> >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |