Re: troubleshooting ceph performance

Maged Mokhtar <mmokhtar@xxxxxxxxxxx> · Wed, 31 Jan 2018 09:43:03 +0200

On 2018-01-31 08:14, Manuel Sopena Ballesteros wrote:

Dear Ceph community,

I have a very small ceph cluster for testing with this configuration:

·         2x compute nodes each with:
·         dual port of 25 nic
·         2x socket (56 cores with hyperthreading)
·         X10 intel nvme DC P3500 drives
·         512 GB RAM

One of the nodes is also running as a monitor.
Installation has been done using ceph-ansible.

Ceph version: jewel
Storage engine: filestore

Performance test below:

[root@zeus-59 ceph-block-device]# ceph osd pool ls detail
pool 0 'rbd' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 115 flags hashpspool stripe_width 0
pool 1 'images' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 118 flags hashpspool stripe_width 0
        removed_snaps [1~3,7~4]
pool 3 'backups' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 120 flags hashpspool stripe_width 0
pool 4 'vms' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 122 flags hashpspool stripe_width 0
        removed_snaps [1~7]
pool 5 'volumes' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 124 flags hashpspool stripe_width 0
        removed_snaps [1~3]
pool 6 'scbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 126 flags hashpspool stripe_width 0
pool 7 'rbdbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 128 flags hashpspool stripe_width 0
        removed_snaps [1~3]

[root@zeus-59 ceph-block-device]# ceph osd tree
ID WEIGHT   TYPE NAME        UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 36.17371 root default
-2 18.08685     host zeus-58
0  1.80869         osd.0         up  1.00000          1.00000
2  1.80869         osd.2         up  1.00000          1.00000
4  1.80869         osd.4         up  1.00000          1.00000
6  1.80869         osd.6         up  1.00000          1.00000
8  1.80869         osd.8         up  1.00000          1.00000
10  1.80869         osd.10        up  1.00000          1.00000
12  1.80869         osd.12        up  1.00000          1.00000
14  1.80869         osd.14        up  1.00000          1.00000
16  1.80869         osd.16        up  1.00000          1.00000
18  1.80869         osd.18        up  1.00000          1.00000
-3 18.08685     host zeus-59
1  1.80869         osd.1         up  1.00000          1.00000
3  1.80869         osd.3         up  1.00000          1.00000
5  1.80869         osd.5         up  1.00000          1.00000
7  1.80869         osd.7         up  1.00000          1.00000
9  1.80869         osd.9         up  1.00000          1.00000
11  1.80869         osd.11        up  1.00000          1.00000
13  1.80869         osd.13        up  1.00000          1.00000
15  1.80869         osd.15        up  1.00000          1.00000
17  1.80869         osd.17        up  1.00000          1.00000
19  1.80869         osd.19        up  1.00000          1.00000
[root@zeus-59 ceph-block-device]# ceph status
    cluster 8e930b6c-455e-4328-872d-cb9f5c0359ae
     health HEALTH_OK
     monmap e1: 1 mons at {zeus-59=10.0.32.59:6789/0}
            election epoch 3, quorum 0 zeus-59
     osdmap e129: 20 osds: 20 up, 20 in
            flags sortbitwise,require_jewel_osds
      pgmap v1166945: 776 pgs, 7 pools, 1183 GB data, 296 kobjects
            2363 GB used, 34678 GB / 37042 GB avail
                 775 active+clean
                   1 active+clean+scrubbing+deep
[root@zeus-59 ceph-block-device]# rados bench -p scbench 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_zeus-59.localdomain_2844050
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       644       628    2511.4      2512   0.0210273    0.025206
    2      16      1319      1303   2605.49      2700   0.0238678   0.0243974
    3      16      2003      1987   2648.89      2736   0.0201334   0.0240726
    4      16      2669      2653   2652.59      2664   0.0258618   0.0240468
    5      16      3349      3333   2666.01      2720   0.0189464   0.0239484
    6      16      4026      4010   2672.96      2708     0.02215   0.0238954
    7      16      4697      4681   2674.49      2684   0.0217258   0.0238887
    8      16      5358      5342   2670.64      2644   0.0265384   0.0239066
    9      16      6043      6027    2678.3      2740   0.0260798   0.0238637
   10      16      6731      6715   2685.64      2752   0.0174624   0.0237982
Total time run:         10.026091
Total writes made:      6731
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     2685.39
Stddev Bandwidth:       70.0286
Max bandwidth (MB/sec): 2752
Min bandwidth (MB/sec): 2512
Average IOPS:           671
Stddev IOPS:            17
Max IOPS:               688
Min IOPS:               628
Average Latency(s):     0.023819
Stddev Latency(s):      0.00463709
Max latency(s):         0.0594516
Min latency(s):         0.0138556
[root@zeus-59 ceph-block-device]# rados bench -p scbench 10 seq
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15      1150      1135   4498.75      4540   0.0146433   0.0131456
    2      15      2313      2298   4571.38      4652   0.0144489   0.0131564
    3      15      3468      3453   4585.68      4620  0.00975626   0.0131211
    4      15      4663      4648   4633.41      4780   0.0163181   0.0130076
    5      15      5949      5934   4734.49      5144  0.00944718   0.0127327
Total time run:       5.643929
Total reads made:     6731
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   4770.43
Average IOPS          1192
Stddev IOPS:          59
Max IOPS:             1286
Min IOPS:             1135
Average Latency(s):   0.0126349
Max latency(s):       0.0490061
Min latency(s):       0.00613382
[root@zeus-59 ceph-block-device]# rados bench -p scbench 10 rand
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15      1197      1182    4726.8      4728   0.0130331    0.012711
    2      15      2364      2349   4697.02      4668   0.0105971   0.0128123
    3      15      3686      3671   4893.78      5288  0.00906867   0.0123103
    4      15      4994      4979   4978.16      5232  0.00946901    0.012104
    5      15      6302      6287   5028.83      5232   0.0115159   0.0119879
    6      15      7620      7605   5069.28      5272  0.00986636   0.0118935
    7      15      8912      8897   5083.31      5168   0.0106201   0.0118648
    8      15     10185     10170   5084.34      5092   0.0116891   0.0118632
    9      15     11484     11469   5096.68      5196  0.00911787   0.0118354
   10      16     12748     12732   5092.16      5052   0.0111988   0.0118476
Total time run:       10.020135
Total reads made:     12748
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   5088.95
Average IOPS:         1272
Stddev IOPS:          55
Max IOPS:             1322
Min IOPS:             1167
Average Latency(s):   0.0118531
Max latency(s):       0.0441046
Min latency(s):       0.00590162
[root@zeus-59 ceph-block-device]# rbd bench-write image01 --pool=rbdbench
bench-write  io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     56159  56180.51  230115361.66
    2    119975  59998.28  245752967.01
    3    182956  60990.78  249818235.33
    4    244195  61054.17  250077889.88
elapsed:     4  ops:   262144  ops/sec: 60006.56  bytes/sec: 245786880.86
[root@zeus-59 ceph-block-device]#

I am far from a ceph/storage expert but my feeling is that the numbers provided by rbd bench-write are quite poor considering the hardware I am using (please correct me if I am wrong).

I would like to ask for some help from the community in order to dig into this issue and find what is throttling the performance (cpu? Memory? Network configuration? Not enough data nodes? Not enough OSDs per disk? Cpu pinning? Etc.). 

Apologies beforehand as I know this is a quite a broad topic and not easy to give an exact answer but I would like to have some guidance and hope we can make an interesting topic for performance troubleshooting for other people who is learning distributed storage and ceph.

Thank you very much

Manuel Sopena Ballesteros | Systems engineer
Garvan Institute of Medical Research 
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb@xxxxxxxxxxxxx

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

_______________________________________________
 ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Hi Manuel
To get the complete picture, you should run a tool to measure resource load while you are performing the rados benchmark. You could use tools like atop/sysstat/collectl. The most important metrics to watch is cpu % utilization, disk % utilization (% busy), and interface throuput ( or % util if using sysstat ). This will show you where the bottleneck is, note that the bottleck can be different for different io patterns ( seq/rand/4k-4Mblock sizes ).
Some other points:
For filestore there are double writes due journal, for collocated journal you half your bandwidth and reduce iops by nearly 3 (3 seeks per write). If you add in your replica writes there is another factor of 2. 
For your 2.5GB/s rand write, each disk will be doing 500MB/s. For 60K iops rand write, each disk will do about 18K write iops. Bluestore will give you better performance since it does not have double writes.
For reads the 4.5 GB/s you are seeing gives 225 MB/s per disk which is low, but i suspect this is due to the default thread count of 16 which is lower than your disk count, so some will be idle. You should increase your threads to 64 or 128. The 4.5GB/s could also be a limit from your client machine, the rados benchmark is a client side application, in many cases the bottleneck will be due to client resources. Best run several clients and aggregate their result, do not run clients from the same nodes as OSDs else your resources will be double loaded. Keep adding more clients until your total cluster output saturates. There is a tool than can automate this if you need that level of detail:
https://github.com/ceph/cbt
Last i would recommend running fio to test raw disk performance, note the journal does sequential writes with O_DSYNC:
4M sequential write O_DSYNC 1 Worker:
fio --name=xx --filename=/dev/sdX --iodepth=1 --rw=write --bs=4M --direct=1 --sync=1 --runtime=20 --time_based --numjobs=1 --group_reporting
4M random write 1 Worker:
fio --name=xx --filename=/dev/sdX --iodepth=1 --rw=randwrite --bs=4M --direct=1 --runtime=20 --time_based --numjobs=1 --group_reporting
4M sequential write O_DSYNC 8 Workers:
fio --name=xx --filename=/dev/sdX --iodepth=1 --rw=write --bs=4M --direct=1 --sync=1 --runtime=20 --time_based --numjobs=8 --group_reporting
4M random write 8 Workers:
fio --name=xx --filename=/dev/sdX --iodepth=1 --rw=randwrite --bs=4M --direct=1 --runtime=20 --time_based --numjobs=8 --group_reporting 
Hope this helps
Maged

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com