|
Dear Ceph experts, I found something strange about the performance of my Ceph cluster: Read-out much slower than write-in. I have 3 machines running OSDs, each has 8 OSDs running on 8 raid0s (each made up of 2 HDDs) respectively. The OSD journal and data the is on the same device. All machines in my clusters have 10Gb network. I used both Ceph RBD and CephFS, the client on another machine outside cluster or on one of the running OSD (to rule out possible network issue), an so on. All of these end up in a similar results: write-in can almost reach the network limit, say 1200 MB/s, while read-out is only 350~450 MB/s. Trying to figure out, I did an extra test using CephFS: Version and Config: [root@dl-disk1 ~]# ceph --version ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b) [root@dl-disk1 ~]# cat /etc/ceph/ceph.conf [global] fsid = (hidden) mon_initial_members = dl-disk1, dl-disk2, dl-disk3 mon_host = (hidden) auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true OSD tree: # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 258.88000 root default -2 87.28000 host dl-disk1 0 10.90999 osd.0 up 1.00000 1.00000 1 10.90999 osd.1 up 1.00000 1.00000 2 10.90999 osd.2 up 1.00000 1.00000 3 10.90999 osd.3 up 1.00000 1.00000 4 10.90999 osd.4 up 1.00000 1.00000 5 10.90999 osd.5 up 1.00000 1.00000 6 10.90999 osd.6 up 1.00000 1.00000 7 10.90999 osd.7 up 1.00000 1.00000 -3 87.28000 host dl-disk2 8 10.90999 osd.8 up 1.00000 1.00000 9 10.90999 osd.9 up 1.00000 1.00000 10 10.90999 osd.10 up 1.00000 1.00000 11 10.90999 osd.11 up 1.00000 1.00000 12 10.90999 osd.12 up 1.00000 1.00000 13 10.90999 osd.13 up 1.00000 1.00000 14 10.90999 osd.14 up 1.00000 1.00000 15 10.90999 osd.15 up 1.00000 1.00000 -4 84.31999 host dl-disk3 16 10.53999 osd.16 up 1.00000 1.00000 17 10.53999 osd.17 up 1.00000 1.00000 18 10.53999 osd.18 up 1.00000 1.00000 19 10.53999 osd.19 up 1.00000 1.00000 20 10.53999 osd.20 up 1.00000 1.00000 21 10.53999 osd.21 up 1.00000 1.00000 22 10.53999 osd.22 up 1.00000 1.00000 23 10.53999 osd.23 up 1.00000 1.00000 Pools and PG (each pool has 128 PGs): # ceph osd lspools 0 rbd,2 fs_meta,3 fs_data0,4 fs_data1, # ceph pg dump pools dumped pools in format plain pg_stat objects mip degr misp unf bytes log disklog pool 0 0 0 0 0 0 0 0 0 pool 2 20 0 0 0 0 356958 264 264 pool 3 3264 0 0 0 0 16106127360 14657 14657 pool 4 0 0 0 0 0 0 0 0 To simplify the problem, I made a new crush rule that the CephFS data pool use OSDs on only one machine (dl-disk1 here), and size = 1. # ceph osd crush rule dump osd_in_dl-disk1__ruleset { "rule_id": 1, "rule_name": "osd_in_dl-disk1__ruleset", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -2, "item_name": "dl-disk1" }, { "op": "chooseleaf_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] } # ceph osd pool get fs_data0 crush_ruleset crush_ruleset: 1 # ceph osd pool get fs_data0 size size: 1 Here starts the test. On an client machine, I used dd to write a 4GB-file to CephFS, and checked dstat on the OSD node dl-disk1: [root@client ~]# dd of=/mnt/cephfs/4Gfile if=/dev/zero bs=4096k count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB) copied, 3.69993 s, 1.2 GB/s [root@dl-disk1 ~]# dstat ... ---total-cpu-usage---- ------memory-usage----- -net/total- --dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde-----dsk/sdf-----dsk/sdg-----dsk/sdh-----dsk/sdi-- usr sys idl wai hiq siq| used buff cach free| recv send| read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ 0 0 100 0 0 0|3461M 67.2M 15.1G 44.3G| 19k 20k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3461M 67.2M 15.1G 44.3G| 32k 32k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 8 18 74 0 0 0|3364M 67.2M 11.1G 48.4G| 391k 391k| 0 2712k: 0 1096k: 0 556k: 0 1084k: 0 1200k: 0 1196k: 0 688k: 0 1252k 0 0 100 0 0 0|3364M 67.2M 11.1G 48.4G| 82k 127k| 0 0 : 0 0 : 0 0 : 0 928k: 0 540k: 0 0 : 0 0 : 0 0 8 16 72 3 0 1|3375M 67.2M 11.8G 47.7G| 718M 2068k| 0 120M: 0 172M: 0 76M: 0 220M: 0 188M: 16k 289M: 0 53M: 0 36M 6 13 77 4 0 1|3391M 67.2M 12.3G 47.1G| 553M 1517k| 0 160M: 0 176M: 0 88M: 0 208M: 0 225M: 0 213M: 0 8208k: 0 49M 6 13 77 3 0 1|3408M 67.2M 12.9G 46.6G| 544M 1272k| 0 212M: 0 8212k: 0 36M: 0 0 : 0 37M: 0 3852k: 0 497M: 0 337M 0 0 99 0 0 0|3407M 67.3M 12.9G 46.6G| 53k 114k| 0 36M: 0 37M: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3407M 67.3M 12.9G 46.6G| 68k 110k| 0 0 : 0 0 : 0 0 : 0 36M: 0 0 : 0 0 : 0 0 : 0 0 0 0 99 0 0 0|3407M 67.3M 12.9G 46.6G| 38k 328k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 36M: 0 0 0 1 99 0 0 0|3406M 67.3M 12.9G 46.6G| 11M 132k| 0 0 : 0 0 : 0 8224k: 0 0 : 0 0 : 0 32M: 0 0 : 0 36M 14 24 52 8 0 2|3436M 67.3M 13.8G 45.6G|1026M 2897k| 0 100M: 0 409M: 0 164M: 0 313M: 0 253M: 0 321M: 0 84M: 0 76M 14 24 34 27 0 1|3461M 67.3M 14.7G 44.7G| 990M 2565k| 0 354M: 0 72M: 0 0 : 0 164M: 0 313M: 0 188M: 0 308M: 0 333M 4 9 70 16 0 0|3474M 67.3M 15.1G 44.3G| 269M 646k| 0 324M: 0 0 : 0 0 : 0 36M: 0 0 : 0 0 : 0 349M: 0 172M 0 0 99 0 0 0|3474M 67.3M 15.1G 44.3G| 24k 315k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 37M: 0 0 0 0 99 0 0 0|3474M 67.4M 15.1G 44.3G| 38k 102k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 36M: 0 0 : 0 36M 0 0 99 0 0 0|3473M 67.4M 15.1G 44.3G| 22k 23k| 0 0 : 0 0 : 0 36M: 0 0 : 0 36M: 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3473M 67.4M 15.1G 44.3G| 39k 40k| 0 304k: 0 16k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3472M 67.4M 15.1G 44.3G| 28k 64k| 0 64M: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3471M 67.4M 15.1G 44.3G| 31k 94k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3472M 67.4M 15.1G 44.3G| 38k 39k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 The throughput is 1.2 GB/s, able to reach the network limit 10Gb. Then, on the client machine, I used dd to read that file back from CephFS, redirecting the file to /dev/zero (or /dev/null) to rule out local HDD's IO: [root@client ~]# dd if=/mnt/cephfs/4Gfile of=/dev/zero bs=4096k count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB) copied, 8.85246 s, 485 MB/s [root@dl-disk1 ~]# dstat ... 0 0 100 0 0 0|3462M 67.4M 15.1G 44.3G| 36k 36k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3462M 67.4M 15.1G 44.3G| 22k 22k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3463M 67.4M 15.1G 44.3G| 49k 49k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 1 99 0 0 0|3464M 67.4M 15.1G 44.3G| 282k 111M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 5 93 0 0 0|3466M 67.4M 15.1G 44.3G|1171k 535M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 5 93 0 0 0|3467M 67.4M 15.1G 44.3G|1124k 535M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3467M 67.4M 15.1G 44.3G|1124k 535M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3467M 67.4M 15.1G 44.3G|1109k 527M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 93 0 0 0|3471M 67.4M 15.1G 44.3G|1044k 504M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3470M 67.4M 15.1G 44.3G|1031k 504M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 5 93 0 0 0|3470M 67.4M 15.1G 44.3G|1103k 527M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 93 0 0 0|3471M 67.5M 15.1G 44.3G|1084k 504M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3470M 67.5M 15.1G 44.3G| 25k 24k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 ----total-cpu-usage---- ------memory-usage----- -net/total- --dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde-----dsk/sdf-----dsk/sdg-----dsk/sdh-----dsk/sdi-- usr sys idl wai hiq siq| used buff cach free| recv send| read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ 0 0 100 0 0 0|3470M 67.5M 15.1G 44.3G| 43k 44k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3470M 67.5M 15.1G 44.3G| 22k 23k| 0 48k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 35k 38k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 23k 85k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 44k 44k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 24k 25k| 0 12k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 45k 43k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3468M 67.5M 15.1G 44.3G| 17k 18k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 The throughput here was only 400~500 MB/s here. I noticed that there was NO disk I/O during the read-out, that means all the objects of the file were already cached in memory on the OSD node. Thus, HDDs does NOT seem to cause the lower throughput. I also tried read-out using cat (in case dd may not use read-ahead in file system. ), ended up getting similar result: [root@client ~]# time cat /mnt/cephfs/4Gfile > /dev/zero real 0m9.352s user 0m0.002s sys 0m4.147s [root@dl-disk1 ~]# dstat ... 0 0 100 0 0 0|3465M 67.5M 15.1G 44.3G| 23k 22k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3465M 67.5M 15.1G 44.3G| 17k 18k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3465M 67.5M 15.1G 44.3G| 37k 37k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 1 2 97 0 0 0|3466M 67.5M 15.1G 44.3G| 633k 280M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3467M 67.5M 15.1G 44.3G|1057k 498M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3470M 67.5M 15.1G 44.3G|1078k 498M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3470M 67.5M 15.1G 44.3G| 996k 486M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3469M 67.5M 15.1G 44.3G| 988k 489M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3469M 67.5M 15.1G 44.3G|1012k 489M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3470M 67.5M 15.1G 44.3G|1017k 497M| 0 0 : 0 8192B: 0 28k: 0 0 : 0 0 : 0 0 : 0 0 : 0 0 2 4 94 0 0 0|3469M 67.5M 15.1G 44.3G|1032k 498M| 0 0 : 0 0 : 0 0 : 0 8192B: 0 104k: 0 0 : 0 0 : 0 0 ----total-cpu-usage---- ------memory-usage----- -net/total- --dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde-----dsk/sdf-----dsk/sdg-----dsk/sdh-----dsk/sdi-- usr sys idl wai hiq siq| used buff cach free| recv send| read writ: read writ: read writ: read writ: read writ: read writ: read writ: read writ 2 4 94 0 0 0|3469M 67.5M 15.1G 44.3G|1025k 496M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 40k: 0 80k: 0 0 0 1 99 0 0 0|3469M 67.5M 15.1G 44.3G| 127k 52M| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 120k 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 21k 21k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 66k 66k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 0 0 100 0 0 0|3469M 67.5M 15.1G 44.3G| 35k 38k| 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 : 0 0 The average throughput is 4GB / 9.35s = 438 MB/s. Still, unlikely to be HDD's issue. I'm sure that the network can reach 10Gb in both ways via iperf or other test, and there's no other user process occupying bandwidth. Could you please help me some to find out the main reason for this issue? Thank you. Best Regards, FaHui |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
