Looks like it has been done https://github.com/zhouyuan/ceph/commit/f352b8b908e8788d053cbe15fa3632b226a6758d > -----Original Message----- > From: Robert LeBlanc [mailto:robert@xxxxxxxxxxxxx] > Sent: 08 January 2016 18:23 > To: Nick Fisk <nick@xxxxxxxxxx> > Cc: Wade Holler <wade.holler@xxxxxxxxx>; hnuzhoulin > <hnuzhoulin2@xxxxxxxxx>; Ceph-User <ceph-users@xxxxxxxx> > Subject: Re: using cache-tier with writeback mode, raods bench > result degrade > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Are you backporting that to hammer? We'd love it. > - ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > > On Fri, Jan 8, 2016 at 9:28 AM, Nick Fisk wrote: > > There was/is a bug in Infernalis and older, where objects will always get > promoted on the 2nd read/write regardless of what you set the > min_recency_promote settings to. This can have a dramatic effect on > performance. I wonder if this is what you are experiencing? > > > > This has been fixed in Jewel https://github.com/ceph/ceph/pull/6702 . > > > > You can compile the changes above to see if it helps or I have a .deb for > Infernalis where this is fixed if it's easier. > > > > Nick > > > >> -----Original Message----- > >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf > >> Of Wade Holler > >> Sent: 08 January 2016 16:14 > >> To: hnuzhoulin ; ceph-devel@xxxxxxxxxxxxxxx > >> Cc: ceph-users@xxxxxxxx > >> Subject: Re: using cache-tier with writeback mode, raods > >> bench result degrade > >> > >> My experience is performance degrades dramatically when dirty objects > >> are flushed. > >> > >> Best Regards, > >> Wade > >> > >> > >> On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin wrote: > >> Hi,guyes > >> Recentlly,I am testing cache-tier using writeback mode.but I found a > >> strange things. > >> the performance using rados bench degrade.Is it correct? > >> If so,how to explain.following some info about my test: > >> > >> storage node:4 machine,two INTEL SSDSC2BB120G4(one for systaem,the > >> other one used as OSD),four sata as OSD. > >> > >> before using cache-tier: > >> root@ceph1:~# rados bench -p coldstorage 300 write --no-cleanup > >> ---------------------------------------------------------------- > >> Total time run: 301.236355 > >> Total writes made: 6041 > >> Write size: 4194304 > >> Bandwidth (MB/sec): 80.216 > >> > >> Stddev Bandwidth: 10.5358 > >> Max bandwidth (MB/sec): 104 > >> Min bandwidth (MB/sec): 0 > >> Average Latency: 0.797838 > >> Stddev Latency: 0.619098 > >> Max latency: 4.89823 > >> Min latency: 0.158543 > >> > >> root@ceph1:/root/cluster# rados bench -p coldstorage 300 seq > >> Total time run: 133.563980 > >> Total reads made: 6041 > >> Read size: 4194304 > >> Bandwidth (MB/sec): 180.917 > >> > >> Average Latency: 0.353559 > >> Max latency: 1.83356 > >> Min latency: 0.027878 > >> > >> after configure cache-tier: > >> root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier add > coldstorage > >> hotstorage pool 'hotstorage' is now (or already was) a tier of > >> 'coldstorage' > >> > >> root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier cache-mode > >> hotstorage writeback set cache-mode for pool 'hotstorage' to > >> writeback > >> > >> root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier set-overlay > >> coldstorage hotstorage overlay for 'coldstorage' is now (or already > >> was) 'hotstorage' > >> > >> oot@ubuntu:~# ceph osd dump|grep storage pool 6 'coldstorage' > >> replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins > >> pg_num 512 pgp_num 512 last_change 216 lfor 216 flags hashpspool > >> tiers 7 read_tier 7 write_tier 7 stripe_width 0 pool 7 'hotstorage' > >> replicated size 3 min_size 1 crush_ruleset 1 object_hash rjenkins > >> pg_num 128 pgp_num 128 last_change 228 flags > >> hashpspool,incomplete_clones tier_of 6 cache_mode writeback > >> target_bytes > >> 100000000000 hit_set bloom{false_positive_probability: 0.05, target_size: > >> 0, seed: 0} 3600s x6 stripe_width 0 > >> ------------------------------------------------------------- > >> rados bench -p coldstorage 300 write --no-cleanup Total time run: > >> 302.207573 Total writes made: 4315 Write size: 4194304 Bandwidth > >> (MB/sec): 57.113 > >> > >> Stddev Bandwidth: 23.9375 > >> Max bandwidth (MB/sec): 104 > >> Min bandwidth (MB/sec): 0 > >> Average Latency: 1.1204 > >> Stddev Latency: 0.717092 > >> Max latency: 6.97288 > >> Min latency: 0.158371 > >> > >> root@ubuntu:/# rados bench -p coldstorage 300 seq Total time run: > >> 153.869741 Total reads made: 4315 Read size: 4194304 Bandwidth > >> (MB/sec): 112.173 > >> > >> Average Latency: 0.570487 > >> Max latency: 1.75137 > >> Min latency: 0.039635 > >> > >> > >> ceph.conf: > >> -------------------------------------------- > >> [global] > >> fsid = 4ec1eb64-226c-4d90-8c5c-b6b6644be831 > >> mon_initial_members = ceph2, ceph3, ceph4 mon_host = > >> 10.**.**.241,10.**.**.242,10.**.**.243 > >> auth_cluster_required = cephx > >> auth_service_required = cephx > >> auth_client_required = cephx > >> filestore_xattr_use_omap = true > >> osd_pool_default_size = 3 > >> osd_pool_default_min_size = 1 > >> auth_supported = cephx > >> osd_journal_size = 10240 > >> osd_mkfs_type = xfs > >> osd crush update on start = false > >> > >> [client] > >> rbd_cache = true > >> rbd_cache_writethrough_until_flush = false rbd_cache_size = 33554432 > >> rbd_cache_max_dirty = 25165824 rbd_cache_target_dirty = 16777216 > >> rbd_cache_max_dirty_age = 1 rbd_cache_block_writes_upfront = false > >> [osd] filestore_omap_header_cache_size = 40000 > >> filestore_fd_cache_size = 40000 filestore_fiemap = true > >> client_readahead_min = 2097152 client_readahead_max_bytes = 0 > >> client_readahead_max_periods = 4 filestore_journal_writeahead = false > >> filestore_max_sync_interval = 10 filestore_queue_max_ops = 500 > >> filestore_queue_max_bytes = 1048576000 > >> filestore_queue_committing_max_ops = 5000 > >> filestore_queue_committing_max_bytes = 1048576000 > >> keyvaluestore_queue_max_ops = 500 keyvaluestore_queue_max_bytes > = > >> 1048576000 journal_queue_max_ops = 30000 journal_queue_max_bytes > = > >> 3355443200 osd_op_threads = 20 osd_disk_threads = 8 > >> filestore_op_threads = 4 osd_mount_options_xfs = > >> rw,noatime,nobarrier,inode64,logbsize=256k,delaylog > >> > >> [mon] > >> mon_osd_allow_primary_affinity=true > >> > >> -- > >> 使用Opera的电子邮件客户端：http://www.opera.com/mail/ > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -----BEGIN PGP SIGNATURE----- > Version: Mailvelope v1.3.3 > Comment: https://www.mailvelope.com > > wsFcBAEBCAAQBQJWj/52CRDmVDuy+mK58QAAI1UP/jDBE95QirkgybKwOS5 > V > fpCn+9tAk3psAkVEg3f9XdXtr2et0XSRiZ2Nce20Dj8OFsLcvNLuDXNEzwVH > DLJTmQIvxyEFliXZyW+Tp1aumQylhlWMT/4zgc8H1YUxo0EBaYG8xXsItTBx > DRHykhYPokn0lJxYBpqM6YyiTvAh1OsPL8fwDZaSan5uIaPD7VnbXmoYdkI3 > jJ2TosK7qANlHCeDu2u4Ppc3Re3cA0+cCcf15vGabNe3yGbA+k1GvKLFgLLV > zuveqUdxfkDQuTRoepZo44nLVkTnsbQrrTyR2cYJxEI+p2UHh6gO0gzK4VOC > nR6/rhnw6ZEXOlVR4pU1Gc854D+yZx1u+VLDEO1Z8FBquRtfiReOj9qvnarM > PwuVSstLmL5Z9hOJ3+fL1ZGvBn+W8lO7SUurdi4IKGkuRygX7OrCvGkOVuzq > SxHce5u79GIRyIHT7SIeX2gLMTu1ox+ThM9S9iyAtUzVxHMA6qcOe1KzS0dR > hnG5K68bE3i7ynwYa6U5MGTTxZKX6oQn6FZ58jBZeAejacPVRT954YdbKMEJ > bN1qCuCrSTa8sufGNltb64y+crYBUQlGjvU78I0PcjfVIYH5irxc6qVGdzB1 > eUqsz8tWaQPDCYAsG661YmSHf1fghNs8BpDST0u5uItvcIQ8BMeqwWeA+eED > Cpzq > =4mWK > -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |