There was/is a bug in Infernalis and older, where objects will always get promoted on the 2nd read/write regardless of what you set the min_recency_promote settings to. This can have a dramatic effect on performance. I wonder if this is what you are experiencing? This has been fixed in Jewel https://github.com/ceph/ceph/pull/6702 . You can compile the changes above to see if it helps or I have a .deb for Infernalis where this is fixed if it's easier. Nick > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Wade Holler > Sent: 08 January 2016 16:14 > To: hnuzhoulin <hnuzhoulin2@xxxxxxxxx>; ceph-devel@xxxxxxxxxxxxxxx > Cc: ceph-users@xxxxxxxx > Subject: Re: using cache-tier with writeback mode, raods bench > result degrade > > My experience is performance degrades dramatically when dirty objects are > flushed. > > Best Regards, > Wade > > > On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin <hnuzhoulin2@xxxxxxxxx> wrote: > Hi,guyes > Recentlly,I am testing cache-tier using writeback mode.but I found a > strange things. > the performance using rados bench degrade.Is it correct? > If so,how to explain.following some info about my test: > > storage node:4 machine,two INTEL SSDSC2BB120G4(one for systaem,the > other > one used as OSD),four sata as OSD. > > before using cache-tier: > root@ceph1:~# rados bench -p coldstorage 300 write --no-cleanup > ---------------------------------------------------------------- > Total time run: 301.236355 > Total writes made: 6041 > Write size: 4194304 > Bandwidth (MB/sec): 80.216 > > Stddev Bandwidth: 10.5358 > Max bandwidth (MB/sec): 104 > Min bandwidth (MB/sec): 0 > Average Latency: 0.797838 > Stddev Latency: 0.619098 > Max latency: 4.89823 > Min latency: 0.158543 > > root@ceph1:/root/cluster# rados bench -p coldstorage 300 seq > Total time run: 133.563980 > Total reads made: 6041 > Read size: 4194304 > Bandwidth (MB/sec): 180.917 > > Average Latency: 0.353559 > Max latency: 1.83356 > Min latency: 0.027878 > > after configure cache-tier: > root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier add coldstorage > hotstorage > pool 'hotstorage' is now (or already was) a tier of 'coldstorage' > > root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier cache-mode > hotstorage writeback > set cache-mode for pool 'hotstorage' to writeback > > root@ubuntu:~/benchmarkcollect/Monitor# ceph osd tier set-overlay > coldstorage hotstorage > overlay for 'coldstorage' is now (or already was) 'hotstorage' > > oot@ubuntu:~# ceph osd dump|grep storage > pool 6 'coldstorage' replicated size 3 min_size 1 crush_ruleset 0 > object_hash rjenkins pg_num 512 pgp_num 512 last_change 216 lfor 216 > flags > hashpspool tiers 7 read_tier 7 write_tier 7 stripe_width 0 > pool 7 'hotstorage' replicated size 3 min_size 1 crush_ruleset 1 > object_hash rjenkins pg_num 128 pgp_num 128 last_change 228 flags > hashpspool,incomplete_clones tier_of 6 cache_mode writeback > target_bytes > 100000000000 hit_set bloom{false_positive_probability: 0.05, target_size: > 0, seed: 0} 3600s x6 stripe_width 0 > ------------------------------------------------------------- > rados bench -p coldstorage 300 write --no-cleanup > Total time run: 302.207573 > Total writes made: 4315 > Write size: 4194304 > Bandwidth (MB/sec): 57.113 > > Stddev Bandwidth: 23.9375 > Max bandwidth (MB/sec): 104 > Min bandwidth (MB/sec): 0 > Average Latency: 1.1204 > Stddev Latency: 0.717092 > Max latency: 6.97288 > Min latency: 0.158371 > > root@ubuntu:/# rados bench -p coldstorage 300 seq > Total time run: 153.869741 > Total reads made: 4315 > Read size: 4194304 > Bandwidth (MB/sec): 112.173 > > Average Latency: 0.570487 > Max latency: 1.75137 > Min latency: 0.039635 > > > ceph.conf: > -------------------------------------------- > [global] > fsid = 4ec1eb64-226c-4d90-8c5c-b6b6644be831 > mon_initial_members = ceph2, ceph3, ceph4 > mon_host = 10.**.**.241,10.**.**.242,10.**.**.243 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > filestore_xattr_use_omap = true > osd_pool_default_size = 3 > osd_pool_default_min_size = 1 > auth_supported = cephx > osd_journal_size = 10240 > osd_mkfs_type = xfs > osd crush update on start = false > > [client] > rbd_cache = true > rbd_cache_writethrough_until_flush = false > rbd_cache_size = 33554432 > rbd_cache_max_dirty = 25165824 > rbd_cache_target_dirty = 16777216 > rbd_cache_max_dirty_age = 1 > rbd_cache_block_writes_upfront = false > [osd] > filestore_omap_header_cache_size = 40000 > filestore_fd_cache_size = 40000 > filestore_fiemap = true > client_readahead_min = 2097152 > client_readahead_max_bytes = 0 > client_readahead_max_periods = 4 > filestore_journal_writeahead = false > filestore_max_sync_interval = 10 > filestore_queue_max_ops = 500 > filestore_queue_max_bytes = 1048576000 > filestore_queue_committing_max_ops = 5000 > filestore_queue_committing_max_bytes = 1048576000 > keyvaluestore_queue_max_ops = 500 > keyvaluestore_queue_max_bytes = 1048576000 > journal_queue_max_ops = 30000 > journal_queue_max_bytes = 3355443200 > osd_op_threads = 20 > osd_disk_threads = 8 > filestore_op_threads = 4 > osd_mount_options_xfs = > rw,noatime,nobarrier,inode64,logbsize=256k,delaylog > > [mon] > mon_osd_allow_primary_affinity=true > > -- > 使用Opera的电子邮件客户端：http://www.opera.com/mail/ > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |