Hi,
i am for sure not really experienced yet with ceph or with cache tier,
but to me it seems to behave strange.
Setup:
pool 3 'ssd_cache' replicated size 2 min_size 1 crush_ruleset 1
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 190 flags
hashpspool,incomplete_clones tier_of 4 cache_mode writeback target_bytes
800000000000 hit_set bloom{false_positive_probability: 0.05,
target_size: 0, seed: 0} 3600s x1 decay_rate 0 search_last_n 0
stripe_width 0
pool 4 'cephfs_data' replicated size 2 min_size 1 crush_ruleset 2
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 169 lfor 144
flags hashpspool crash_replay_interval 45 tiers 3 read_tier 3 write_tier
3 stripe_width 0
pool 5 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 1
object_hash rjenkins pg_num 128 pgp_num 128 last_change 191 flags
hashpspool stripe_width 0
hit_set_count: 1
hit_set_period: 120
target_max_bytes: 800000000000
min_read_recency_for_promote: 0
min_write_recency_for_promote: 0
target_max_objects: 0
cache_target_dirty_ratio: 0.5
cache_target_dirty_high_ratio: 0.8
cache_target_full_ratio: 0.9
cache_min_flush_age: 1800
cache_min_evict_age: 3600
rule ssd-cache-rule {
ruleset 1
type replicated
min_size 2
max_size 10
step take ssd-cache
step chooseleaf firstn 0 type host
step emit
}
rule cold-storage-rule {
ruleset 2
type replicated
min_size 2
max_size 10
step take cold-storage
step chooseleaf firstn 0 type host
step emit
}
[root@cephmon1 ceph-cluster-gen2]# rados -p ssd_cache ls
[root@cephmon1 ceph-cluster-gen2]#
-> empty
Now, on a cephfs mounted client i have files.
Read operation:
dd if=testfile of=/dev/zero
1494286336 bytes (1.5 GB) copied, 11.047 s, 135 MB/s
[root@cephosd1 ~]# rados -p ssd_cache ls
1000000001e.00000010
1000000001e.00000004
1000000001e.00000001
1000000001e.0000000c
1000000001e.00000008
1000000001e.00000003
1000000001e.00000000
1000000001e.00000002
Running this multiple times after one another, does not change the
content. Its always the same objects.
-------------
Ok, so according to the documents, writeback mode, it moved from cold
storeage to hot storage ( cephfs_data to ssd_cache in my case ).
Now i repeat it:
dd if=testfile of=/dev/zero
1494286336 bytes (1.5 GB) copied, 11.311 s, 132 MB/s
[root@cephosd1 ~]# rados -p ssd_cache ls
1000000001e.00000010
1000000001e.00000004
1000000001e.00000001
1000000001e.0000000c
1000000001e.0000000d
1000000001e.00000005
1000000001e.00000008
1000000001e.00000015
1000000001e.00000011
1000000001e.00000006
1000000001e.00000003
1000000001e.00000009
1000000001e.00000000
1000000001e.0000000a
1000000001e.0000001b
1000000001e.00000002
So why are there now the old objects ( 8 ) plus another 8 objects ?
Repeating this, will extend the numbers of objects endless without
speeding up the dd. in the ssd_cache.
So every new dd read, of exact the same file ( to me that means, same
PGs/objects ) the (same) data is copied from cold pool to cache pool.
And from there pushed to the client ( without any speed gain ).
And thats not supposed to happen ( according to the documentation with
writeback cache mode ).
Similar happens when i am writing.
If i write, it will store the data on cold pool and cache pool equally.
For my understanding, with my configuration, at least 1800 seconds (
cache_min_flush_age ) should pass by before the agent starts to flush
from the cache pool to the cold pool.
But it does not.
So, is there something specific with cephfs, or is my config just too
much crappy and i have no idea what i am doing here ?
Anything is highly welcome !
Thank you !
--
Mit freundlichen Gruessen / Best regards
Oliver Dzombic
IP-Interactive
mailto:info@xxxxxxxxxxxxxxxxx
Anschrift:
IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen
HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic
Steuer Nr.: 35 236 3622 1
UST ID: DE274086107
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
