I setup a cephfs file system with a cache tier over an erasure coded tier as an experiment:
ceph osd erasure-code-profile set raid6 k=4 m=2
ceph osd pool create cephfs-metadata 512 512
ceph osd pool set cephfs-metadata size 3
ceph osd pool create cache-data 2048 2048
ceph osd pool create cephfs-data 256 256 erasure raid6 default_erasure
ceph osd tier add cephfs-data cache-data
ceph osd tier cache-mode cache-data writeback
ceph osd tier set-overlay cephfs-data cache-data
ceph osd pool set cache-data hit_set_type bloom
ceph osd pool set cache-data target_max_bytes 1099511627776
The file system was created from the cephfs-metadata and cephfs-data pools
After adding a lot of data to this and waiting for the pools to idle down and stabilize I removing the file system content with rm. I am seeing very strange behavior, the file system remove was quick, and then it started removing the data from the pools. However it appears to be reading the data from the erasure coded pool and creating empty content in the cache pool.
At its peak capacity the system looked like this:
NAME ID CATEGORY USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE
cache-data 21 - 791G 5.25 6755G 256302 140k 22969 2138k
cephfs-data 22 - 4156G 27.54 4503G 1064086 1039k 51271 1046k
2 hours later it looked like this:
NAME ID CATEGORY USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE
cache-data 21 - 326G 2.17 7576G 702142 559k 22969 2689k
cephfs-data 22 - 3964G 26.27 5051G 1014842 991k 476k 1143k
The object count in the erasure coded pool has gone down a little, the count in the cache pool has gone up a lot, there has been a lot of read activity in the erasure coded pool and write activity into both pools. The used count in the cache pool is also going down. It looks like the cache pool is gaining 9 objects for each one removed from the erasure code pool. Looking at the actual files being created by the OSD for this, they are empty.
What is going on here? It looks like this will take a day or so to complete at this rate of progress.
The ceph version here is the master branch from a couple of days ago.
Thanks
Steve Lord
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
