On Thu, Feb 4, 2016 at 4:37 PM, Stephen Lord <Steve.Lord@xxxxxxxxxxx> wrote: > I setup a cephfs file system with a cache tier over an erasure coded tier as an experiment: > > ceph osd erasure-code-profile set raid6 k=4 m=2 > ceph osd pool create cephfs-metadata 512 512 > ceph osd pool set cephfs-metadata size 3 > ceph osd pool create cache-data 2048 2048 > ceph osd pool create cephfs-data 256 256 erasure raid6 default_erasure > ceph osd tier add cephfs-data cache-data > ceph osd tier cache-mode cache-data writeback > ceph osd tier set-overlay cephfs-data cache-data > ceph osd pool set cache-data hit_set_type bloom > ceph osd pool set cache-data target_max_bytes 1099511627776 > > The file system was created from the cephfs-metadata and cephfs-data pools > > After adding a lot of data to this and waiting for the pools to idle down and stabilize I removing the file system content with rm. I am seeing very strange behavior, the file system remove was quick, and then it started removing the data from the pools. However it appears to be reading the data from the erasure coded pool and creating empty content in the cache pool. > > At its peak capacity the system looked like this: > > NAME ID CATEGORY USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE > cache-data 21 - 791G 5.25 6755G 256302 140k 22969 2138k > cephfs-data 22 - 4156G 27.54 4503G 1064086 1039k 51271 1046k > > > 2 hours later it looked like this: > > NAME ID CATEGORY USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE > cache-data 21 - 326G 2.17 7576G 702142 559k 22969 2689k > cephfs-data 22 - 3964G 26.27 5051G 1014842 991k 476k 1143k > > The object count in the erasure coded pool has gone down a little, the count in the cache pool has gone up a lot, there has been a lot of read activity in the erasure coded pool and write activity into both pools. The used count in the cache pool is also going down. It looks like the cache pool is gaining 9 objects for each one removed from the erasure code pool. Looking at the actual files being created by the OSD for this, they are empty. > > What is going on here? It looks like this will take a day or so to complete at this rate of progress. I presume we're doing reads in order to gather some object metadata from the cephfs-data pool; and the (small) newly-created objects in cache-data are definitely whiteout objects indicating the object no longer exists logically. What kinds of reads are you actually seeing? Does it appear to be transferring data, or merely doing a bunch of seeks? I thought we were trying to avoid doing reads-to-delete, but perhaps the way we're handling snapshots or something is invoking behavior that isn't amicable to a full-FS delete. I presume you're trying to characterize the system's behavior, but of course if you just want to empty it out entirely you're better off deleting the pools and the CephFS instance entirely and then starting it over again from scratch. -Greg > > The ceph version here is the master branch from a couple of days ago. > > Thanks > > Steve Lord > > > > ---------------------------------------------------------------------- > The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt. > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |