It may be a bug. We should set target_max_bytes or target_max_objects before tier pool process request. 2015-03-10 20:39 GMT+08:00 Kamil Kuramshin <kamil.kuramshin@xxxxxxxx>: > Thanks a lot to Be-El from #ceph (irc://irc.oftc.net/ceph) > The problem is resolved after setting 'target_max_bytes' for cache pool: > > $ ceph osd pool set cache target_max_bytes 184000000000 > > Because setting only 'cache_target_full_ratio' to 0.7 - is not sufficient > for cache tiering agent, it must know the number of bytes for specified > pool that allowed to write, before starting evicting process. And then with > both options are in place, cache tiering agent will start eviting objects > from cache pool when there is 70% of target_max_bytes are written to that > pool (when reacing 129Gb in my case) > > here how it was: > > [14:45] <Be-El> stannum: there's no limit configured for the cache pool. the > system cannot know when to evict data > [14:46] <stannum> Be-El: what kind of limit? > [14:46] <Be-El> stannum: a limit for the size or the number of objects in > the cache pool > [14:48] <stannum> Be-El: setting cache_target_full_ratio: 0.7 is not > sufficient? > [14:48] <Be-El> stannum: that's a ratio. you need a reference for it > [14:51] <Be-El> stannum: try setting 'target_max_bytes' to the maximum size > of the underlying storage minus some percent overhead and keep the > replication factor in mind > [14:51] <stannum> Be-El: oh, there is not clear from documentaion. it say > that: The cache tiering agent can flush or evict objects relative to the > size of the cache pool. And later it says: The cache tiering agent can flush > or evict objects based upon the total number of bytes or the total number of > objects. > [14:52] <stannum> Be-El: and nothing said about to setting both options > [14:52] <Be-El> stannum: yes, the documentation is somewhat lacking. ceph > cannot determine the amount of available space (and thus the maximum > possible size of a pool) > > > 10.03.2015 14:41, Kamil Kuramshin пишет: > > hi, folks! I'm testing cache tier for erasure coded pool and with RBD image > on it. And now I'm facing a problem with full cache pool and object are not > evicted automatically, Only if I run manually rados -p cache > cache-flush-evict-all > > client side is: > superuser@share:~$ uname -a > Linux share 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 > GNU/Linux > > > ceph node all are debian wheezy > superuser~$ dpkg -l | grep ceph > ii ceph 0.87-1~bpo70+1 amd64 > distributed storage and file system > ii ceph-common 0.87-1~bpo70+1 amd64 > common utilities to mount and interact with a ceph storage cluster > ii ceph-fs-common 0.87-1~bpo70+1 amd64 > common utilities to mount and interact with a ceph file system > ii ceph-mds 0.87-1~bpo70+1 amd64 > metadata server for the ceph distributed file system > ii libcephfs1 0.87-1~bpo70+1 amd64 > Ceph distributed file system client library > ii libcurl3-gnutls:amd64 7.29.0-1~bpo70+1.ceph amd64 > easy-to-use client-side URL transfer library (GnuTLS flavour) > ii python-ceph 0.87-1~bpo70+1 amd64 > Python libraries for the Ceph distributed filesystem > > > > There are all steps to reproduce (excepting creation of pools): > > > > superuser@admin:~$ ceph osd pool get ec_backup-storage erasure_code_profile > erasure_code_profile: default > superuser@admin:~$ ceph osd erasure-code-profile get default > directory=/usr/lib/ceph/erasure-code > k=2 > m=1 > plugin=jerasure > technique=reed_sol_van > > > *********** ADMIN NODE OPERATIONS ************ > > superuser@admin:~$ ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 242T 224T 6092G 2.46 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > ec_backup-storage 4 0 0 147T 0 > cache 5 0 0 185G 0 > block-devices 6 1948G 0.79 75638G 498771 > superuser@admin:~$ rados df > pool name category KB objects clones > degraded unfound rd rd KB wr wr KB > block-devices - 2042805201 498771 0 > 0 0 67127 259320535 2070571 2403248346 > cache - 0 0 0 > 0 0 60496 247235411 966553 499544074 > ec_backup-storage - 0 0 0 > 0 0 156988 537227276 400355 819838985 > total used 6388431372 498771 > total avail 240559782780 > total space 260163775608 > > ***** 'cache' pool is replicated pool, 'ec_backup-storage' - Erasure Encoded > pool ***** > ***** running simple script for enabling cache tiering: > > > ***** There is my simple script for enabling cache tier: > > superuser@admin:~$ ./enable_cache_tier.sh cache ec_backup-storage > pool 'cache' is now (or already was) a tier of 'ec_backup-storage' > set cache-mode for pool 'cache' to writeback > overlay for 'ec_backup-storage' is now (or already was) 'cache' > set pool 5 hit_set_type to bloom > set pool 5 cache_target_dirty_ratio to 0.4 > set pool 5 cache_target_full_ratio to 0.7 > set pool 5 cache_min_flush_age to 10 > set pool 5 cache_min_evict_age to 10 > > ***** Displaying some cache pool parameters: > > superuser@admin:~$ for param in cache_target_dirty_ratio > cache_target_full_ratio target_max_bytes target_max_objects > cache_min_flush_age cache_min_evict_age; do ceph osd pool get cache $param; > done > cache_target_dirty_ratio: 0.4 > cache_target_full_ratio: 0.7 > target_max_bytes: 0 > target_max_objects: 0 > cache_min_flush_age: 10 > cache_min_evict_age: 10 > > > *********** END ADMIN NODE OPERATIONS ************ > > *********** CEPH CLIENT OPERATIONS ************ > > superuser@share:~$ rbd create -p ec_backup-storage ec_image.img --size > 500000 --image-format 2 > superuser@share:~$ rbd -p ec_backup-storage ls > ec_image.img > superuser@share:~$ sudo rbd map -p ec_backup-storage ec_image.img > /dev/rbd0 > superuser@share:~$ rbd showmapped > id pool image snap device > 0 ec_backup-storage ec_image.img - /dev/rbd0 > superuser@share:~$ sudo parted /dev/rbd0 p > Error: /dev/rbd0: unrecognised disk label > Model: Unknown (unknown) > Disk /dev/rbd0: 524GB > Sector size (logical/physical): 512B/512B > Partition Table: unknown > Disk Flags: > > > **** now start filling the rbd disk with 400Gb of 'zeros' **** > superuser@share:~$ sudo dd if=/dev/zero of=/dev/rbd0 bs=4M count=100000 > oflag=direct > ^C39901+0 записей получено ; There I'm pressed CTRL+C because already > getting warnings in cpeh -s (see below) > 39901+0 записей отправлено > скопировано 167356923904 байта (167 GB), 3387,59 c, 49,4 MB/c > > > *********** END CEPH CLIENT OPERATIONS ************ > > > *********** ADMIN NODE OPERATIONS ************ > > > **** Now check ceph df: 145G of 172G 'cache' pool it is about 90% is > occupied already! > superuser@admin:~$ ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 242T 223T 6555G 2.64 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > ec_backup-storage 4 0 0 147T 0 > cache 5 155G 0.06 12333M 39906 > block-devices 6 1948G 0.79 75642G 498771 > > > ***** ec_backup-storage - tier pool for cold data still empty???? > ***** but in the same time: > superuser@admin:~$ ceph health detail > HEALTH_WARN 3 near full osd(s) > osd.45 is near full at 88% > osd.95 is near full at 87% > osd.100 is near full at 86% > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
