Hi, you need to set the max dirty bytes and/or max dirty objects as these 2 parameters will default to 0 for your cache pool. ceph osd pool set <cache_pool_name> target_max_objects xxxxx ceph osd pool set <cache_pool_name> target_max_bytes xxxxx The ratios you already set (dirty_ratio = 0.4 and full_ratio = 0.7) will be applied based on the above 2 values. The first one reached (objects and/or bytes) will trigger flush and eviction e.g. You set target_max_bytes = 10000 - Flushing will start when your cache pool reaches 40000 dirty bytes - Evicting will start when your cache pool reaches 70000 dirty bytes Cheers JC > On 10 Mar 2015, at 22:41, Kamil Kuramshin <kamil.kuramshin@xxxxxxxx> wrote: > > hi, folks! I'm testing cache tier for erasure coded pool and with RBD image on it. And now I'm facing a problem with full cache pool and object are not evicted automatically, Only if I run manually rados -p cache cache-flush-evict-all > > client side is: > superuser@share:~$ uname -a > Linux share 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 GNU/Linux > > > ceph node all are debian wheezy > superuser~$ dpkg -l | grep ceph > ii ceph 0.87-1~bpo70+1 amd64 distributed storage and file system > ii ceph-common 0.87-1~bpo70+1 amd64 common utilities to mount and interact with a ceph storage cluster > ii ceph-fs-common 0.87-1~bpo70+1 amd64 common utilities to mount and interact with a ceph file system > ii ceph-mds 0.87-1~bpo70+1 amd64 metadata server for the ceph distributed file system > ii libcephfs1 0.87-1~bpo70+1 amd64 Ceph distributed file system client library > ii libcurl3-gnutls:amd64 7.29.0-1~bpo70+1.ceph amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) > ii python-ceph 0.87-1~bpo70+1 amd64 Python libraries for the Ceph distributed filesystem > > > > There are all steps to reproduce (excepting creation of pools): > > > > superuser@admin:~$ ceph osd pool get ec_backup-storage erasure_code_profile > erasure_code_profile: default > superuser@admin:~$ ceph osd erasure-code-profile get default > directory=/usr/lib/ceph/erasure-code > k=2 > m=1 > plugin=jerasure > technique=reed_sol_van > > > *********** ADMIN NODE OPERATIONS ************ > > superuser@admin:~$ ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 242T 224T 6092G 2.46 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > ec_backup-storage 4 0 0 147T 0 > cache 5 0 0 185G 0 > block-devices 6 1948G 0.79 75638G 498771 > superuser@admin:~$ rados df > pool name category KB objects clones degraded unfound rd rd KB wr wr KB > block-devices - 2042805201 498771 0 0 0 67127 259320535 2070571 2403248346 > cache - 0 0 0 0 0 60496 247235411 966553 499544074 > ec_backup-storage - 0 0 0 0 0 156988 537227276 400355 819838985 > total used 6388431372 498771 > total avail 240559782780 > total space 260163775608 > > ***** 'cache' pool is replicated pool, 'ec_backup-storage' - Erasure Encoded pool ***** > ***** running simple script for enabling cache tiering: > > > ***** There is my simple script for enabling cache tier: > > superuser@admin:~$ ./enable_cache_tier.sh cache ec_backup-storage > pool 'cache' is now (or already was) a tier of 'ec_backup-storage' > set cache-mode for pool 'cache' to writeback > overlay for 'ec_backup-storage' is now (or already was) 'cache' > set pool 5 hit_set_type to bloom > set pool 5 cache_target_dirty_ratio to 0.4 > set pool 5 cache_target_full_ratio to 0.7 > set pool 5 cache_min_flush_age to 10 > set pool 5 cache_min_evict_age to 10 > > ***** Displaying some cache pool parameters: > > superuser@admin:~$ for param in cache_target_dirty_ratio cache_target_full_ratio target_max_bytes target_max_objects cache_min_flush_age cache_min_evict_age; do ceph osd pool get cache $param; done > cache_target_dirty_ratio: 0.4 > cache_target_full_ratio: 0.7 > target_max_bytes: 0 > target_max_objects: 0 > cache_min_flush_age: 10 > cache_min_evict_age: 10 > > > *********** END ADMIN NODE OPERATIONS ************ > > *********** CEPH CLIENT OPERATIONS ************ > > superuser@share:~$ rbd create -p ec_backup-storage ec_image.img --size 500000 --image-format 2 > superuser@share:~$ rbd -p ec_backup-storage ls > ec_image.img > superuser@share:~$ sudo rbd map -p ec_backup-storage ec_image.img > /dev/rbd0 > superuser@share:~$ rbd showmapped > id pool image snap device > 0 ec_backup-storage ec_image.img - /dev/rbd0 > superuser@share:~$ sudo parted /dev/rbd0 p > Error: /dev/rbd0: unrecognised disk label > Model: Unknown (unknown) > Disk /dev/rbd0: 524GB > Sector size (logical/physical): 512B/512B > Partition Table: unknown > Disk Flags: > > > **** now start filling the rbd disk with 400Gb of 'zeros' **** > superuser@share:~$ sudo dd if=/dev/zero of=/dev/rbd0 bs=4M count=100000 oflag=direct > ^C39901+0 записей получено ; There I'm pressed CTRL+C because already getting warnings in cpeh -s (see below) > 39901+0 записей отправлено > скопировано 167356923904 байта (167 GB), 3387,59 c, 49,4 MB/c > > > *********** END CEPH CLIENT OPERATIONS ************ > > > *********** ADMIN NODE OPERATIONS ************ > > > **** Now check ceph df: 145G of 172G 'cache' pool it is about 90% is occupied already! > superuser@admin:~$ ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 242T 223T 6555G 2.64 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > ec_backup-storage 4 0 0 147T 0 > cache 5 155G 0.06 12333M 39906 > block-devices 6 1948G 0.79 75642G 498771 > > > ***** ec_backup-storage - tier pool for cold data still empty???? > ***** but in the same time: > superuser@admin:~$ ceph health detail > HEALTH_WARN 3 near full osd(s) > osd.45 is near full at 88% > osd.95 is near full at 87% > osd.100 is near full at 86% > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |