-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Try:
rados -p {cachepool} cache-flush-evict-all
and see if the objects clean up.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Nov 3, 2015 at 8:02 AM, Gregory Farnum wrote:
> When you have a caching pool in writeback mode, updates to objects
> (including deletes) are handled by writeback rather than writethrough.
> Since there's no other activity against these pools, there is nothing
> prompting the cache pool to flush updates out to the backing pool, so
> the backing pool hasn't deleted its objects because nothing's told it
> to. You'll find that the cache pool has deleted the data for its
> objects, but it's keeping around a small "whiteout" and the object
> info metadata.
> The "rados ls" you're using has never played nicely with cache tiering
> and probably never will. :( Listings are expensive operations and
> modifying them to do more than the simple info scan would be fairly
> expensive in terms of computation and IO.
>
> I think there are some caching commands you can send to flush updates
> which would cause the objects to be entirely deleted, but I don't have
> them off-hand. You can probably search the mailing list archives or
> the docs for tiering commands. :)
> -Greg
>
> On Tue, Nov 3, 2015 at 12:40 AM, Дмитрий Глушенок wrote:
>> Hi,
>>
>> While benchmarking tiered pool using rados bench it was noticed that objects are not being removed after test.
>>
>> Test was performed using "rados -p rbd bench 3600 write". The pool is not used by anything else.
>>
>> Just before end of test:
>> POOLS:
>> NAME ID USED %USED MAX AVAIL OBJECTS
>> rbd-cache 36 33110M 3.41 114G 8366
>> rbd 37 43472M 4.47 237G 10858
>>
>> Some time later (few hundreds of writes are flushed, rados automatic cleanup finished):
>> POOLS:
>> NAME ID USED %USED MAX AVAIL OBJECTS
>> rbd-cache 36 22998 0 157G 16342
>> rbd 37 46050M 4.74 234G 11503
>>
>> # rados -p rbd-cache ls | wc -l
>> 16242
>> # rados -p rbd ls | wc -l
>> 11503
>> #
>>
>> # rados -p rbd cleanup
>> error during cleanup: -2
>> error 2: (2) No such file or directory
>> #
>>
>> # rados -p rbd cleanup --run-name "" --prefix prefix ""
>> Warning: using slow linear search
>> Removed 0 objects
>> #
>>
>> # rados -p rbd ls | head -5
>> benchmark_data_dropbox01.tzk_7641_object10901
>> benchmark_data_dropbox01.tzk_7641_object9645
>> benchmark_data_dropbox01.tzk_7641_object10389
>> benchmark_data_dropbox01.tzk_7641_object10090
>> benchmark_data_dropbox01.tzk_7641_object11204
>> #
>>
>> # rados -p rbd-cache ls | head -5
>> benchmark_data_dropbox01.tzk_7641_object10901
>> benchmark_data_dropbox01.tzk_7641_object9645
>> benchmark_data_dropbox01.tzk_7641_object10389
>> benchmark_data_dropbox01.tzk_7641_object5391
>> benchmark_data_dropbox01.tzk_7641_object10090
>> #
>>
>> So, it looks like the objects are still in place (in both pools?). But it is not possible to remove them:
>>
>> # rados -p rbd rm benchmark_data_dropbox01.tzk_7641_object10901
>> error removing rbd>benchmark_data_dropbox01.tzk_7641_object10901: (2) No such file or directory
>> #
>>
>> # ceph health
>> HEALTH_OK
>> #
>>
>>
>> Can somebody explain the behavior? And is it possible to cleanup the benchmark data without recreating the pools?
>>
>>
>> ceph version 0.94.5
>>
>> # ceph osd dump | grep rbd
>> pool 36 'rbd-cache' replicated size 3 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 100 pgp_num 100 last_change 755 flags hashpspool,incomplete_clones tier_of 37 cache_mode writeback target_bytes 107374182400 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 stripe_width 0
>> pool 37 'rbd' erasure size 5 min_size 3 crush_ruleset 2 object_hash rjenkins pg_num 100 pgp_num 100 last_change 745 lfor 745 flags hashpspool tiers 36 read_tier 36 write_tier 36 stripe_width 4128
>> #
>>
>> # ceph osd pool get rbd-cache hit_set_type
>> hit_set_type: bloom
>> # ceph osd pool get rbd-cache hit_set_period
>> hit_set_period: 3600
>> # ceph osd pool get rbd-cache hit_set_count
>> hit_set_count: 1
>> # ceph osd pool get rbd-cache target_max_objects
>> target_max_objects: 0
>> # ceph osd pool get rbd-cache target_max_bytes
>> target_max_bytes: 107374182400
>> # ceph osd pool get rbd-cache cache_target_dirty_ratio
>> cache_target_dirty_ratio: 0.1
>> # ceph osd pool get rbd-cache cache_target_full_ratio
>> cache_target_full_ratio: 0.2
>> #
>>
>> Crush map:
>> root cache_tier {
>> id -7 # do not change unnecessarily
>> # weight 0.450
>> alg straw
>> hash 0 # rjenkins1
>> item osd.0 weight 0.090
>> item osd.1 weight 0.090
>> item osd.2 weight 0.090
>> item osd.3 weight 0.090
>> item osd.4 weight 0.090
>> }
>> root store_tier {
>> id -8 # do not change unnecessarily
>> # weight 0.450
>> alg straw
>> hash 0 # rjenkins1
>> item osd.5 weight 0.090
>> item osd.6 weight 0.090
>> item osd.7 weight 0.090
>> item osd.8 weight 0.090
>> item osd.9 weight 0.090
>> }
>> rule cache {
>> ruleset 1
>> type replicated
>> min_size 0
>> max_size 5
>> step take cache_tier
>> step chooseleaf firstn 0 type osd
>> step emit
>> }
>> rule store {
>> ruleset 2
>> type erasure
>> min_size 0
>> max_size 5
>> step take store_tier
>> step chooseleaf firstn 0 type osd
>> step emit
>> }
>>
>> Thanks
>>
>> --
>> Dmitry Glushenok
>> Jet Infosystems
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.2.3
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJWOOqoCRDmVDuy+mK58QAAScEP/jdBEK0lH2u38S7hOwll
FA2J5+9lY0QAzxyTaRIzsZu0g+LSrLek39fFMTX/zHI0TMfSTM6dnPs6ucO2
X59wbIyJQPEzzWEa2qn4F0j/QmWkMGfuMKDjxyT6OudpZtQKDS8mt13rnqAc
QH+0bQaVfjbKGowCAHyJXTsPK4qgew7dV3JDW2hVX/vIDjYAJomkF8Ll4miT
IQ6ViV2+9u4uW99ty4aSRnYUwaf7vqycK+qUT0Uohi6iTeym7s78O42Qa0p2
WYHcXzAdYnBiR+qTIVWvKXn81tm4gmP4lSh0gpRoJ007c0hu5vTAnUvHRh0Z
070NTrmAAJXN0oZ7lkoksYZVkXDJkwBZpdif69OQU3No/HhcY9JtagEMCXcc
7/bUACaKjyKRzRmT3VNPQuMI0ix+tdi3PU4dL+16eBO832BNsqnyNHPxu570
su1m4UQVGmoXCTUOeXYe9j4jzlHO/QRXcp/soFW5DgYO6JmylZzbyNmHPjMx
CiOsdhnjAylG/zq42S4zTfd+F//aRODGJ0JNHmQYm7M2sezAvQD1HyBCAwds
VfyOcfZwyeUNubtqssmOQ+n8/EDQciK4RH/hxG8bC8xsZaNgum7Z4/zA+Efc
gMuplOsBECODAoBlfA2TP3/XixzTXoVGMmdXolOhs+Z+tT+O22eKUEK7GbMG
rIWX
=qbDR
-----END PGP SIGNATURE-----
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
