When you have a caching pool in writeback mode, updates to objects
(including deletes) are handled by writeback rather than writethrough.
Since there's no other activity against these pools, there is nothing
prompting the cache pool to flush updates out to the backing pool, so
the backing pool hasn't deleted its objects because nothing's told it
to. You'll find that the cache pool has deleted the data for its
objects, but it's keeping around a small "whiteout" and the object
info metadata.
The "rados ls" you're using has never played nicely with cache tiering
and probably never will. :( Listings are expensive operations and
modifying them to do more than the simple info scan would be fairly
expensive in terms of computation and IO.
I think there are some caching commands you can send to flush updates
which would cause the objects to be entirely deleted, but I don't have
them off-hand. You can probably search the mailing list archives or
the docs for tiering commands. :)
-Greg
On Tue, Nov 3, 2015 at 12:40 AM, Дмитрий Глушенок <glush@xxxxxxxxxx> wrote:
> Hi,
>
> While benchmarking tiered pool using rados bench it was noticed that objects are not being removed after test.
>
> Test was performed using "rados -p rbd bench 3600 write". The pool is not used by anything else.
>
> Just before end of test:
> POOLS:
> NAME ID USED %USED MAX AVAIL OBJECTS
> rbd-cache 36 33110M 3.41 114G 8366
> rbd 37 43472M 4.47 237G 10858
>
> Some time later (few hundreds of writes are flushed, rados automatic cleanup finished):
> POOLS:
> NAME ID USED %USED MAX AVAIL OBJECTS
> rbd-cache 36 22998 0 157G 16342
> rbd 37 46050M 4.74 234G 11503
>
> # rados -p rbd-cache ls | wc -l
> 16242
> # rados -p rbd ls | wc -l
> 11503
> #
>
> # rados -p rbd cleanup
> error during cleanup: -2
> error 2: (2) No such file or directory
> #
>
> # rados -p rbd cleanup --run-name "" --prefix prefix ""
> Warning: using slow linear search
> Removed 0 objects
> #
>
> # rados -p rbd ls | head -5
> benchmark_data_dropbox01.tzk_7641_object10901
> benchmark_data_dropbox01.tzk_7641_object9645
> benchmark_data_dropbox01.tzk_7641_object10389
> benchmark_data_dropbox01.tzk_7641_object10090
> benchmark_data_dropbox01.tzk_7641_object11204
> #
>
> # rados -p rbd-cache ls | head -5
> benchmark_data_dropbox01.tzk_7641_object10901
> benchmark_data_dropbox01.tzk_7641_object9645
> benchmark_data_dropbox01.tzk_7641_object10389
> benchmark_data_dropbox01.tzk_7641_object5391
> benchmark_data_dropbox01.tzk_7641_object10090
> #
>
> So, it looks like the objects are still in place (in both pools?). But it is not possible to remove them:
>
> # rados -p rbd rm benchmark_data_dropbox01.tzk_7641_object10901
> error removing rbd>benchmark_data_dropbox01.tzk_7641_object10901: (2) No such file or directory
> #
>
> # ceph health
> HEALTH_OK
> #
>
>
> Can somebody explain the behavior? And is it possible to cleanup the benchmark data without recreating the pools?
>
>
> ceph version 0.94.5
>
> # ceph osd dump | grep rbd
> pool 36 'rbd-cache' replicated size 3 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 100 pgp_num 100 last_change 755 flags hashpspool,incomplete_clones tier_of 37 cache_mode writeback target_bytes 107374182400 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 stripe_width 0
> pool 37 'rbd' erasure size 5 min_size 3 crush_ruleset 2 object_hash rjenkins pg_num 100 pgp_num 100 last_change 745 lfor 745 flags hashpspool tiers 36 read_tier 36 write_tier 36 stripe_width 4128
> #
>
> # ceph osd pool get rbd-cache hit_set_type
> hit_set_type: bloom
> # ceph osd pool get rbd-cache hit_set_period
> hit_set_period: 3600
> # ceph osd pool get rbd-cache hit_set_count
> hit_set_count: 1
> # ceph osd pool get rbd-cache target_max_objects
> target_max_objects: 0
> # ceph osd pool get rbd-cache target_max_bytes
> target_max_bytes: 107374182400
> # ceph osd pool get rbd-cache cache_target_dirty_ratio
> cache_target_dirty_ratio: 0.1
> # ceph osd pool get rbd-cache cache_target_full_ratio
> cache_target_full_ratio: 0.2
> #
>
> Crush map:
> root cache_tier {
> id -7 # do not change unnecessarily
> # weight 0.450
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 0.090
> item osd.1 weight 0.090
> item osd.2 weight 0.090
> item osd.3 weight 0.090
> item osd.4 weight 0.090
> }
> root store_tier {
> id -8 # do not change unnecessarily
> # weight 0.450
> alg straw
> hash 0 # rjenkins1
> item osd.5 weight 0.090
> item osd.6 weight 0.090
> item osd.7 weight 0.090
> item osd.8 weight 0.090
> item osd.9 weight 0.090
> }
> rule cache {
> ruleset 1
> type replicated
> min_size 0
> max_size 5
> step take cache_tier
> step chooseleaf firstn 0 type osd
> step emit
> }
> rule store {
> ruleset 2
> type erasure
> min_size 0
> max_size 5
> step take store_tier
> step chooseleaf firstn 0 type osd
> step emit
> }
>
> Thanks
>
> --
> Dmitry Glushenok
> Jet Infosystems
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
