Re: RGW: vstart and ceph df detail showing 0 B stored

[Curt Bruns <curt.e.bruns@xxxxxxxxx>]

Hi Laura and Greg,

Thanks for the response and further explanation. It does make sense why it was implemented for scaling teuthology tests and not impacting/filling backend devices. I totally agree on gating it with a config option.

Thanks,

- Curt

On Mon, May 16, 2022 at 4:14 PM Laura Flores <lflores@xxxxxxxxxx> wrote:

Hi Curt,

Yes, to elaborate on Greg's reply, we will be turning this feature off by default in the next point release. The "bluestore zero block detection" feature was intended for scale testing in teuthology without filling the devices, but it negatively affects some RBD and CephFS features. A global config option to turn this feature off by default is being worked on in https://github.com/ceph/ceph/pull/46193, with the intent of having it off by default on all clusters, but still in use for some teuthology tests.

Let me know if you have any more questions.

- Laura Flores

On Mon, May 16, 2022 at 5:58 PM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:

This commit had a number of unintended consequences. I believe the
zero-detection is going to be gated behind a developer flag in all
subsequent point releases. :)
-Greg

On Mon, May 16, 2022 at 3:43 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote:
>
> Hi Devs,
>
> I figured out the issue with ceph df mismatch on Object count vs Stored data for my issue - and posting it here in case anyone encounters it in their testing.
>
> The below commit modified Bluestore to not write bufferlists with zeroes as data into the pool/backend. My test objects were all created with 0s as their data, which triggered this "skipping bufferlists with zeroes as data", which is why I wasn't seeing the "Stored" data count increasing. The object count increments, but Bluestore doesn't write the zeroes to the pool/backend so the "ceph df" metrics don't see any stored data.
>
> So, lesson here is, create test objects with /dev/random, not /dev/zero.
>
> You can figure out how many bytes have NOT been written into the backend with:
> ./bin/ceph daemon osd.0 perf dump | grep skipped
>
> Commit where bufferlists with zeroes are not written:
> 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3 is the first bad commit
>
> commit 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3
>
> Author: Laura Flores <lflores@xxxxxxxxxx>
>
> Date:   Fri Dec 24 01:18:13 2021 +0000
>
>
>     os/bluestore: detect unnecessary zeros in _do_write_big()
>
>
>     Bluestore's `_do_write()` method handles writing data from bufferlists. Currently, it writes data from bufferlists without checking for unnecessary zeros. The lack zero detection may negatively impact performance.
>
>
> Best regards,
> - Curt
>
>
>
> On Wed, May 11, 2022 at 1:49 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote:
>>
>> Hi -
>>
>> I'm working with the Quincy tag (commit: 6551f450487afe6c3f7bdd7a7c9c891e447045ed) and I vstart a cluster to do RGW testing.
>>
>> I put a 8 objects into my "test_bucket" which goes to default.rgw.buckets.data and when I do:
>> ./bin/ceph df detail
>> I see the object count increased correctly, but df reports "0 B" stored.
>>
>> [ubuntu-validation-01][~/source/ceph/build] (quincy)proxy$ ./bin/ceph df detail
>> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
>> 2022-05-11T13:44:14.824-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
>> 2022-05-11T13:44:14.832-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
>> --- RAW STORAGE ---
>> CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
>> qlc    303 GiB  300 GiB  3.0 GiB   3.0 GiB       1.00
>> ssd    303 GiB  300 GiB  3.0 GiB   3.0 GiB       0.99
>> TOTAL  606 GiB  600 GiB  6.0 GiB   6.0 GiB       0.99
>>
>> --- POOLS ---
>> POOL                       ID  PGS   STORED   (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
>> .rgw.root                   1    1  1.4 KiB  1.4 KiB     0 B        4   48 KiB   48 KiB     0 B      0    198 GiB            N/A          N/A    N/A         0 B          0 B
>> default.rgw.log             2    1  7.5 KiB  7.5 KiB     0 B      179  4.8 MiB  4.8 MiB     0 B      0    198 GiB            N/A          N/A    N/A         0 B          0 B
>> default.rgw.control         3    1      0 B      0 B     0 B        8      0 B      0 B     0 B      0    198 GiB            N/A          N/A    N/A         0 B          0 B
>> default.rgw.meta            4    1  5.3 KiB  5.3 KiB     0 B       21  2.4 MiB  2.4 MiB     0 B      0    198 GiB            N/A          N/A    N/A         0 B          0 B
>> default.rgw.buckets.index   5    1      0 B      0 B     0 B       11      0 B      0 B     0 B      0    198 GiB            N/A          N/A    N/A         0 B          0 B
>> default.rgw.buckets.data    6    1      0 B      0 B     0 B        8      0 B      0 B     0 B      0     99 GiB            N/A          N/A    N/A         0 B          0 B
>>
>> I can get object_stats and see their size is definitely non-zero.
>>
>> Did anything change in Quincy with df reporting?
>>
>> Thanks,
>> - Curt
>
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx

--

Laura Flores

She/Her/Hers

Associate Software Engineer, Ceph Storage

Red Hat Inc.

La Grange Park, IL

lflores@xxxxxxxxxx   
M: +17087388804    

@RedHat   Red Hat   Red Hat

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx