Hi Curt,
Yes, to elaborate on Greg's reply, we will be turning this feature off by default in the next point release. The "bluestore zero block detection" feature was intended for scale testing in teuthology without filling the devices, but it negatively affects some RBD and CephFS features. A global config option to turn this feature off by default is being worked on in https://github.com/ceph/ceph/pull/46193, with the intent of having it off by default on all clusters, but still in use for some teuthology tests.
Let me know if you have any more questions.
- Laura Flores
On Mon, May 16, 2022 at 5:58 PM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
This commit had a number of unintended consequences. I believe the
zero-detection is going to be gated behind a developer flag in all
subsequent point releases. :)
-Greg
On Mon, May 16, 2022 at 3:43 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote:
>
> Hi Devs,
>
> I figured out the issue with ceph df mismatch on Object count vs Stored data for my issue - and posting it here in case anyone encounters it in their testing.
>
> The below commit modified Bluestore to not write bufferlists with zeroes as data into the pool/backend. My test objects were all created with 0s as their data, which triggered this "skipping bufferlists with zeroes as data", which is why I wasn't seeing the "Stored" data count increasing. The object count increments, but Bluestore doesn't write the zeroes to the pool/backend so the "ceph df" metrics don't see any stored data.
>
> So, lesson here is, create test objects with /dev/random, not /dev/zero.
>
> You can figure out how many bytes have NOT been written into the backend with:
> ./bin/ceph daemon osd.0 perf dump | grep skipped
>
> Commit where bufferlists with zeroes are not written:
> 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3 is the first bad commit
>
> commit 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3
>
> Author: Laura Flores <lflores@xxxxxxxxxx>
>
> Date: Fri Dec 24 01:18:13 2021 +0000
>
>
> os/bluestore: detect unnecessary zeros in _do_write_big()
>
>
> Bluestore's `_do_write()` method handles writing data from bufferlists. Currently, it writes data from bufferlists without checking for unnecessary zeros. The lack zero detection may negatively impact performance.
>
>
> Best regards,
> - Curt
>
>
>
> On Wed, May 11, 2022 at 1:49 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote:
>>
>> Hi -
>>
>> I'm working with the Quincy tag (commit: 6551f450487afe6c3f7bdd7a7c9c891e447045ed) and I vstart a cluster to do RGW testing.
>>
>> I put a 8 objects into my "test_bucket" which goes to default.rgw.buckets.data and when I do:
>> ./bin/ceph df detail
>> I see the object count increased correctly, but df reports "0 B" stored.
>>
>> [ubuntu-validation-01][~/source/ceph/build] (quincy)proxy$ ./bin/ceph df detail
>> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
>> 2022-05-11T13:44:14.824-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
>> 2022-05-11T13:44:14.832-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
>> --- RAW STORAGE ---
>> CLASS SIZE AVAIL USED RAW USED %RAW USED
>> qlc 303 GiB 300 GiB 3.0 GiB 3.0 GiB 1.00
>> ssd 303 GiB 300 GiB 3.0 GiB 3.0 GiB 0.99
>> TOTAL 606 GiB 600 GiB 6.0 GiB 6.0 GiB 0.99
>>
>> --- POOLS ---
>> POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
>> .rgw.root 1 1 1.4 KiB 1.4 KiB 0 B 4 48 KiB 48 KiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
>> default.rgw.log 2 1 7.5 KiB 7.5 KiB 0 B 179 4.8 MiB 4.8 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
>> default.rgw.control 3 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
>> default.rgw.meta 4 1 5.3 KiB 5.3 KiB 0 B 21 2.4 MiB 2.4 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
>> default.rgw.buckets.index 5 1 0 B 0 B 0 B 11 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
>> default.rgw.buckets.data 6 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 99 GiB N/A N/A N/A 0 B 0 B
>>
>> I can get object_stats and see their size is definitely non-zero.
>>
>> Did anything change in Quincy with df reporting?
>>
>> Thanks,
>> - Curt
>
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx
--
Laura Flores
She/Her/Hers
Associate Software Engineer, Ceph Storage
La Grange Park, IL
lflores@xxxxxxxxxx
M: +17087388804
| |
| |
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
