This commit had a number of unintended consequences. I believe the zero-detection is going to be gated behind a developer flag in all subsequent point releases. :) -Greg On Mon, May 16, 2022 at 3:43 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote: > > Hi Devs, > > I figured out the issue with ceph df mismatch on Object count vs Stored data for my issue - and posting it here in case anyone encounters it in their testing. > > The below commit modified Bluestore to not write bufferlists with zeroes as data into the pool/backend. My test objects were all created with 0s as their data, which triggered this "skipping bufferlists with zeroes as data", which is why I wasn't seeing the "Stored" data count increasing. The object count increments, but Bluestore doesn't write the zeroes to the pool/backend so the "ceph df" metrics don't see any stored data. > > So, lesson here is, create test objects with /dev/random, not /dev/zero. > > You can figure out how many bytes have NOT been written into the backend with: > ./bin/ceph daemon osd.0 perf dump | grep skipped > > Commit where bufferlists with zeroes are not written: > 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3 is the first bad commit > > commit 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3 > > Author: Laura Flores <lflores@xxxxxxxxxx> > > Date: Fri Dec 24 01:18:13 2021 +0000 > > > os/bluestore: detect unnecessary zeros in _do_write_big() > > > Bluestore's `_do_write()` method handles writing data from bufferlists. Currently, it writes data from bufferlists without checking for unnecessary zeros. The lack zero detection may negatively impact performance. > > > Best regards, > - Curt > > > > On Wed, May 11, 2022 at 1:49 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote: >> >> Hi - >> >> I'm working with the Quincy tag (commit: 6551f450487afe6c3f7bdd7a7c9c891e447045ed) and I vstart a cluster to do RGW testing. >> >> I put a 8 objects into my "test_bucket" which goes to default.rgw.buckets.data and when I do: >> ./bin/ceph df detail >> I see the object count increased correctly, but df reports "0 B" stored. >> >> [ubuntu-validation-01][~/source/ceph/build] (quincy)proxy$ ./bin/ceph df detail >> *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** >> 2022-05-11T13:44:14.824-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled. >> 2022-05-11T13:44:14.832-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled. >> --- RAW STORAGE --- >> CLASS SIZE AVAIL USED RAW USED %RAW USED >> qlc 303 GiB 300 GiB 3.0 GiB 3.0 GiB 1.00 >> ssd 303 GiB 300 GiB 3.0 GiB 3.0 GiB 0.99 >> TOTAL 606 GiB 600 GiB 6.0 GiB 6.0 GiB 0.99 >> >> --- POOLS --- >> POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR >> .rgw.root 1 1 1.4 KiB 1.4 KiB 0 B 4 48 KiB 48 KiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B >> default.rgw.log 2 1 7.5 KiB 7.5 KiB 0 B 179 4.8 MiB 4.8 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B >> default.rgw.control 3 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B >> default.rgw.meta 4 1 5.3 KiB 5.3 KiB 0 B 21 2.4 MiB 2.4 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B >> default.rgw.buckets.index 5 1 0 B 0 B 0 B 11 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B >> default.rgw.buckets.data 6 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 99 GiB N/A N/A N/A 0 B 0 B >> >> I can get object_stats and see their size is definitely non-zero. >> >> Did anything change in Quincy with df reporting? >> >> Thanks, >> - Curt > > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
