Hi Devs,
I figured out the issue with ceph df mismatch on Object count vs Stored data for my issue - and posting it here in case anyone encounters it in their testing.
The below commit modified Bluestore to not write bufferlists with zeroes as data into the pool/backend. My test objects were all created with 0s as their data, which triggered this "skipping bufferlists with zeroes as data", which is why I wasn't seeing the "Stored" data count increasing. The object count increments, but Bluestore doesn't write the zeroes to the pool/backend so the "ceph df" metrics don't see any stored data.
So, lesson here is, create test objects with /dev/random, not /dev/zero.
You can figure out how many bytes have NOT been written into the backend with:
./bin/ceph daemon osd.0 perf dump | grep skipped
Commit where bufferlists with zeroes are not written:
914debdd4a5c3dd8f7b7762cd138e971a44cfdc3 is the first bad commit
commit 914debdd4a5c3dd8f7b7762cd138e971a44cfdc3
Author: Laura Flores <lflores@xxxxxxxxxx>
Date: Fri Dec 24 01:18:13 2021 +0000
os/bluestore: detect unnecessary zeros in _do_write_big()
Bluestore's `_do_write()` method handles writing data from bufferlists. Currently, it writes data from bufferlists without checking for unnecessary zeros. The lack zero detection may negatively impact performance.
Best regards,
- Curt
On Wed, May 11, 2022 at 1:49 PM Curt Bruns <curt.e.bruns@xxxxxxxxx> wrote:
Hi -I'm working with the Quincy tag (commit: 6551f450487afe6c3f7bdd7a7c9c891e447045ed) and I vstart a cluster to do RGW testing.I put a 8 objects into my "test_bucket" which goes to default.rgw.buckets.data and when I do:./bin/ceph df detailI see the object count increased correctly, but df reports "0 B" stored.[ubuntu-validation-01][~/source/ceph/build] (quincy)proxy$ ./bin/ceph df detail
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2022-05-11T13:44:14.824-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
2022-05-11T13:44:14.832-0700 7fa924c41700 -1 WARNING: all dangerous and experimental features are enabled.
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
qlc 303 GiB 300 GiB 3.0 GiB 3.0 GiB 1.00
ssd 303 GiB 300 GiB 3.0 GiB 3.0 GiB 0.99
TOTAL 606 GiB 600 GiB 6.0 GiB 6.0 GiB 0.99
--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
.rgw.root 1 1 1.4 KiB 1.4 KiB 0 B 4 48 KiB 48 KiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
default.rgw.log 2 1 7.5 KiB 7.5 KiB 0 B 179 4.8 MiB 4.8 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
default.rgw.control 3 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
default.rgw.meta 4 1 5.3 KiB 5.3 KiB 0 B 21 2.4 MiB 2.4 MiB 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
default.rgw.buckets.index 5 1 0 B 0 B 0 B 11 0 B 0 B 0 B 0 198 GiB N/A N/A N/A 0 B 0 B
default.rgw.buckets.data 6 1 0 B 0 B 0 B 8 0 B 0 B 0 B 0 99 GiB N/A N/A N/A 0 B 0 BI can get object_stats and see their size is definitely non-zero.Did anything change in Quincy with df reporting?Thanks,- Curt
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
