There are a couple gaps, yes: https://termbin.com/9mx1 What should I do? -- dan On Thu, Nov 26, 2020 at 7:52 PM Igor Fedotov <ifedotov@xxxxxxx> wrote: > > Does "ceph osd df tree" show stats properly (I mean there are no evident > gaps like unexpected zero values) for all the daemons? > > > > 1. Anyway, I found something weird... > > > > I created a new 1-PG pool "foo" on a different cluster and wrote some > > data to it. > > > > The stored and used are equal. > > > > Thu 26 Nov 19:26:58 CET 2020 > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 > > TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.31 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED MAX AVAIL > > public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB > > test 71 29 MiB 6.56k 29 MiB 0 269 TiB > > foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB > > > > But I tried restarting the relevant three OSDs, and the bytes_used are > > temporarily reported correctly: > > > > Thu 26 Nov 19:27:00 CET 2020 > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 > > TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.62 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED MAX AVAIL > > public 68 2.9 PiB 143.54M 4.3 PiB 84.55 538 TiB > > test 71 29 MiB 6.56k 1.2 GiB 0 269 TiB > > foo 72 1.2 GiB 308 3.6 GiB 0 269 TiB > > > > But then a few seconds later it's back to used == stored: > > > > Thu 26 Nov 19:27:03 CET 2020 > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 > > TOTAL 5.5 PiB 1.2 PiB 4.3 PiB 4.3 PiB 78.47 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED MAX AVAIL > > public 68 2.9 PiB 143.54M 2.9 PiB 78.49 538 TiB > > test 71 29 MiB 6.56k 29 MiB 0 269 TiB > > foo 72 1.2 GiB 308 1.2 GiB 0 269 TiB > > > > It seems to report the correct stats only when the PG is peering (so > > some other transition state). > > I've restarted all three relevant OSDs now -- the stats are reported > > as stored == used. > > > > 2. Another data point -- I found another old cluster that reports > > stored/used correctly. I have no idea what might be different about > > that cluster -- we updated it just like the others. > > > > Cheers, Dan > > > > On Thu, Nov 26, 2020 at 6:22 PM Igor Fedotov <ifedotov@xxxxxxx> wrote: > >> For specific BlueStore instance you can learn relevant statfs output by > >> > >> setting debug_bluestore to 20 and leaving OSD for 5-10 seconds (or may > >> be a couple of minutes - don't remember exact statsfs poll period ). > >> > >> Then grep osd log for "statfs" and/or "pool_statfs" and get the output > >> formatted as per the following operator (taken from src/osd/osd_types.cc): > >> > >> ostream& operator<<(ostream& out, const store_statfs_t &s) > >> { > >> out << std::hex > >> << "store_statfs(0x" << s.available > >> << "/0x" << s.internally_reserved > >> << "/0x" << s.total > >> << ", data 0x" << s.data_stored > >> << "/0x" << s.allocated > >> << ", compress 0x" << s.data_compressed > >> << "/0x" << s.data_compressed_allocated > >> << "/0x" << s.data_compressed_original > >> << ", omap 0x" << s.omap_allocated > >> << ", meta 0x" << s.internal_metadata > >> << std::dec > >> << ")"; > >> return out; > >> } > >> > >> But honestly I doubt this is BlueStore which reports incorrectly since > >> it doesn't care about replication. > >> > >> It rather looks like lack of stats from some replicas or improper pg > >> replica factor processing... > >> > >> Perhaps legacy vs. new pool what matters... Can you try to create a new > >> pool at old cluster and fill it with some data (e.g. just a single 64K > >> object) and check the stats? > >> > >> > >> Thanks, > >> > >> Igor > >> > >> On 11/26/2020 8:00 PM, Dan van der Ster wrote: > >>> Hi Igor, > >>> > >>> No BLUESTORE_LEGACY_STATFS warning, and > >>> bluestore_warn_on_legacy_statfs is the default true on this (and all) > >>> clusters. > >>> I'm quite sure we did the statfs conversion during one of the recent > >>> upgrades (I forget which one exactly). > >>> > >>> # ceph tell osd.* config get bluestore_warn_on_legacy_statfs | grep -v true > >>> # > >>> > >>> Is there a command to see the statfs reported by an individual OSD ? > >>> We have a mix of ~year old and recently recreated OSDs, so I could try > >>> to see if they differ. > >>> > >>> Thanks! > >>> > >>> Dan > >>> > >>> > >>> On Thu, Nov 26, 2020 at 5:50 PM Igor Fedotov <ifedotov@xxxxxxx> wrote: > >>>> Hi Dan > >>>> > >>>> don't you have BLUESTORE_LEGACY_STATFS alert raised (might be silenced > >>>> by bluestore_warn_on_legacy_statfs param) for the older cluster? > >>>> > >>>> > >>>> Thanks, > >>>> > >>>> Igor > >>>> > >>>> > >>>> On 11/26/2020 7:29 PM, Dan van der Ster wrote: > >>>>> Hi, > >>>>> > >>>>> Depending on which cluster I look at (all running v14.2.11), the > >>>>> bytes_used is reporting raw space or stored bytes variably. > >>>>> > >>>>> Here's a 7 year old cluster: > >>>>> > >>>>> # ceph df -f json | jq .pools[0] > >>>>> { > >>>>> "name": "volumes", > >>>>> "id": 4, > >>>>> "stats": { > >>>>> "stored": 1229308190855881, > >>>>> "objects": 294401604, > >>>>> "kb_used": 1200496280133, > >>>>> "bytes_used": 1229308190855881, > >>>>> "percent_used": 0.4401889145374298, > >>>>> "max_avail": 521125025021952 > >>>>> } > >>>>> } > >>>>> > >>>>> Note that stored == bytes_used for that pool. (this is a 3x replica pool). > >>>>> > >>>>> But here's a newer cluster (installed recently with nautilus) > >>>>> > >>>>> # ceph df -f json | jq .pools[0] > >>>>> { > >>>>> "name": "volumes", > >>>>> "id": 1, > >>>>> "stats": { > >>>>> "stored": 680977600893041, > >>>>> "objects": 163155803, > >>>>> "kb_used": 1995736271829, > >>>>> "bytes_used": 2043633942351985, > >>>>> "percent_used": 0.23379847407341003, > >>>>> "max_avail": 2232457428467712 > >>>>> } > >>>>> } > >>>>> > >>>>> In the second cluster, bytes_used is 3x stored. > >>>>> > >>>>> Does anyone know why these are not reported consistently? > >>>>> Noticing this just now, I'll update our monitoring to plot stored > >>>>> rather than bytes_used from now on. > >>>>> > >>>>> Thanks! > >>>>> > >>>>> Dan > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users@xxxxxxx > >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |