Yes, ceph df GLOBAL section supplement with a view based on the CRUSH root(s) and the device classes - that would be useful to get better insight what is going on in cluster. Uģis ceturtd., 2018. g. 11. okt., plkst. 01:15 — lietotājs Kyle Bader (<kyle.bader@xxxxxxxxx>) rakstīja: > > +1 for output that breaks out device classes > > On Wed, Oct 10, 2018 at 11:27 Sage Weil <sage@xxxxxxxxxxxx> wrote: >> >> On Wed, 10 Oct 2018, Sage Weil wrote: >> > On Wed, 10 Oct 2018, Ugis wrote: >> > > Ok, that was useful. >> > > I guess explanation then could be that we have device classes for HDD >> > > and SSD https://ceph.com/community/new-luminous-crush-device-classes/ >> > >> > Bingo. :) >> > >> > > that adds to GLOBAL AVAIL >> > > OSDs are ballanced more or less evenly USE% = 54-70%. >> > > >> > > So more reliable number is per pool AVAIL as if that runs out - pool >> > > cannot expand. >> > >> > Right. If you can point to places in the documentation or in how the >> > output is presented that can be improved to avoid this confusion, let us >> > know! >> >> Hmm, I wonder if that GLOBAL section could be supplemented with a view >> based on the CRUSH root(s) (usually there is just one, 'default'), and the >> device classes, and breaks down usage that way. >> >> sage >> >> >> > >> > sage >> > >> > > >> > > Ugis >> > > >> > > >> > > trešd., 2018. g. 10. okt., plkst. 17:49 — lietotājs Sage Weil >> > > (<sage@xxxxxxxxxxxx>) rakstīja: >> > > > >> > > > On Wed, 10 Oct 2018, Ugis wrote: >> > > > > Hi, >> > > > > >> > > > > ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) >> > > > > >> > > > > Cannot understand why ceph shows only 17TiB "MAX AVAIL" for pools >> > > > > while GLOBAL AVAIL=105TiB. >> > > > > What I expect is that for pools with replica count 3 MAX AVAIL should >> > > > > be roughly GLOBAL AVAIL/3 ~ 35TiB >> > > > > >> > > > > In excerpt below pool xxxxxxxxxxxx has replica count 2, the rest have 3 >> > > > >> > > > The 2x vs 3x is the reason you see 17 TiB vs 25 TiB. The reason it is so >> > > > much lower than global avail is harder to see from the info below, though. >> > > > The per-pool avail is calculated by looking at the OSDs touched by that >> > > > pool and calculating which will fill up first (as that practially limits >> > > > how much you can store), while global avail just adds up free space >> > > > everywhere. So maybe your osds are imbalanced, or maybe you have rules >> > > > for those pools that only point to a subset of the OSDs in the system? >> > > > >> > > > sage >> > > > >> > > > >> > > > > >> > > > > # ceph df detail >> > > > > GLOBAL: >> > > > > SIZE AVAIL RAW USED %RAW USED OBJECTS >> > > > > 250 TiB 105 TiB 145 TiB 57.98 12.71 M >> > > > > POOLS: >> > > > > NAME ID QUOTA OBJECTS QUOTA >> > > > > BYTES USED %USED MAX AVAIL OBJECTS DIRTY >> > > > > READ WRITE RAW USED >> > > > > xxxxxxxxxxxx 9 N/A N/A >> > > > > 1.0 GiB 0 25 TiB 298 298 85 >> > > > > KiB 9.9 KiB 2.0 GiB >> > > > > ssssssss 17 N/A N/A >> > > > > 10 TiB 37.72 17 TiB 2621445 2.62 M 471 >> > > > > MiB 367 MiB 30 TiB >> > > > > eeeeeeeee 47 N/A N/A >> > > > > 532 B 0 17 TiB 1 1 1 >> > > > > B 9 B 1.6 KiB >> > > > > yyyyyyyyyyyyyyyyy 48 N/A N/A >> > > > > 2.1 TiB 11.36 17 TiB 554933 554.9 k 194 >> > > > > MiB 39 MiB 6.4 TiB >> > > > > ...... >> > > > > >> > > > > Any ideas where space is hidden or I wrongly interpret GLOBAL AVAIL? >> > > > > >> > > > > Best ragards >> > > > > Ugis >> > > > > >> > > > > >> > > >> > >
