On Wed, 10 Oct 2018, Sage Weil wrote: > On Wed, 10 Oct 2018, Ugis wrote: > > Ok, that was useful. > > I guess explanation then could be that we have device classes for HDD > > and SSD https://ceph.com/community/new-luminous-crush-device-classes/ > > Bingo. :) > > > that adds to GLOBAL AVAIL > > OSDs are ballanced more or less evenly USE% = 54-70%. > > > > So more reliable number is per pool AVAIL as if that runs out - pool > > cannot expand. > > Right. If you can point to places in the documentation or in how the > output is presented that can be improved to avoid this confusion, let us > know! Hmm, I wonder if that GLOBAL section could be supplemented with a view based on the CRUSH root(s) (usually there is just one, 'default'), and the device classes, and breaks down usage that way. sage > > sage > > > > > Ugis > > > > > > trešd., 2018. g. 10. okt., plkst. 17:49 — lietotājs Sage Weil > > (<sage@xxxxxxxxxxxx>) rakstīja: > > > > > > On Wed, 10 Oct 2018, Ugis wrote: > > > > Hi, > > > > > > > > ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) > > > > > > > > Cannot understand why ceph shows only 17TiB "MAX AVAIL" for pools > > > > while GLOBAL AVAIL=105TiB. > > > > What I expect is that for pools with replica count 3 MAX AVAIL should > > > > be roughly GLOBAL AVAIL/3 ~ 35TiB > > > > > > > > In excerpt below pool xxxxxxxxxxxx has replica count 2, the rest have 3 > > > > > > The 2x vs 3x is the reason you see 17 TiB vs 25 TiB. The reason it is so > > > much lower than global avail is harder to see from the info below, though. > > > The per-pool avail is calculated by looking at the OSDs touched by that > > > pool and calculating which will fill up first (as that practially limits > > > how much you can store), while global avail just adds up free space > > > everywhere. So maybe your osds are imbalanced, or maybe you have rules > > > for those pools that only point to a subset of the OSDs in the system? > > > > > > sage > > > > > > > > > > > > > > # ceph df detail > > > > GLOBAL: > > > > SIZE AVAIL RAW USED %RAW USED OBJECTS > > > > 250 TiB 105 TiB 145 TiB 57.98 12.71 M > > > > POOLS: > > > > NAME ID QUOTA OBJECTS QUOTA > > > > BYTES USED %USED MAX AVAIL OBJECTS DIRTY > > > > READ WRITE RAW USED > > > > xxxxxxxxxxxx 9 N/A N/A > > > > 1.0 GiB 0 25 TiB 298 298 85 > > > > KiB 9.9 KiB 2.0 GiB > > > > ssssssss 17 N/A N/A > > > > 10 TiB 37.72 17 TiB 2621445 2.62 M 471 > > > > MiB 367 MiB 30 TiB > > > > eeeeeeeee 47 N/A N/A > > > > 532 B 0 17 TiB 1 1 1 > > > > B 9 B 1.6 KiB > > > > yyyyyyyyyyyyyyyyy 48 N/A N/A > > > > 2.1 TiB 11.36 17 TiB 554933 554.9 k 194 > > > > MiB 39 MiB 6.4 TiB > > > > ...... > > > > > > > > Any ideas where space is hidden or I wrongly interpret GLOBAL AVAIL? > > > > > > > > Best ragards > > > > Ugis > > > > > > > > > > > >
