On Wed, 10 Oct 2018, Ugis wrote: > Ok, that was useful. > I guess explanation then could be that we have device classes for HDD > and SSD https://ceph.com/community/new-luminous-crush-device-classes/ Bingo. :) > that adds to GLOBAL AVAIL > OSDs are ballanced more or less evenly USE% = 54-70%. > > So more reliable number is per pool AVAIL as if that runs out - pool > cannot expand. Right. If you can point to places in the documentation or in how the output is presented that can be improved to avoid this confusion, let us know! sage > > Ugis > > > trešd., 2018. g. 10. okt., plkst. 17:49 — lietotājs Sage Weil > (<sage@xxxxxxxxxxxx>) rakstīja: > > > > On Wed, 10 Oct 2018, Ugis wrote: > > > Hi, > > > > > > ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) > > > > > > Cannot understand why ceph shows only 17TiB "MAX AVAIL" for pools > > > while GLOBAL AVAIL=105TiB. > > > What I expect is that for pools with replica count 3 MAX AVAIL should > > > be roughly GLOBAL AVAIL/3 ~ 35TiB > > > > > > In excerpt below pool xxxxxxxxxxxx has replica count 2, the rest have 3 > > > > The 2x vs 3x is the reason you see 17 TiB vs 25 TiB. The reason it is so > > much lower than global avail is harder to see from the info below, though. > > The per-pool avail is calculated by looking at the OSDs touched by that > > pool and calculating which will fill up first (as that practially limits > > how much you can store), while global avail just adds up free space > > everywhere. So maybe your osds are imbalanced, or maybe you have rules > > for those pools that only point to a subset of the OSDs in the system? > > > > sage > > > > > > > > > > # ceph df detail > > > GLOBAL: > > > SIZE AVAIL RAW USED %RAW USED OBJECTS > > > 250 TiB 105 TiB 145 TiB 57.98 12.71 M > > > POOLS: > > > NAME ID QUOTA OBJECTS QUOTA > > > BYTES USED %USED MAX AVAIL OBJECTS DIRTY > > > READ WRITE RAW USED > > > xxxxxxxxxxxx 9 N/A N/A > > > 1.0 GiB 0 25 TiB 298 298 85 > > > KiB 9.9 KiB 2.0 GiB > > > ssssssss 17 N/A N/A > > > 10 TiB 37.72 17 TiB 2621445 2.62 M 471 > > > MiB 367 MiB 30 TiB > > > eeeeeeeee 47 N/A N/A > > > 532 B 0 17 TiB 1 1 1 > > > B 9 B 1.6 KiB > > > yyyyyyyyyyyyyyyyy 48 N/A N/A > > > 2.1 TiB 11.36 17 TiB 554933 554.9 k 194 > > > MiB 39 MiB 6.4 TiB > > > ...... > > > > > > Any ideas where space is hidden or I wrongly interpret GLOBAL AVAIL? > > > > > > Best ragards > > > Ugis > > > > > > > >
