Ok, that was useful. I guess explanation then could be that we have device classes for HDD and SSD https://ceph.com/community/new-luminous-crush-device-classes/ that adds to GLOBAL AVAIL OSDs are ballanced more or less evenly USE% = 54-70%. So more reliable number is per pool AVAIL as if that runs out - pool cannot expand. Ugis trešd., 2018. g. 10. okt., plkst. 17:49 — lietotājs Sage Weil (<sage@xxxxxxxxxxxx>) rakstīja: > > On Wed, 10 Oct 2018, Ugis wrote: > > Hi, > > > > ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) > > > > Cannot understand why ceph shows only 17TiB "MAX AVAIL" for pools > > while GLOBAL AVAIL=105TiB. > > What I expect is that for pools with replica count 3 MAX AVAIL should > > be roughly GLOBAL AVAIL/3 ~ 35TiB > > > > In excerpt below pool xxxxxxxxxxxx has replica count 2, the rest have 3 > > The 2x vs 3x is the reason you see 17 TiB vs 25 TiB. The reason it is so > much lower than global avail is harder to see from the info below, though. > The per-pool avail is calculated by looking at the OSDs touched by that > pool and calculating which will fill up first (as that practially limits > how much you can store), while global avail just adds up free space > everywhere. So maybe your osds are imbalanced, or maybe you have rules > for those pools that only point to a subset of the OSDs in the system? > > sage > > > > > > # ceph df detail > > GLOBAL: > > SIZE AVAIL RAW USED %RAW USED OBJECTS > > 250 TiB 105 TiB 145 TiB 57.98 12.71 M > > POOLS: > > NAME ID QUOTA OBJECTS QUOTA > > BYTES USED %USED MAX AVAIL OBJECTS DIRTY > > READ WRITE RAW USED > > xxxxxxxxxxxx 9 N/A N/A > > 1.0 GiB 0 25 TiB 298 298 85 > > KiB 9.9 KiB 2.0 GiB > > ssssssss 17 N/A N/A > > 10 TiB 37.72 17 TiB 2621445 2.62 M 471 > > MiB 367 MiB 30 TiB > > eeeeeeeee 47 N/A N/A > > 532 B 0 17 TiB 1 1 1 > > B 9 B 1.6 KiB > > yyyyyyyyyyyyyyyyy 48 N/A N/A > > 2.1 TiB 11.36 17 TiB 554933 554.9 k 194 > > MiB 39 MiB 6.4 TiB > > ...... > > > > Any ideas where space is hidden or I wrongly interpret GLOBAL AVAIL? > > > > Best ragards > > Ugis > > > >
