|
Hi, I have the following configuration of OSDs: ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 hdd 5.45599 1.00000 5587G 2259G 3327G 40.45 1.10 234 1 hdd 5.45599 1.00000 5587G 2295G 3291G 41.08 1.11 231 2 hdd 5.45599 1.00000 5587G 2321G 3265G 41.56 1.13 232 3 hdd 5.45599 1.00000 5587G 2313G 3273G 41.42 1.12 234 4 hdd 5.45599 1.00000 5587G 2105G 3481G 37.68 1.02 212 5 hdd 5.45599 1.00000 5587G 2231G 3355G 39.94 1.08 218 41 ssd 0.87299 1.00000 894G 9637M 884G 1.05 0.03 31 42 ssd 0.87299 1.00000 894G 13361M 881G 1.46 0.04 41 6 hdd 5.45599 1.00000 5587G 2404G 3182G 43.03 1.17 239 7 hdd 5.45599 1.00000 5587G 2226G 3360G 39.85 1.08 222 8 hdd 5.45599 1.00000 5587G 2668G 2918G 47.76 1.30 256 9 hdd 5.45599 1.00000 5587G 2366G 3220G 42.36 1.15 236 10 hdd 5.45599 1.00000 5587G 2454G 3132G 43.92 1.19 253 11 hdd 5.45599 1.00000 5587G 2405G 3181G 43.06 1.17 245 43 ssd 0.87299 1.00000 894G 15498M 879G 1.69 0.05 47 44 ssd 0.87299 1.00000 894G 10104M 884G 1.10 0.03 27 12 hdd 5.45599 1.00000 5587G 2242G 3344G 40.14 1.09 229 13 hdd 5.45599 1.00000 5587G 2551G 3035G 45.67 1.24 247 14 hdd 5.45599 1.00000 5587G 2513G 3074G 44.98 1.22 245 15 hdd 5.45599 1.00000 5587G 2014G 3572G 36.06 0.98 209 16 hdd 5.45599 1.00000 5587G 2586G 3000G 46.29 1.26 249 17 hdd 5.45599 1.00000 5587G 2459G 3127G 44.02 1.19 243 45 ssd 0.87299 1.00000 894G 9697M 884G 1.06 0.03 35 46 ssd 0.87299 1.00000 894G 12975M 881G 1.42 0.04 37 18 hdd 3.63699 1.00000 3724G 1595G 2128G 42.84 1.16 156 19 hdd 3.63699 1.00000 3724G 1387G 2336G 37.25 1.01 147 20 hdd 3.63699 1.00000 3724G 1551G 2172G 41.67 1.13 157 21 hdd 3.63699 1.00000 3724G 1535G 2189G 41.22 1.12 155 22 hdd 3.63699 1.00000 3724G 1459G 2264G 39.20 1.06 155 23 hdd 3.63699 1.00000 3724G 1395G 2329G 37.46 1.02 147 24 hdd 3.63699 1.00000 3724G 1489G 2234G 40.00 1.08 160 25 hdd 3.63699 1.00000 3724G 1634G 2090G 43.88 1.19 159 26 hdd 3.63699 1.00000 3724G 1566G 2157G 42.06 1.14 154 49 ssd 0.87299 1.00000 894G 9385M 884G 1.03 0.03 32 50 ssd 0.87299 1.00000 894G 12757M 881G 1.39 0.04 36 27 hdd 5.45599 1.00000 5587G 2462G 3124G 44.08 1.20 244 28 hdd 5.45599 1.00000 5587G 2314G 3272G 41.43 1.12 237 29 hdd 5.45599 1.00000 5587G 2166G 3420G 38.79 1.05 221 30 hdd 5.45599 1.00000 5587G 2484G 3102G 44.47 1.21 242 31 hdd 5.45599 1.00000 5587G 2292G 3294G 41.03 1.11 225 32 hdd 5.45599 1.00000 5587G 1982G 3604G 35.49 0.96 208 47 ssd 0.87299 1.00000 894G 12015M 882G 1.31 0.04 39 48 ssd 0.87299 1.00000 894G 14820M 879G 1.62 0.04 46 33 hdd 5.45599 1.00000 5587G 2002G 3584G 35.85 0.97 205 34 hdd 5.45599 1.00000 5587G 2069G 3517G 37.04 1.00 209 35 hdd 5.45599 1.00000 5587G 2187G 3399G 39.16 1.06 226 36 hdd 5.45599 1.00000 5587G 1821G 3765G 32.60 0.88 185 37 hdd 5.45599 1.00000 5587G 2123G 3463G 38.01 1.03 205 38 hdd 5.45599 1.00000 5587G 2197G 3390G 39.32 1.07 228 39 hdd 5.45599 1.00000 5587G 2180G 3406G 39.02 1.06 217 40 hdd 5.45599 1.00000 5587G 2232G 3354G 39.97 1.08 228 51 ssd 0.87320 1.00000 894G 14747M 879G 1.61 0.04 38 52 ssd 0.87320 1.00000 894G 7716M 886G 0.84 0.02 18 53 ssd 0.87320 1.00000 894G 12660M 881G 1.38 0.04 33 54 ssd 0.87320 1.00000 894G 11155M 883G 1.22 0.03 31 55 ssd 0.87320 1.00000 894G 9350M 885G 1.02 0.03 24 56 ssd 0.87320 1.00000 894G 13816M 880G 1.51 0.04 38 57 ssd 0.87320 1.00000 894G 10122M 884G 1.11 0.03 28 58 ssd 0.87320 1.00000 894G 10096M 884G 1.10 0.03 32 59 ssd 0.87320 1.00000 894G 13750M 880G 1.50 0.04 36 60 ssd 0.87320 1.00000 894G 16168M 878G 1.77 0.05 35 61 ssd 0.87320 1.00000 894G 11401M 883G 1.25 0.03 31 62 ssd 0.87320 1.00000 894G 12105M 882G 1.32 0.04 32 63 ssd 0.87320 1.00000 894G 13998M 880G 1.53 0.04 40 64 ssd 0.87320 1.00000 894G 17468M 877G 1.91 0.05 42 65 ssd 0.87320 1.00000 894G 9540M 884G 1.04 0.03 25 66 ssd 0.87320 1.00000 894G 15109M 879G 1.65 0.04 42 TOTAL 230T 86866G 145T 36.88 MIN/MAX VAR: 0.02/1.30 STDDEV: 22.48 With the following crush ruleset: # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18 osd.18 class hdd device 19 osd.19 class hdd device 20 osd.20 class hdd device 21 osd.21 class hdd device 22 osd.22 class hdd device 23 osd.23 class hdd device 24 osd.24 class hdd device 25 osd.25 class hdd device 26 osd.26 class hdd device 27 osd.27 class hdd device 28 osd.28 class hdd device 29 osd.29 class hdd device 30 osd.30 class hdd device 31 osd.31 class hdd device 32 osd.32 class hdd device 33 osd.33 class hdd device 34 osd.34 class hdd device 35 osd.35 class hdd device 36 osd.36 class hdd device 37 osd.37 class hdd device 38 osd.38 class hdd device 39 osd.39 class hdd device 40 osd.40 class hdd device 41 osd.41 class ssd device 42 osd.42 class ssd device 43 osd.43 class ssd device 44 osd.44 class ssd device 45 osd.45 class ssd device 46 osd.46 class ssd device 47 osd.47 class ssd device 48 osd.48 class ssd device 49 osd.49 class ssd device 50 osd.50 class ssd device 51 osd.51 class ssd device 52 osd.52 class ssd device 53 osd.53 class ssd device 54 osd.54 class ssd device 55 osd.55 class ssd device 56 osd.56 class ssd device 57 osd.57 class ssd device 58 osd.58 class ssd device 59 osd.59 class ssd device 60 osd.60 class ssd device 61 osd.61 class ssd device 62 osd.62 class ssd device 63 osd.63 class ssd device 64 osd.64 class ssd device 65 osd.65 class ssd device 66 osd.66 class ssd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root # buckets host node1 { id -2 # do not change unnecessarily id -8 class hdd # do not change unnecessarily id -15 class ssd # do not change unnecessarily # weight 34.482 alg straw2 hash 0 # rjenkins1 item osd.0 weight 5.456 item osd.1 weight 5.456 item osd.2 weight 5.456 item osd.3 weight 5.456 item osd.4 weight 5.456 item osd.5 weight 5.456 item osd.41 weight 0.873 item osd.42 weight 0.873 } host node2 { id -3 # do not change unnecessarily id -9 class hdd # do not change unnecessarily id -16 class ssd # do not change unnecessarily # weight 34.482 alg straw2 hash 0 # rjenkins1 item osd.6 weight 5.456 item osd.7 weight 5.456 item osd.8 weight 5.456 item osd.9 weight 5.456 item osd.10 weight 5.456 item osd.11 weight 5.456 item osd.43 weight 0.873 item osd.44 weight 0.873 } host node3 { id -4 # do not change unnecessarily id -10 class hdd # do not change unnecessarily id -17 class ssd # do not change unnecessarily # weight 34.482 alg straw2 hash 0 # rjenkins1 item osd.12 weight 5.456 item osd.13 weight 5.456 item osd.14 weight 5.456 item osd.15 weight 5.456 item osd.16 weight 5.456 item osd.17 weight 5.456 item osd.45 weight 0.873 item osd.46 weight 0.873 } host node4 { id -5 # do not change unnecessarily id -11 class hdd # do not change unnecessarily id -18 class ssd # do not change unnecessarily # weight 34.479 alg straw2 hash 0 # rjenkins1 item osd.18 weight 3.637 item osd.21 weight 3.637 item osd.24 weight 3.637 item osd.25 weight 3.637 item osd.22 weight 3.637 item osd.26 weight 3.637 item osd.19 weight 3.637 item osd.23 weight 3.637 item osd.20 weight 3.637 item osd.49 weight 0.873 item osd.50 weight 0.873 } host node5 { id -6 # do not change unnecessarily id -12 class hdd # do not change unnecessarily id -19 class ssd # do not change unnecessarily # weight 34.482 alg straw2 hash 0 # rjenkins1 item osd.27 weight 5.456 item osd.28 weight 5.456 item osd.29 weight 5.456 item osd.30 weight 5.456 item osd.31 weight 5.456 item osd.32 weight 5.456 item osd.47 weight 0.873 item osd.48 weight 0.873 } host node6 { id -7 # do not change unnecessarily id -13 class hdd # do not change unnecessarily id -20 class ssd # do not change unnecessarily # weight 43.648 alg straw2 hash 0 # rjenkins1 item osd.33 weight 5.456 item osd.34 weight 5.456 item osd.35 weight 5.456 item osd.36 weight 5.456 item osd.37 weight 5.456 item osd.38 weight 5.456 item osd.39 weight 5.456 item osd.40 weight 5.456 } host node7 { id -22 # do not change unnecessarily id -23 class hdd # do not change unnecessarily id -24 class ssd # do not change unnecessarily # weight 6.986 alg straw2 hash 0 # rjenkins1 item osd.51 weight 0.873 item osd.52 weight 0.873 item osd.53 weight 0.873 item osd.54 weight 0.873 item osd.55 weight 0.873 item osd.56 weight 0.873 item osd.57 weight 0.873 item osd.58 weight 0.873 } host node8 { id -25 # do not change unnecessarily id -26 class hdd # do not change unnecessarily id -27 class ssd # do not change unnecessarily # weight 6.986 alg straw2 hash 0 # rjenkins1 item osd.59 weight 0.873 item osd.60 weight 0.873 item osd.61 weight 0.873 item osd.62 weight 0.873 item osd.63 weight 0.873 item osd.64 weight 0.873 item osd.65 weight 0.873 item osd.66 weight 0.873 } root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily id -21 class ssd # do not change unnecessarily # weight 230.027 alg straw2 hash 0 # rjenkins1 item node1 weight 34.482 item node2 weight 34.482 item node3 weight 34.482 item node4 weight 34.479 item node5 weight 34.482 item node6 weight 43.649 item node7 weight 6.986 item node8 weight 6.986 } # rules rule replicated_ruleset { id 0 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit } rule hybrid { id 1 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn -1 type host step emit } rule flash { id 2 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } # end crush map And this is shown by ceph df, with the used CRUSH Rule added, all pools using a replication of 3: GLOBAL: SIZE AVAIL RAW USED %RAW USED 230T 145T 86866G 36.88 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS pool1 7 10454G 23.82 33430G 2871159 -> replicated_ruleset pool2 8 13333G 28.51 33430G 3660994 -> replicated_ruleset rbd 18 3729G 10.04 33430G 1021785 -> replicated_ruleset cephfs_data 24 270G 3.62 7213G 77568 -> hybrid cephfs_metadata 25 39781k 0 7213G 44 -> flash I have 23244G in SSDs in total. As you can see, the „replicated_ruleset“ chooses only HDDs, the „flash“ rule only SSDs and the „hybrid“ rule uses one SSD (as primary, since HDDs primary affinity is set to 0.5, while the SSDs is on 1.0) and the rest as HDDs. ceph df shows MAX AVAIL for the „replicated_ruleset“ and „flash“ rule correctly for my pools with replication 3. However the „hybrid“ rule shows the available space as if it was using 3 SSDs, even though it’s only using one, even with a
replication of 3. Is this a display bug or is something wrong with my configuration? And if it’s a bug, will it break anything if MAX AVAIL or %USED fills up to over 100%? |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
