|
Hi, Problem: The balancer still refuses to run and distribution is not what it was on Luminous. I presume a flag somewhere wasn’t unset whilst the autoscaling was reducing PGs? I wasted a lot of time this last week after having enabled pg_autoscale_mode on pools in a cluster. I had to manually reweight some OSDs that were near full
and the pg_num changes would constantly degrade redundancy so the balancer wouldn’t run. I eventually disabled the autoscaling mode on pools and manually set pg_num on pools. The cluster has finished moving everything around and everything is back to full
health again. I was planning on setting the reweight on the OSDs I’d changed back to 1 and then immediately running a manual balancer plan. I could also run a manual balancing plan first, adjust the reweight a little bit and repeat this cycle until all
OSDs again have a reweight of 1. Commands: ceph balancer eval ceph balancer optimize myplan ceph balancer eval myplan ceph balancer show myplan ceph balancer execute myplan ceph balancer reset Output: current cluster score 0.018841 (lower is better) Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect Error EINVAL: option "myplan" not a plan or a pool Error ENOENT: plan myplan not found Error ENOENT: plan myplan not found Balancer mode: [admin@kvm7e ~]#
ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Current distribution on Ceph Nautilus: [admin@kvm7e ~]#
ceph osd df | sort -g -k 5 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 100 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 6.8 MiB 3.1 GiB 509 GiB
72.68 0.97 56 up 101 hdd 1.81929
0.89999 1.8 TiB 1.3 TiB 1.3 TiB 5.2 MiB 3.0 GiB 523 GiB
71.92 0.96 56 up 200 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 5.8 MiB 3.3 GiB 373 GiB
79.99 1.07 62 up 201 hdd 1.81929
0.84999 1.8 TiB 1.5 TiB 1.5 TiB 5.3 MiB 3.4 GiB 298 GiB
83.98 1.13 65 up 300 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 5.8 MiB 3.2 GiB 311 GiB
83.31 1.12 65 up 301 hdd 1.81929 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 3.2 GiB 385 GiB
79.31 1.06 61 up 400 hdd 1.81929 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 4.4 MiB 2.8 GiB 446 GiB
76.06 1.02 61 up 401 hdd 1.81929
0.84999 1.8 TiB 1.4 TiB 1.4 TiB 4.7 MiB 3.2 GiB 413 GiB
77.82 1.04 62 up 500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 3.9 MiB 2.7 GiB 546 GiB
70.69 0.95 56 up 501 hdd 1.81929
0.89999 1.8 TiB 1.3 TiB 1.3 TiB 4.1 MiB 3.1 GiB 520 GiB
72.07 0.97 56 up 102 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 13 MiB 8.0 GiB 1.5 TiB
72.76 0.98 179 up 103 hdd 5.45789 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 17 MiB 10 GiB 1.2 TiB
78.54 1.05 185 up 202 hdd 5.45789
0.89999 5.5 TiB 3.9 TiB 3.9 TiB 7.9 MiB 8.3 GiB 1.6 TiB
71.08 0.95 170 up 203 hdd 5.45789
0.89999 5.5 TiB 4.0 TiB 4.0 TiB 11 MiB 9.0 GiB 1.4 TiB
73.44 0.98 174 up 302 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 13 MiB 9.3 GiB 1.3 TiB
77.05 1.03 185 up 303 hdd 5.45789 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 13 MiB 10 GiB 1.1 TiB
78.99 1.06 191 up 402 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 18 MiB 8.9 GiB 1.2 TiB
77.44 1.04 186 up 403 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 13 MiB 8.5 GiB 1.2 TiB
77.46 1.04 185 up 502 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 18 MiB 8.9 GiB 1.2 TiB
77.85 1.04 186 up 503 hdd 5.45789 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 13 MiB 9.6 GiB 1.3 TiB
75.32 1.01 181 up 104 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 22 MiB 14 GiB 2.1 TiB
76.43 1.02 305 up 105 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 23 MiB 14 GiB 2.1 TiB
77.34 1.04 311 up 204 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 31 MiB 15 GiB 2.1 TiB
77.17 1.03 311 up 205 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 20 MiB 14 GiB 2.1 TiB
76.82 1.03 312 up 304 hdd 9.09560
0.89999 9.1 TiB 6.4 TiB 6.4 TiB 18 MiB 13 GiB 2.7 TiB
70.45 0.94 283 up 305 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 17 MiB 15 GiB 2.1 TiB
76.93 1.03 311 up 404 hdd 9.09560
0.89999 9.1 TiB 6.6 TiB 6.6 TiB 18 MiB 14 GiB 2.5 TiB
72.58 0.97 288 up 405 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 22 MiB 14 GiB 2.1 TiB
77.20 1.03 311 up 504 hdd 9.09560 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 30 MiB 14 GiB 2.0 TiB
77.68 1.04 311 up 505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 24 MiB 14 GiB 2.2 TiB
75.37 1.01 304 up 10100 ssd 0.37700 1.00000 386 GiB 68 GiB 67 GiB 4 KiB 1024 MiB 318 GiB 17.65 0.24 35 up 10101 ssd 0.37700 1.00000 386 GiB 68 GiB 67 GiB 24 KiB 1024 MiB 318 GiB 17.73 0.24 36 up 10200 ssd 0.37700 1.00000 386 GiB 65 GiB 64 GiB 28 KiB 1024 MiB 321 GiB 16.82 0.23 40 up 10201 ssd 0.37700 1.00000 386 GiB 57 GiB 56 GiB 20 KiB 1024 MiB 329 GiB 14.68 0.20 37 up 10500 ssd 0.37700 1.00000 386 GiB 63 GiB 62 GiB 12 KiB 1024 MiB 323 GiB 16.29 0.22 38 up 10501 ssd 0.37700 1.00000 386 GiB 77 GiB 76 GiB 41 KiB 1024 MiB 309 GiB 19.98 0.27 40 up 10300 ssd 0.40630 1.00000 416 GiB 68 GiB 67 GiB 23 MiB 1001 MiB 348 GiB 16.42 0.22 38 up 10301 ssd 0.40630 1.00000 416 GiB 71 GiB 70 GiB 28 MiB 996 MiB 345 GiB 17.06 0.23 39 up 10400 ssd 0.40630 1.00000 416 GiB 67 GiB 66 GiB 31 MiB 993 MiB 349 GiB 16.14 0.22 40 up 10401 ssd 0.40630 1.00000 416 GiB 68 GiB 67 GiB 29 MiB 995 MiB 348 GiB 16.38 0.22 41 up TOTAL 168 TiB 125 TiB 125 TiB 524 MiB 271 GiB 43 TiB 74.60 Previous distribution, also on Ceph Nautilus, shortly after upgrading from Luminous: [admin@kvm7e ~]#
ceph osd df | sort -g -k 5 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 100 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 9.2 MiB 2.5 GiB 490 GiB 73.71
1.01 59 up 101 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 489 GiB 73.77
1.01 58 up 200 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.1 MiB 2.5 GiB 492 GiB 73.61
1.01 59 up 201 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 491 GiB 73.65
1.01 58 up 300 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 6.1 MiB 2.5 GiB 490 GiB 73.68
1.01 58 up 301 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 13 MiB 2.5 GiB 490 GiB 73.72
1.01 59 up 400 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.6 MiB 2.5 GiB 489 GiB 73.75
1.01 59 up 401 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.7 MiB 2.5 GiB 493 GiB 73.56
1.01 59 up 500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.8 MiB 2.5 GiB 491 GiB 73.66
1.01 59 up 501 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 10 MiB 2.5 GiB 491 GiB 73.63
1.01 58 up 102 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB 73.63
1.01 175 up 103 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 20 MiB 7.4 GiB 1.4 TiB 73.49
1.00 176 up 202 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB 73.60
1.01 175 up 203 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4 TiB 73.51
1.00 175 up 302 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 13 MiB 7.4 GiB 1.4 TiB 73.57
1.01 174 up 303 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4 TiB 73.56
1.01 177 up 402 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 25 MiB 7.4 GiB 1.4 TiB 73.51
1.00 176 up 403 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4 TiB 73.59
1.01 175 up 502 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4 TiB 73.62
1.01 174 up 503 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB 73.54
1.01 177 up 104 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 MiB 13 GiB 2.2 TiB 75.68
1.03 304 up 105 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 23 MiB 13 GiB 2.2 TiB 75.63
1.03 301 up 204 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB 75.65
1.03 301 up 205 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 41 MiB 13 GiB 2.2 TiB 75.71
1.03 302 up 304 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2 TiB 75.68
1.03 302 up 305 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB 75.67
1.03 304 up 404 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB 75.68
1.03 298 up 405 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB 75.69
1.03 301 up 504 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 38 MiB 13 GiB 2.2 TiB 75.67
1.03 301 up 505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2 TiB 75.67
1.03 300 up 10100 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 28 MiB 996 MiB 376 GiB 9.60 0.13 35 up 10101 ssd 0.40630 1.00000 416 GiB 44 GiB 43 GiB 20 MiB 1004 MiB 372 GiB 10.57 0.14 37 up 10200 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 15 MiB 1009 MiB 384 GiB 7.80 0.11 36 up 10201 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 16 MiB 1008 MiB 384 GiB 7.67 0.10 37 up 10300 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 17 MiB 1007 MiB 376 GiB 9.71 0.13 36 up 10301 ssd 0.40630 1.00000 416 GiB 43 GiB 42 GiB 18 MiB 1006 MiB 373 GiB 10.31 0.14 37 up 10400 ssd 0.40630 1.00000 416 GiB 34 GiB 33 GiB 17 MiB 1007 MiB 382 GiB 8.14 0.11 35 up 10401 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 19 MiB 1005 MiB 376 GiB 9.64 0.13 36 up 10500 ssd 0.40630 1.00000 416 GiB 38 GiB 37 GiB 18 MiB 1006 MiB 378 GiB 9.11 0.12 37 up 10501 ssd 0.40630 1.00000 416 GiB 46 GiB 45 GiB 18 MiB 1006 MiB 370 GiB 10.98 0.15 37 up TOTAL 168 TiB 123 TiB 123 TiB 839 MiB 234 GiB 45 TiB 73.16 Additional cluster information: [admin@kvm7e ~]#
ceph -s cluster: id: cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1 health: HEALTH_WARN 15 pgs not scrubbed in time services: mon: 3 daemons, quorum kvm7a,kvm7b,kvm7e (age 19h) mgr: kvm7e(active, since 2d), standbys: kvm7a, kvm7b mds: cephfs:1 {0=kvm7e=up:active} 2 up:standby osd: 40 osds: 40 up (since 3h), 40 in (since 2d) data: pools: 7 pools, 1251 pgs objects: 18.85M objects, 72 TiB usage: 125 TiB used, 43 TiB / 168 TiB avail pgs: 1250 active+clean 1 active+clean+scrubbing io: client: 23 MiB/s rd, 22 MiB/s wr, 193 op/s rd, 12 op/s wr cache: 13 MiB/s flush, 2 op/s promote [admin@kvm7e ~]#
ceph osd dump epoch 13680 fsid cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1 created 2019-06-07 11:49:30.007615 modified 2019-08-03 20:23:01.282690 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 203 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release nautilus pool 1 'rbd_ssd' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 12003 lfor 0/9743/11876 flags hashpspool,selfmanaged_snaps
stripe_width 0 application rbd removed_snaps [1~3] pool 5 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 5 pgp_num 5 last_change 13397 lfor 0/9750/9748 flags hashpspool
stripe_width 0 application cephfs pool 6 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 4 pgp_num 4 last_change 13397 lfor 0/9735/9733 flags hashpspool
stripe_width 0 application cephfs pool 7 'rbd_hdd' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 13448 lfor 369/10607/13428 flags hashpspool,selfmanaged_snaps
tiers 9 read_tier 9 write_tier 9 stripe_width 0 application rbd removed_snaps [1~3] pool 8 'ec_hdd' erasure size 5 min_size 4 crush_rule 3 object_hash rjenkins pg_num 1050 pgp_num 1050 last_change 13397 lfor 384/12035/12033 flags hashpspool,ec_overwrites,selfmanaged_snaps
tiers 10 read_tier 10 write_tier 10 stripe_width 12288 application rbd removed_snaps [1~3] pool 9 'rbd_hdd_cache' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 last_change 13483 lfor 369/9733/13462 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 7 cache_mode writeback target_bytes 137438953472 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 14400s x12 decay_rate 0 search_last_n 0 min_read_recency_for_promote 2 min_write_recency_for_promote
2 stripe_width 0 removed_snaps [1~3] pool 10 'ec_hdd_cache' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 last_change 12035 lfor 384/384/384 flags hashpspool,incomplete_clones,selfmanaged_snaps
tier_of 8 cache_mode writeback target_bytes 137438953472 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 14400s x12 decay_rate 0 search_last_n 0 min_read_recency_for_promote 2 min_write_recency_for_promote 2 stripe_width 0 removed_snaps [1~3] max_osd 10502 osd.100 up in weight 1 up_from 13526 up_thru 13677 down_at 13520 last_clean_interval [10434,13519) [v2:10.250.1.2:6858/214607,v1:10.250.1.2:6859/214607]
[v2:10.250.1.2:6860/214607,v1:10.250.1.2:6861/214607] exists,up 11cc23aa-021c-4145-b051-9492572ce41b osd.101 up in weight 0.899994 up_from 13517 up_thru 13677 down_at 13509 last_clean_interval [10434,13508) [v2:10.250.1.2:6837/198048,v1:10.250.1.2:6839/198048]
[v2:10.250.1.2:6841/198048,v1:10.250.1.2:6843/198048] exists,up bb75d594-c2ee-4606-a101-bdac0990031c osd.102 up in weight 1 up_from 13499 up_thru 13677 down_at 13494 last_clean_interval [10438,13493) [v2:10.250.1.2:6850/61445,v1:10.250.1.2:6851/61445]
[v2:10.250.1.2:6852/61445,v1:10.250.1.2:6853/61445] exists,up 03b5b5bf-b2ae-49dd-9244-daed6a0fdfed osd.103 up in weight 1 up_from 13529 up_thru 13677 down_at 13524 last_clean_interval [11526,13523) [v2:10.250.1.2:6834/251484,v1:10.250.1.2:6835/251484]
[v2:10.250.1.2:6836/251484,v1:10.250.1.2:6838/251484] exists,up 98346e6a-6081-4c22-93cc-685d920315ae osd.104 up in weight 1 up_from 13513 up_thru 13677 down_at 13502 last_clean_interval [10446,13501) [v2:10.250.1.2:6803/180986,v1:10.250.1.2:6806/180986]
[v2:10.250.1.2:6808/180986,v1:10.250.1.2:6810/180986] exists,up 23ff8534-df68-4c94-b7f1-d6a3b03ec267 osd.105 up in weight 1 up_from 13504 up_thru 13677 down_at 13496 last_clean_interval [11944,13495) [v2:10.250.1.2:6824/114849,v1:10.250.1.2:6826/114849]
[v2:10.250.1.2:6828/114849,v1:10.250.1.2:6829/114849] exists,up 29dbbe08-735c-4094-80df-f93f3f37ec57 osd.200 up in weight 1 up_from 13562 up_thru 13677 down_at 13555 last_clean_interval [10683,13554) [v2:10.250.1.3:6819/2991067,v1:10.250.1.3:6822/2991067]
[v2:10.250.1.3:6824/2991067,v1:10.250.1.3:6827/2991067] exists,up 71d27d56-59bf-4b89-ab19-a43602fee8cf osd.201 up in weight 0.849994 up_from 13552 up_thru 13677 down_at 13543 last_clean_interval [10679,13542) [v2:10.250.1.3:6818/2962845,v1:10.250.1.3:6821/2962845]
[v2:10.250.1.3:6825/2962845,v1:10.250.1.3:6828/2962845] exists,up 3865fc54-bc1a-406c-b4da-40420979f1e9 osd.202 up in weight 0.899994 up_from 13539 up_thru 13677 down_at 13533 last_clean_interval [10689,13532) [v2:10.250.1.3:6851/2932338,v1:10.250.1.3:6853/2932338]
[v2:10.250.1.3:6855/2932338,v1:10.250.1.3:6857/2932338] exists,up f0da6bae-bc91-4e49-a97c-1e3b0a8edc8b osd.203 up in weight 0.899994 up_from 13564 up_thru 13677 down_at 13560 last_clean_interval [12286,13559) [v2:10.250.1.3:6842/3004897,v1:10.250.1.3:6843/3004897]
[v2:10.250.1.3:6844/3004897,v1:10.250.1.3:6845/3004897] exists,up 49f58c54-c420-4e80-a837-bbd7213d37d3 osd.204 up in weight 1 up_from 13557 up_thru 13677 down_at 13550 last_clean_interval [10691,13549) [v2:10.250.1.3:6820/2985153,v1:10.250.1.3:6823/2985153]
[v2:10.250.1.3:6826/2985153,v1:10.250.1.3:6830/2985153] exists,up 1f45ea2a-ce81-4b58-ba44-3f23849a0d33 osd.205 up in weight 1 up_from 13547 up_thru 13677 down_at 13537 last_clean_interval [10694,13536) [v2:10.250.1.3:6850/2956087,v1:10.250.1.3:6852/2956087]
[v2:10.250.1.3:6854/2956087,v1:10.250.1.3:6856/2956087] exists,up b6af027b-e2a1-4682-a348-95d62064dd7c osd.300 up in weight 1 up_from 13595 up_thru 13677 down_at 13589 last_clean_interval [3035,13588) [v2:10.250.1.4:6843/3569006,v1:10.250.1.4:6845/3569006]
[v2:10.250.1.4:6847/3569006,v1:10.250.1.4:6848/3569006] exists,up f97770f2-8370-4c50-9f5a-661bad636581 osd.301 up in weight 1 up_from 13597 up_thru 13677 down_at 13593 last_clean_interval [3032,13592) [v2:10.250.1.4:6840/3570563,v1:10.250.1.4:6841/3570563]
[v2:10.250.1.4:6842/3570563,v1:10.250.1.4:6844/3570563] exists,up 6a36c28a-98fc-4feb-9151-f8ced7934e11 osd.302 up in weight 1 up_from 13591 up_thru 13677 down_at 13585 last_clean_interval [3049,13584) [v2:10.250.1.4:6856/3567489,v1:10.250.1.4:6857/3567489]
[v2:10.250.1.4:6858/3567489,v1:10.250.1.4:6859/3567489] exists,up 97a86edc-67ae-46db-bfc6-eb84cd1ddfb8 osd.303 up in weight 1 up_from 13579 up_thru 13677 down_at 13573 last_clean_interval [3038,13572) [v2:10.250.1.4:6818/3549971,v1:10.250.1.4:6819/3549971]
[v2:10.250.1.4:6820/3549971,v1:10.250.1.4:6821/3549971] exists,up 05ee7eb9-01bf-480a-bd0a-21dbdcd49b9c osd.304 up in weight 0.899994 up_from 13583 up_thru 13677 down_at 13577 last_clean_interval [3045,13576) [v2:10.250.1.4:6826/3556874,v1:10.250.1.4:6827/3556874]
[v2:10.250.1.4:6828/3556874,v1:10.250.1.4:6829/3556874] exists,up d1ffa811-5205-4d1e-919f-a15c0f0354af osd.305 up in weight 1 up_from 13587 up_thru 13677 down_at 13581 last_clean_interval [3043,13580) [v2:10.250.1.4:6800/3563705,v1:10.250.1.4:6801/3563705]
[v2:10.250.1.4:6834/3563705,v1:10.250.1.4:6835/3563705] exists,up 6b3ce147-4d11-45c7-b39d-9ae6f8c3652d osd.400 up in weight 1 up_from 13619 up_thru 13677 down_at 13612 last_clean_interval [2999,13611) [v2:10.250.1.5:6800/3440750,v1:10.250.1.5:6801/3440750]
[v2:10.250.1.5:6802/3440750,v1:10.250.1.5:6803/3440750] exists,up 93c4fb8a-7cac-4c35-8a4e-60f6129b2624 osd.401 up in weight 0.849994 up_from 13627 up_thru 13677 down_at 13621 last_clean_interval [3001,13620) [v2:10.250.1.5:6816/3451440,v1:10.250.1.5:6817/3451440]
[v2:10.250.1.5:6818/3451440,v1:10.250.1.5:6819/3451440] exists,up 0415bcaa-2531-49f6-bd53-3c9b059c2cca osd.402 up in weight 1 up_from 13642 up_thru 13677 down_at 13629 last_clean_interval [3005,13628) [v2:10.250.1.5:6808/3456506,v1:10.250.1.5:6809/3456506]
[v2:10.250.1.5:6810/3456506,v1:10.250.1.5:6811/3456506] exists,up fe1771d3-5990-43ad-9e50-d91f8b526bc4 osd.403 up in weight 1 up_from 13614 up_thru 13677 down_at 13601 last_clean_interval [3019,13600) [v2:10.250.1.5:6822/3438702,v1:10.250.1.5:6825/3438702]
[v2:10.250.1.5:6828/3438702,v1:10.250.1.5:6831/3438702] exists,up 92f38a49-ae2d-41cf-a3a2-ce4d05f90254 osd.404 up in weight 0.899994 up_from 13645 up_thru 13677 down_at 13640 last_clean_interval [3014,13639) [v2:10.250.1.5:6823/3465084,v1:10.250.1.5:6829/3465084]
[v2:10.250.1.5:6851/3465084,v1:10.250.1.5:6852/3465084] exists,up cbc6cb36-1c0e-4621-83dc-2a5f54b0fe87 osd.405 up in weight 1 up_from 13623 up_thru 13677 down_at 13617 last_clean_interval [3010,13616) [v2:10.250.1.5:6848/3449867,v1:10.250.1.5:6849/3449867]
[v2:10.250.1.5:6855/3449867,v1:10.250.1.5:6856/3449867] exists,up f80757ab-358c-4c90-ae80-f15a21e869d0 osd.500 up in weight 1 up_from 13669 up_thru 13669 down_at 13663 last_clean_interval [2969,13662) [v2:10.250.1.6:6810/3764835,v1:10.250.1.6:6812/3764835]
[v2:10.250.1.6:6814/3764835,v1:10.250.1.6:6816/3764835] exists,up bb06e92f-f816-4679-a3b7-bb1280842fe4 osd.501 up in weight 0.899994 up_from 13651 up_thru 13651 down_at 13647 last_clean_interval [2970,13646) [v2:10.250.1.6:6858/3751653,v1:10.250.1.6:6859/3751653]
[v2:10.250.1.6:6860/3751653,v1:10.250.1.6:6861/3751653] exists,up 80031729-eefa-4ae2-9dd2-672802fcf7f4 osd.502 up in weight 1 up_from 13677 up_thru 13677 down_at 13667 last_clean_interval [2982,13666) [v2:10.250.1.6:6839/3767566,v1:10.250.1.6:6842/3767566]
[v2:10.250.1.6:6845/3767566,v1:10.250.1.6:6848/3767566] exists,up 68edb6b0-ef54-41dd-85fa-ae7bf266acdf osd.503 up in weight 1 up_from 13656 up_thru 13656 down_at 13649 last_clean_interval [2972,13648) [v2:10.250.1.6:6811/3754044,v1:10.250.1.6:6813/3754044]
[v2:10.250.1.6:6815/3754044,v1:10.250.1.6:6817/3754044] exists,up cf84dd7b-8113-4178-a027-3d9b38aca683 osd.504 up in weight 1 up_from 13665 up_thru 13665 down_at 13659 last_clean_interval [2975,13658) [v2:10.250.1.6:6834/3763908,v1:10.250.1.6:6835/3763908]
[v2:10.250.1.6:6836/3763908,v1:10.250.1.6:6837/3763908] exists,up ea02de0f-0bdf-4ea1-8101-84e7a3282087 osd.505 up in weight 1 up_from 13661 up_thru 13661 down_at 13653 last_clean_interval [2976,13652) [v2:10.250.1.6:6840/3759237,v1:10.250.1.6:6843/3759237]
[v2:10.250.1.6:6846/3759237,v1:10.250.1.6:6849/3759237] exists,up 8f5e335e-d2a2-4899-a026-0b0c25378edc osd.10100 up in weight 1 up_from 13522 up_thru 13678 down_at 13515 last_clean_interval [10432,13514) [v2:10.250.1.2:6802/199825,v1:10.250.1.2:6804/199825]
[v2:10.250.1.2:6805/199825,v1:10.250.1.2:6807/199825] exists,up dc24ae90-6ac9-4414-a7ee-086cbb5de4b5 osd.10101 up in weight 1 up_from 13511 up_thru 13675 down_at 13507 last_clean_interval [10429,13506) [v2:10.250.1.2:6814/182479,v1:10.250.1.2:6816/182479]
[v2:10.250.1.2:6818/182479,v1:10.250.1.2:6820/182479] exists,up 733b43e5-988c-4f1d-9161-b9b55e7db98e osd.10200 up in weight 1 up_from 13535 up_thru 13678 down_at 13531 last_clean_interval [10681,13530) [v2:10.250.1.3:6803/2919718,v1:10.250.1.3:6805/2919718]
[v2:10.250.1.3:6807/2919718,v1:10.250.1.3:6809/2919718] exists,up 7fbb3d00-6333-4353-afc5-f11bb2a1ce7a osd.10201 up in weight 1 up_from 13545 up_thru 13678 down_at 13541 last_clean_interval [10699,13540) [v2:10.250.1.3:6802/2956573,v1:10.250.1.3:6804/2956573]
[v2:10.250.1.3:6806/2956573,v1:10.250.1.3:6808/2956573] exists,up 07b8ec5e-36d5-4739-ae4c-9698e524ba69 osd.10300 up in weight 1 up_from 13575 up_thru 13678 down_at 13569 last_clean_interval [3030,13568) [v2:10.250.1.4:6810/3545209,v1:10.250.1.4:6811/3545209]
[v2:10.250.1.4:6812/3545209,v1:10.250.1.4:6813/3545209] exists,up 016ec42d-08ee-4f59-afdd-c48ad0c046a7 osd.10301 up in weight 1 up_from 13571 up_thru 13678 down_at 13566 last_clean_interval [3030,13565) [v2:10.250.1.4:6802/3544816,v1:10.250.1.4:6803/3544816]
[v2:10.250.1.4:6804/3544816,v1:10.250.1.4:6805/3544816] exists,up f364315a-92b1-4fc6-b6cd-235fc6a87186 osd.10400 up in weight 1 up_from 13603 up_thru 13678 down_at 13599 last_clean_interval [2997,13598) [v2:10.250.1.5:6830/3434175,v1:10.250.1.5:6834/3434175]
[v2:10.250.1.5:6837/3434175,v1:10.250.1.5:6839/3434175] exists,up 1f7c6e5e-2bb7-4e95-8da0-2dfc5d285a17 osd.10401 up in weight 1 up_from 13631 up_thru 13678 down_at 13625 last_clean_interval [2997,13624) [v2:10.250.1.5:6821/3451855,v1:10.250.1.5:6826/3451855]
[v2:10.250.1.5:6833/3451855,v1:10.250.1.5:6836/3451855] exists,up 9ecd21dc-c610-4cf1-a270-f407eedcf85d osd.10500 up in weight 1 up_from 13675 up_thru 13675 down_at 13671 last_clean_interval [10932,13670) [v2:10.250.1.6:6802/3767787,v1:10.250.1.6:6803/3767787]
[v2:10.250.1.6:6804/3767787,v1:10.250.1.6:6805/3767787] exists,up ddff6c5c-2d93-48ba-9662-8a02ba2fab49 osd.10501 up in weight 1 up_from 13678 up_thru 13678 down_at 13673 last_clean_interval [10937,13672) [v2:10.250.1.6:6826/3768089,v1:10.250.1.6:6827/3768089]
[v2:10.250.1.6:6828/3768089,v1:10.250.1.6:6829/3768089] exists,up 7a6df9ba-e51d-4817-bc4c-c695bcf748c2 pg_upmap_items 1.c [10400,10401] pg_upmap_items 1.29 [10400,10401] pg_upmap_items 6.1 [503,502] pg_upmap_items 7.0 [305,304] pg_upmap_items 7.2 [102,103,305,304] pg_upmap_items 7.3 [305,302] pg_upmap_items 7.4 [305,302] pg_upmap_items 7.8 [305,302] pg_upmap_items 7.a [305,304] pg_upmap_items 7.c [305,304] pg_upmap_items 7.d [505,502] pg_upmap_items 7.15 [204,203] pg_upmap_items 7.16 [305,302] pg_upmap_items 7.17 [202,203] pg_upmap_items 7.1a [503,502] pg_upmap_items 7.1b [104,102] pg_upmap_items 7.27 [305,300] pg_upmap_items 7.2f [102,101] pg_upmap_items 7.30 [402,400] pg_upmap_items 7.38 [405,404] pg_upmap_items 8.0 [404,400] pg_upmap_items 8.5 [302,305,400,402] pg_upmap_items 8.6 [200,204] pg_upmap_items 8.a [303,305,404,403] pg_upmap_items 8.c [501,505] pg_upmap_items 8.d [404,405] pg_upmap_items 8.10 [304,305,202,205] pg_upmap_items 8.11 [200,204] pg_upmap_items 8.12 [302,305] pg_upmap_items 8.17 [103,102] pg_upmap_items 8.18 [304,305] pg_upmap_items 8.19 [101,104] pg_upmap_items 8.1b [501,503] pg_upmap_items 8.25 [304,302] pg_upmap_items 8.26 [201,204] pg_upmap_items 8.28 [304,305,203,205] pg_upmap_items 8.38 [501,503] pg_upmap_items 8.45 [202,204] pg_upmap_items 8.46 [304,305] pg_upmap_items 8.48 [404,403] pg_upmap_items 8.49 [404,403] pg_upmap_items 8.51 [103,102] pg_upmap_items 8.52 [202,205] pg_upmap_items 8.54 [404,403] pg_upmap_items 8.5a [201,205,404,405] pg_upmap_items 8.5d [404,403] pg_upmap_items 8.60 [202,205] pg_upmap_items 8.63 [404,403] pg_upmap_items 8.6a [304,302] pg_upmap_items 8.71 [202,205] pg_upmap_items 8.7d [501,503] pg_upmap_items 8.85 [103,102] pg_upmap_items 8.8b [304,305] pg_upmap_items 8.91 [304,305] pg_upmap_items 8.92 [303,301] pg_upmap_items 8.97 [404,403] pg_upmap_items 8.a0 [404,403] pg_upmap_items 8.a1 [501,503] pg_upmap_items 8.a3 [404,403] pg_upmap_items 8.a6 [404,403] pg_upmap_items 8.a9 [404,403] pg_upmap_items 8.aa [404,403] pg_upmap_items 8.ac [304,302] pg_upmap_items 8.ad [304,302] pg_upmap_items 8.b2 [404,403] pg_upmap_items 8.b9 [304,305,202,205] pg_upmap_items 8.c0 [201,205] pg_upmap_items 8.c2 [501,503] pg_upmap_items 8.c3 [202,205] pg_upmap_items 8.c7 [304,305,201,205] pg_upmap_items 8.d6 [202,205] pg_upmap_items 8.d7 [203,204] pg_upmap_items 8.da [202,205] pg_upmap_items 8.de [404,403] pg_upmap_items 8.e7 [304,305] pg_upmap_items 8.e8 [404,405] pg_upmap_items 8.f9 [501,503] pg_upmap_items 8.106 [303,305] pg_upmap_items 8.10f [504,503] pg_upmap_items 8.118 [404,400] pg_upmap_items 8.11a [202,205] pg_upmap_items 8.127 [404,403] pg_upmap_items 8.12a [501,503] pg_upmap_items 8.132 [304,305] pg_upmap_items 8.141 [304,302] pg_upmap_items 8.142 [404,405] pg_upmap_items 8.145 [300,305] pg_upmap_items 8.147 [201,204] pg_upmap_items 8.14e [504,503] pg_upmap_items 8.150 [303,302] pg_upmap_items 8.151 [203,205,404,405] pg_upmap_items 8.155 [501,503] pg_upmap_items 8.15d [202,205] pg_upmap_items 8.165 [404,403] pg_upmap_items 8.166 [303,305] pg_upmap_items 8.170 [404,403] pg_upmap_items 8.171 [304,305] pg_upmap_items 8.17e [304,305] pg_upmap_items 8.17f [203,204] pg_upmap_items 8.182 [404,400] pg_upmap_items 8.18c [501,503] pg_upmap_items 8.190 [202,205] pg_upmap_items 8.191 [504,503] pg_upmap_items 8.195 [202,204] pg_upmap_items 8.196 [304,305,201,204] pg_upmap_items 8.19f [304,305] pg_upmap_items 8.1a2 [202,204] pg_upmap_items 8.1a8 [501,500] pg_upmap_items 8.1af [103,102,504,503,404,403] pg_upmap_items 8.1b5 [404,403] pg_upmap_items 8.1bb [404,403] pg_upmap_items 8.1c2 [404,403] pg_upmap_items 8.1c4 [504,503] pg_upmap_items 8.1c7 [202,205] pg_upmap_items 8.1ca [404,403] pg_upmap_items 8.1cd [202,205] pg_upmap_items 8.1cf [304,305] pg_upmap_items 8.1d1 [404,405] pg_upmap_items 8.1d4 [202,205] pg_upmap_items 8.1df [504,503,203,205] pg_upmap_items 8.1e2 [404,403] pg_upmap_items 8.1e3 [501,505] pg_upmap_items 8.1e5 [303,305] pg_upmap_items 8.1ec [404,403] pg_upmap_items 8.1f2 [501,500] pg_upmap_items 8.1f3 [404,403] pg_upmap_items 8.1fb [304,305] pg_upmap_items 8.205 [404,403] pg_upmap_items 8.20d [404,405] pg_upmap_items 8.20f [304,301] pg_upmap_items 8.215 [404,403] pg_upmap_items 8.216 [203,205,404,400] pg_upmap_items 8.21b [404,403] pg_upmap_items 8.224 [404,403] pg_upmap_items 8.230 [202,205] pg_upmap_items 8.237 [201,205,404,400] pg_upmap_items 8.241 [501,503] pg_upmap_items 8.246 [103,102] pg_upmap_items 8.24b [304,305,202,204] pg_upmap_items 8.24f [404,403] pg_upmap_items 8.254 [304,305] pg_upmap_items 8.25c [404,400] pg_upmap_items 8.269 [304,305] pg_upmap_items 8.272 [404,400] pg_upmap_items 8.275 [504,503,404,405] pg_upmap_items 8.279 [404,403] pg_upmap_items 8.27e [303,301] pg_upmap_items 8.280 [202,205] pg_upmap_items 8.289 [501,505] pg_upmap_items 8.290 [501,500] pg_upmap_items 8.299 [501,505] pg_upmap_items 8.29c [404,403] pg_upmap_items 8.29e [401,400] pg_upmap_items 8.2a1 [404,403] pg_upmap_items 8.2a3 [501,500] pg_upmap_items 8.2a4 [404,405] pg_upmap_items 8.2a5 [404,403] pg_upmap_items 8.2a6 [501,503] pg_upmap_items 8.2a8 [203,205] pg_upmap_items 8.2b0 [304,305,404,403] pg_upmap_items 8.2b2 [201,205] pg_upmap_items 8.2b8 [404,403] pg_upmap_items 8.2b9 [501,500] pg_upmap_items 8.2be [303,305] pg_upmap_items 8.2c4 [304,305] pg_upmap_items 8.2ca [501,503] pg_upmap_items 8.2d5 [203,204] pg_upmap_items 8.2d7 [103,102,404,400] pg_upmap_items 8.2da [304,302] pg_upmap_items 8.2dc [103,102] pg_upmap_items 8.2dd [304,305] pg_upmap_items 8.2e3 [203,205] pg_upmap_items 8.2ed [203,205] pg_upmap_items 8.2ef [404,403] pg_upmap_items 8.300 [303,302] pg_upmap_items 8.303 [404,403,304,305] pg_upmap_items 8.304 [203,205] pg_upmap_items 8.307 [304,305] pg_upmap_items 8.309 [202,205] pg_upmap_items 8.30a [201,204] pg_upmap_items 8.312 [304,302] pg_upmap_items 8.319 [501,500] pg_upmap_items 8.31e [304,305] pg_upmap_items 8.324 [303,301] pg_upmap_items 8.336 [404,403] pg_upmap_items 8.338 [303,305] pg_upmap_items 8.339 [303,302] pg_upmap_items 8.33a [404,400] pg_upmap_items 8.33c [304,305] pg_upmap_items 8.344 [404,403] pg_upmap_items 8.346 [201,204] pg_upmap_items 8.349 [504,503] pg_upmap_items 8.350 [501,500] pg_upmap_items 8.356 [101,102] pg_upmap_items 8.358 [404,405] pg_upmap_items 8.363 [103,102] pg_upmap_items 8.364 [404,403] pg_upmap_items 8.366 [404,403] pg_upmap_items 8.369 [304,305] pg_upmap_items 8.36b [103,102] pg_upmap_items 8.373 [404,403] pg_upmap_items 8.383 [404,403] pg_upmap_items 8.39d [203,205] pg_upmap_items 8.3a3 [103,102] pg_upmap_items 8.3a6 [304,305] pg_upmap_items 8.3ab [304,305] pg_upmap_items 8.3af [304,305] pg_upmap_items 8.3b3 [404,405] pg_upmap_items 8.3b4 [303,305] pg_upmap_items 8.3b7 [404,403] pg_upmap_items 8.3b9 [404,403] pg_upmap_items 8.3ba [404,403,201,205] pg_upmap_items 8.3bd [404,405] pg_upmap_items 8.3c0 [304,305] pg_upmap_items 8.3c3 [404,403] pg_upmap_items 8.3ca [404,403] pg_upmap_items 8.3cf [404,405] pg_upmap_items 8.3d0 [404,405] pg_upmap_items 8.3da [404,403] pg_upmap_items 8.3e4 [404,405] pg_upmap_items 8.3ea [404,405] pg_upmap_items 8.3ec [203,205] pg_upmap_items 8.3f3 [501,505] pg_upmap_items 8.3f7 [304,305] pg_upmap_items 8.3fb [404,405] pg_upmap_items 8.3fc [304,305] pg_upmap_items 8.400 [105,102,404,403] pg_upmap_items 8.409 [404,403] pg_upmap_items 8.40b [103,102,404,405] pg_upmap_items 8.40c [404,400] pg_upmap_items 8.410 [404,403] pg_upmap_items 8.411 [404,405] pg_upmap_items 8.417 [404,403] pg_upmap_items 8.418 [404,403] pg_upmap_items 9.2 [10401,10400] pg_upmap_items 9.9 [10200,10201] Regards David Herselman |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
