Re: MDS crashes (80.8)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 26/02/2015 18:07, Wyllys Ingersoll wrote:
Here is 'ceph df', followed by "mds dump" and "osd dump"

Your MDS map is trying to use pool 0 for both data and metadata. Firstly, these should be different pools. Secondly, you have no pool 0. You do have metadata and data pools with ids 6 and 7, so you should be typing "ceph mds newfs 6 7".

Note that this situation can't occur on more recent versions of Ceph, where the filesystem<->pool relations are enforced strictly. If you are intent on using the filesystem, you should consider using more recent versions to benefit from the numerous other filesystem bugfixes.

Cheers,
John

$ ceph df
GLOBAL:
     SIZE      AVAIL     RAW USED     %RAW USED
     1307T     1302T        5039G          0.38
POOLS:
     NAME          ID     USED      %USED     MAX AVAIL     OBJECTS
     ks3backup     3       160M         0             0           9
     test          4          0         0             0           0
     TEST          5      2501G      0.19             0      320263
     metadata      6          0         0             0           0
     data          7          0         0             0           0
     rbd           8          0         0             0           0

$ ceph mds dump
dumped mdsmap epoch 220
epoch 220
flags 0
created 2015-02-26 07:43:23.410584
modified 2015-02-26 07:55:28.817383
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 867
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap}
max_mds 1
in 0
up {0=20039}
failed
stopped
data_pools 0
metadata_pool 0
inline_data disabled
20039: 10.2.3.33:6800/5098 'mdc03' mds.0.91 up:creating seq 1 laggy
since 2015-02-26 07:55:28.817341

$ ceph osd dump
epoch 872
fsid e73bfab8-01f1-4534-9a66-1b425d5c3341
created 2015-02-23 07:53:05.626725
modified 2015-02-26 08:01:48.834331
flags
pool 3 'ks3backup' replicated size 2 min_size 1 crush_ruleset 0
object_hash rjenkins pg_num 512 pgp_num 512 last_change 251 flags
hashpspool stripe_width 0
pool 4 'test' replicated size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 363 flags hashpspool
stripe_width 0
pool 5 'TEST' replicated size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 473 flags hashpspool
stripe_width 0
pool 6 'metadata' replicated size 2 min_size 1 crush_ruleset 0
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 808 flags
hashpspool stripe_width 0
pool 7 'data' replicated size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 810 flags hashpspool
stripe_width 0
pool 8 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 833 flags hashpspool
stripe_width 0
max_osd 242
osd.0 up   in  weight 1 up_from 774 up_thru 833 down_at 748
last_clean_interval [692,747) 10.2.2.109:6851/6280
10.3.2.109:6834/6280 10.3.2.109:6835/6280 10.2.2.109:6852/6280
exists,up 43999652-2966-4b1b-a8f0-c33ef5db0970
osd.1 up   in  weight 1 up_from 648 up_thru 833 down_at 630
last_clean_interval [598,629) 10.2.3.105:6857/6775
10.3.3.105:6838/6775 10.3.3.105:6839/6775 10.2.3.105:6858/6775
exists,up 1e4b6fa3-044c-40b8-9f5b-968bdfae5dd7
osd.2 up   in  weight 1 up_from 627 up_thru 833 down_at 619
last_clean_interval [455,618) 10.2.1.108:6848/5782
10.3.1.108:6832/5782 10.3.1.108:6833/5782 10.2.1.108:6849/5782
exists,up 95000872-72f2-46e8-8d4a-f9b1422032cb
osd.3 up   in  weight 1 up_from 773 up_thru 833 down_at 752
last_clean_interval [693,751) 10.2.2.110:6833/4189
10.3.2.110:6822/4189 10.3.2.110:6823/4189 10.2.2.110:6834/4189
exists,up 980182ab-51bf-4cd9-a9a4-cfca05123592
osd.4 up   in  weight 1 up_from 732 up_thru 833 down_at 722
last_clean_interval [718,721) 10.2.1.101:6806/2789
10.3.1.101:6804/2789 10.3.1.101:6805/2789 10.2.1.101:6807/2789
exists,up 92a8fe48-082d-4ad1-93ba-5a4f9598d866
osd.5 up   in  weight 1 up_from 766 up_thru 810 down_at 739
last_clean_interval [663,738) 10.2.3.111:6800/4353
10.3.3.111:6800/4353 10.3.3.111:6801/4353 10.2.3.111:6801/4353
exists,up d6539347-2c2a-4fba-867d-74c84b2018ad
osd.6 up   in  weight 1 up_from 781 up_thru 833 down_at 760
last_clean_interval [711,759) 10.2.2.104:6830/4317
10.3.2.104:6820/4317 10.3.2.104:6821/4317 10.2.2.104:6831/4317
exists,up 55b05a0f-2b17-44e2-84c5-78db81ea862b
osd.7 up   in  weight 1 up_from 639 up_thru 833 down_at 633
last_clean_interval [608,632) 10.2.3.106:6842/4762
10.3.3.106:6828/4762 10.3.3.106:6829/4762 10.2.3.106:6843/4762
exists,up 6a62d4fc-487c-4b39-a64a-823d675969a9
osd.8 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [671,743) 10.2.3.112:6845/5168
10.3.3.112:6830/5168 10.3.3.112:6831/5168 10.2.3.112:6846/5168
exists,up 9c4202e6-7749-4436-adb1-9c8bf1dc7f04
osd.9 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [719,726) 10.2.1.102:6809/3004
10.3.1.102:6806/3004 10.3.1.102:6807/3004 10.2.1.102:6810/3004
exists,up 2d0a4808-99d7-4a6e-b8d9-15793797830a
osd.10 up   in  weight 1 up_from 780 up_thru 810 down_at 756
last_clean_interval [711,755) 10.2.2.103:6836/4424
10.3.2.103:6824/4424 10.3.2.103:6825/4424 10.2.2.103:6837/4424
exists,up 69efa89e-3a3d-4be4-a737-28ef4ad369c0
osd.11 up   in  weight 1 up_from 625 up_thru 833 down_at 615
last_clean_interval [466,614) 10.2.1.107:6839/4742
10.3.1.107:6826/4742 10.3.1.107:6827/4742 10.2.1.107:6840/4742
exists,up 17d5c13c-bfa1-4291-957e-967d43f2e15b
osd.12 up   in  weight 1 up_from 773 up_thru 833 down_at 748
last_clean_interval [692,747) 10.2.2.109:6833/4409
10.3.2.109:6822/4409 10.3.2.109:6823/4409 10.2.2.109:6834/4409
exists,up 14b42b0f-c20a-4083-b5c5-5960cd221244
osd.13 up   in  weight 1 up_from 639 up_thru 833 down_at 633
last_clean_interval [607,632) 10.2.3.106:6839/4618
10.3.3.106:6826/4618 10.3.3.106:6827/4618 10.2.3.106:6840/4618
exists,up 46240cb7-f6f3-4579-9bc0-6ddb64275172
osd.14 up   in  weight 1 up_from 735 up_thru 833 down_at 727
last_clean_interval [720,726) 10.2.1.102:6827/3978
10.3.1.102:6818/3978 10.3.1.102:6819/3978 10.2.1.102:6828/3978
exists,up e4883def-77b1-4605-bded-ebed75636241
osd.15 up   in  weight 1 up_from 775 up_thru 833 down_at 752
last_clean_interval [694,751) 10.2.2.110:6851/6591
10.3.2.110:6834/6591 10.3.2.110:6835/6591 10.2.2.110:6852/6591
exists,up bf04a229-e851-4907-a9b0-d26b844db037
osd.16 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [662,738) 10.2.3.111:6810/4368
10.3.3.111:6808/4368 10.3.3.111:6812/4368 10.2.3.111:6814/4368
exists,up 551340b1-98d9-4a67-b1d6-aa12d0967196
osd.17 up   in  weight 1 up_from 736 up_thru 833 down_at 722
last_clean_interval [719,721) 10.2.1.101:6851/6534
10.3.1.101:6834/6534 10.3.1.101:6835/6534 10.2.1.101:6852/6534
exists,up a53b2900-30be-4f65-a0c0-3400070706f3
osd.18 up   in  weight 1 up_from 645 up_thru 833 down_at 630
last_clean_interval [602,629) 10.2.3.105:6824/3776
10.3.3.105:6816/3776 10.3.3.105:6817/3776 10.2.3.105:6825/3776
exists,up 38a1f673-c195-4266-a623-2b069e2cebf9
osd.19 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [454,618) 10.2.1.108:6836/4575
10.3.1.108:6824/4575 10.3.1.108:6825/4575 10.2.1.108:6837/4575
exists,up c5f53b42-b05e-4c30-9e96-b5cec2b7fe59
osd.20 up   in  weight 1 up_from 765 up_thru 810 down_at 744
last_clean_interval [667,742) 10.2.3.112:6802/3795
10.3.3.112:6802/3795 10.3.3.112:6805/3795 10.2.3.112:6807/3795
exists,up 081625e4-ca5f-41af-b524-0aff23e2f162
osd.21 up   in  weight 1 up_from 782 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6848/6085
10.3.2.104:6832/6085 10.3.2.104:6833/6085 10.2.2.104:6849/6085
exists,up d99a53ce-98ad-4d6a-9124-454434d24e12
osd.22 up   in  weight 1 up_from 626 up_thru 833 down_at 615
last_clean_interval [466,614) 10.2.1.107:6842/4920
10.3.1.107:6828/4920 10.3.1.107:6829/4920 10.2.1.107:6843/4920
exists,up ab5c5015-c32e-4119-b991-47a497c1fabd
osd.23 up   in  weight 1 up_from 779 up_thru 810 down_at 756
last_clean_interval [708,755) 10.2.2.103:6818/3084
10.3.2.103:6812/3084 10.3.2.103:6813/3084 10.2.2.103:6819/3084
exists,up 36a2ab86-3124-40ca-8ec5-b3a75cd06482
osd.24 up   in  weight 1 up_from 773 up_thru 833 down_at 748
last_clean_interval [693,747) 10.2.2.109:6836/4538
10.3.2.109:6824/4538 10.3.2.109:6825/4538 10.2.2.109:6837/4538
exists,up b4cf2203-b031-4f65-9155-736fd2d78e97
osd.25 up   in  weight 1 up_from 773 up_thru 833 down_at 752
last_clean_interval [694,751) 10.2.2.110:6824/3284
10.3.2.110:6816/3284 10.3.2.110:6817/3284 10.2.2.110:6825/3284
exists,up 102d241e-825f-4dd9-af72-058fed3fbb0f
osd.26 up   in  weight 1 up_from 640 up_thru 833 down_at 633
last_clean_interval [607,632) 10.2.3.106:6848/5787
10.3.3.106:6832/5787 10.3.3.106:6833/5787 10.2.3.106:6849/5787
exists,up c934348b-92d4-4ae2-a40e-795cdd68a947
osd.27 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [661,738) 10.2.3.111:6821/4386
10.3.3.111:6814/4386 10.3.3.111:6815/4386 10.2.3.111:6822/4386
exists,up 6354d07f-3f96-4157-8e5b-bccc69bc2273
osd.28 up   in  weight 1 up_from 736 up_thru 833 down_at 727
last_clean_interval [717,726) 10.2.1.102:6851/6511
10.3.1.102:6834/6511 10.3.1.102:6835/6511 10.2.1.102:6852/6511
exists,up a67195b0-b8cc-4f68-bc95-5d7731188c81
osd.29 up   in  weight 1 up_from 628 up_thru 833 down_at 619
last_clean_interval [452,618) 10.2.1.108:6857/6875
10.3.1.108:6838/6875 10.3.1.108:6839/6875 10.2.1.108:6858/6875
exists,up a887abe7-e9b2-41f3-8a47-9046a0ca6404
osd.30 up   in  weight 1 up_from 781 up_thru 833 down_at 760
last_clean_interval [712,759) 10.2.2.104:6833/4445
10.3.2.104:6822/4445 10.3.2.104:6823/4445 10.2.2.104:6834/4445
exists,up c1c49afb-164c-4b46-9d40-2b3c37811439
osd.31 up   in  weight 1 up_from 768 up_thru 833 down_at 744
last_clean_interval [669,743) 10.2.3.112:6848/7064
10.3.3.112:6832/7064 10.3.3.112:6833/7064 10.2.3.112:6849/7064
exists,up 26121990-1276-4757-b46a-cdb1999f083d
osd.32 up   in  weight 1 up_from 735 up_thru 833 down_at 722
last_clean_interval [718,721) 10.2.1.101:6839/5315
10.3.1.101:6826/5315 10.3.1.101:6827/5315 10.2.1.101:6840/5315
exists,up 9cbc8844-730c-4250-a08d-1eaecc88da99
osd.33 up   in  weight 1 up_from 648 up_thru 833 down_at 630
last_clean_interval [597,629) 10.2.3.105:6854/6295
10.3.3.105:6836/6295 10.3.3.105:6837/6295 10.2.3.105:6855/6295
exists,up 42c72969-8d12-4873-ba31-eaac6df2f099
osd.34 up   in  weight 1 up_from 625 up_thru 833 down_at 615
last_clean_interval [465,614) 10.2.1.107:6830/4031
10.3.1.107:6820/4031 10.3.1.107:6821/4031 10.2.1.107:6831/4031
exists,up 93bb297c-03d3-44cf-ac29-0bf9f6a7056f
osd.35 up   in  weight 1 up_from 780 up_thru 833 down_at 756
last_clean_interval [711,755) 10.2.2.103:6839/4550
10.3.2.103:6826/4550 10.3.2.103:6827/4550 10.2.2.103:6840/4550
exists,up ff0163b7-5eec-4d27-8838-d5b574bbbce1
osd.36 up   in  weight 1 up_from 773 up_thru 833 down_at 748
last_clean_interval [694,747) 10.2.2.109:6839/4693
10.3.2.109:6826/4693 10.3.2.109:6827/4693 10.2.2.109:6840/4693
exists,up 69f50a5f-de37-40a2-b4eb-6aa655f2f18f
osd.37 up   in  weight 1 up_from 776 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6854/7101
10.3.2.110:6836/7101 10.3.2.110:6837/7101 10.2.2.110:6855/7101
exists,up e6810117-8904-49a5-b671-c0b1de7ac80e
osd.38 up   in  weight 1 up_from 640 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6851/6088
10.3.3.106:6834/6088 10.3.3.106:6835/6088 10.2.3.106:6852/6088
exists,up 2414575f-455d-42f7-b4d6-0ec47ac09972
osd.39 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [661,738) 10.2.3.111:6845/4477
10.3.3.111:6830/4477 10.3.3.111:6831/4477 10.2.3.111:6846/4477
exists,up 48ea62f9-862d-496a-bee9-cc461e2110ee
osd.40 up   in  weight 1 up_from 736 up_thru 833 down_at 727
last_clean_interval [717,726) 10.2.1.102:6848/6063
10.3.1.102:6832/6063 10.3.1.102:6833/6063 10.2.1.102:6849/6063
exists,up 726479fc-b165-4244-a3fb-4edf37153299
osd.41 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [452,618) 10.2.1.108:6824/3643
10.3.1.108:6816/3643 10.3.1.108:6817/3643 10.2.1.108:6825/3643
exists,up 615d72c4-96b1-4969-be46-fbbd59e0a560
osd.42 up   in  weight 1 up_from 781 up_thru 810 down_at 760
last_clean_interval [713,759) 10.2.2.104:6827/3944
10.3.2.104:6818/3944 10.3.2.104:6819/3944 10.2.2.104:6828/3944
exists,up b5e0904b-5167-4ede-b50e-43576c3e57a8
osd.43 up   in  weight 1 up_from 647 up_thru 833 down_at 630
last_clean_interval [595,629) 10.2.3.105:6845/5527
10.3.3.105:6830/5527 10.3.3.105:6831/5527 10.2.3.105:6846/5527
exists,up 9a81e3f3-a3cb-4933-829c-dff1fa50e663
osd.44 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [671,743) 10.2.3.112:6800/3801
10.3.3.112:6801/3801 10.3.3.112:6804/3801 10.2.3.112:6805/3801
exists,up 6f747700-67d7-4f39-b7bc-7a090f56914d
osd.45 up   in  weight 1 up_from 734 up_thru 833 down_at 722
last_clean_interval [718,721) 10.2.1.101:6824/3800
10.3.1.101:6816/3800 10.3.1.101:6817/3800 10.2.1.101:6825/3800
exists,up 9b1839da-7771-484b-849d-6f34cd0f71f3
osd.46 up   in  weight 1 up_from 780 up_thru 833 down_at 756
last_clean_interval [710,755) 10.2.2.103:6830/4069
10.3.2.103:6820/4069 10.3.2.103:6821/4069 10.2.2.103:6831/4069
exists,up b1cc292c-b56a-4949-a9cd-345d7591b4be
osd.47 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [466,614) 10.2.1.107:6824/3426
10.3.1.107:6816/3426 10.3.1.107:6817/3426 10.2.1.107:6825/3426
exists,up 646b30a3-3ce1-4f69-a915-b2674db5d4a2
osd.48 up   in  weight 1 up_from 772 up_thru 810 down_at 748
last_clean_interval [693,747) 10.2.2.109:6818/3143
10.3.2.109:6812/3143 10.3.2.109:6813/3143 10.2.2.109:6819/3143
exists,up c763ff49-047e-46b2-b95b-0ce66c9abac4
osd.49 up   in  weight 1 up_from 773 up_thru 810 down_at 752
last_clean_interval [694,751) 10.2.2.110:6827/3710
10.3.2.110:6818/3710 10.3.2.110:6819/3710 10.2.2.110:6828/3710
exists,up 4101b6d3-d149-4c22-8175-e8dfe6e432b2
osd.50 up   in  weight 1 up_from 639 up_thru 810 down_at 633
last_clean_interval [606,632) 10.2.3.106:6830/3753
10.3.3.106:6820/3753 10.3.3.106:6821/3753 10.2.3.106:6831/3753
exists,up 978fcb37-e08e-486d-96c7-32cf476da927
osd.51 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [662,738) 10.2.3.111:6812/4383
10.3.3.111:6809/4383 10.3.3.111:6813/4383 10.2.3.111:6816/4383
exists,up b9061b7c-d712-4739-8528-4acf8a529d13
osd.52 up   in  weight 1 up_from 647 up_thru 833 down_at 630
last_clean_interval [598,629) 10.2.3.105:6842/5408
10.3.3.105:6828/5408 10.3.3.105:6829/5408 10.2.3.105:6843/5408
exists,up d1ac7f8d-450a-4f86-948a-ea9c21b44e6f
osd.53 up   in  weight 1 up_from 627 up_thru 833 down_at 619
last_clean_interval [452,618) 10.2.1.108:6845/5640
10.3.1.108:6830/5640 10.3.1.108:6831/5640 10.2.1.108:6846/5640
exists,up 2b818e93-1ad6-420f-864b-2a81355390fc
osd.54 up   in  weight 1 up_from 779 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6806/2964
10.3.2.104:6804/2964 10.3.2.104:6805/2964 10.2.2.104:6807/2964
exists,up b158bdd0-73e8-4923-96c6-1f5cf63c95d6
osd.55 up   in  weight 1 up_from 736 up_thru 833 down_at 727
last_clean_interval [720,726) 10.2.1.102:6845/5724
10.3.1.102:6830/5724 10.3.1.102:6831/5724 10.2.1.102:6846/5724
exists,up 7484647f-cb80-4d65-9eb1-9aa559f0f978
osd.56 up   in  weight 1 up_from 734 up_thru 833 down_at 722
last_clean_interval [717,721) 10.2.1.101:6836/5114
10.3.1.101:6824/5114 10.3.1.101:6825/5114 10.2.1.101:6837/5114
exists,up 27246b24-7149-481a-b195-73e208265a08
osd.57 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [671,743) 10.2.3.112:6857/7560
10.3.3.112:6838/7560 10.3.3.112:6839/7560 10.2.3.112:6858/7560
exists,up dd22f8d2-03df-41b5-b039-cf3bf41e22e7
osd.58 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [711,755) 10.2.2.103:6815/2992
10.3.2.103:6810/2992 10.3.2.103:6811/2992 10.2.2.103:6816/2992
exists,up 619e2e5b-8c74-4170-a898-67a5aeaebcb6
osd.59 up   in  weight 1 up_from 623 up_thru 810 down_at 615
last_clean_interval [467,614) 10.2.1.107:6809/2975
10.3.1.107:6806/2975 10.3.1.107:6807/2975 10.2.1.107:6810/2975
exists,up 02e15cc1-7380-4235-981e-cd83e909f66d
osd.60 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6827/3851
10.3.2.109:6818/3851 10.3.2.109:6819/3851 10.2.2.109:6828/3851
exists,up c8c17a47-013e-4957-aab8-dc98dad878cd
osd.61 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [608,632) 10.2.3.106:6824/3275
10.3.3.106:6816/3275 10.3.3.106:6817/3275 10.2.3.106:6825/3275
exists,up 8def34c7-a5b8-4d04-b724-ae8627830fe3
osd.62 up   in  weight 1 up_from 777 up_thru 833 down_at 752
last_clean_interval [693,751) 10.2.2.110:6857/8125
10.3.2.110:6838/8125 10.3.2.110:6839/8125 10.2.2.110:6858/8125
exists,up 42259595-c820-407e-9a6e-654702dbc89b
osd.63 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [662,738) 10.2.3.111:6809/4371
10.3.3.111:6806/4371 10.3.3.111:6810/4371 10.2.3.111:6813/4371
exists,up b177106c-c4e2-4d01-bc9f-3df0494e5edb
osd.64 up   in  weight 1 up_from 645 up_thru 833 down_at 630
last_clean_interval [597,629) 10.2.3.105:6818/3419
10.3.3.105:6812/3419 10.3.3.105:6813/3419 10.2.3.105:6819/3419
exists,up 7edc4897-a98d-4810-be69-710a6b46350b
osd.65 up   in  weight 1 up_from 628 up_thru 833 down_at 619
last_clean_interval [455,618) 10.2.1.108:6854/6690
10.3.1.108:6836/6690 10.3.1.108:6837/6690 10.2.1.108:6855/6690
exists,up 11ac2519-2dd3-4674-96c4-90de484fbe73
osd.66 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [718,726) 10.2.1.102:6821/3318
10.3.1.102:6814/3318 10.3.1.102:6815/3318 10.2.1.102:6822/3318
exists,up 1be2f979-0a99-4e66-8899-4e7b496e1002
osd.67 up   in  weight 1 up_from 781 up_thru 833 down_at 760
last_clean_interval [712,759) 10.2.2.104:6839/4806
10.3.2.104:6826/4806 10.3.2.104:6827/4806 10.2.2.104:6840/4806
exists,up 7b6301c7-ec16-4d1f-a163-21c656984245
osd.68 up   in  weight 1 up_from 733 up_thru 833 down_at 722
last_clean_interval [719,721) 10.2.1.101:6812/3044
10.3.1.101:6808/3044 10.3.1.101:6809/3044 10.2.1.101:6813/3044
exists,up 87cd5099-c6ef-4a9a-814a-0e5a8a724e22
osd.69 up   in  weight 1 up_from 766 up_thru 833 down_at 744
last_clean_interval [669,743) 10.2.3.112:6803/3803
10.3.3.112:6806/3803 10.3.3.112:6807/3803 10.2.3.112:6810/3803
exists,up 165d0685-58db-4bfc-a169-db88e0f2c213
osd.70 up   in  weight 1 up_from 781 up_thru 810 down_at 756
last_clean_interval [711,755) 10.2.2.103:6848/5951
10.3.2.103:6832/5951 10.3.2.103:6833/5951 10.2.2.103:6849/5951
exists,up a4117bd8-5b70-48ed-a784-8ef7d0c6edf3
osd.71 up   in  weight 1 up_from 626 up_thru 833 down_at 615
last_clean_interval [465,614) 10.2.1.107:6851/6097
10.3.1.107:6834/6097 10.3.1.107:6835/6097 10.2.1.107:6852/6097
exists,up 4a74899c-32a0-42d0-a963-28311954391f
osd.72 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6812/3118
10.3.2.109:6808/3118 10.3.2.109:6809/3118 10.2.2.109:6813/3118
exists,up 52b0ac5e-aa5c-4821-a4ac-27f2cbdf1e95
osd.73 up   in  weight 1 up_from 639 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6833/4204
10.3.3.106:6822/4204 10.3.3.106:6823/4204 10.2.3.106:6834/4204
exists,up 65bf6fc9-83dd-4330-b10a-313395e51492
osd.74 up   in  weight 1 up_from 775 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6848/6317
10.3.2.110:6832/6317 10.3.2.110:6833/6317 10.2.2.110:6849/6317
exists,up af32455f-b8c1-474a-9f3c-ee2453a14232
osd.75 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [663,738) 10.2.3.111:6848/4652
10.3.3.111:6832/4652 10.3.3.111:6833/4652 10.2.3.111:6849/4652
exists,up 0ce63ecf-80fc-404d-a9a2-4b1549f5f193
osd.76 up   in  weight 1 up_from 644 up_thru 833 down_at 630
last_clean_interval [594,629) 10.2.3.105:6803/2693
10.3.3.105:6802/2693 10.3.3.105:6803/2693 10.2.3.105:6804/2693
exists,up 410d5647-ae84-46a8-8048-4832964a2615
osd.77 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [719,726) 10.2.1.102:6816/3134
10.3.1.102:6811/3134 10.3.1.102:6813/3134 10.2.1.102:6818/3134
exists,up 30723422-ce2a-4354-8549-1465dcc1a05a
osd.78 up   in  weight 1 up_from 628 up_thru 833 down_at 619
last_clean_interval [458,618) 10.2.1.108:6851/6040
10.3.1.108:6834/6040 10.3.1.108:6835/6040 10.2.1.108:6852/6040
exists,up 7b759d33-6511-4f66-af03-3d891aeb1083
osd.79 up   in  weight 1 up_from 781 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6836/4620
10.3.2.104:6824/4620 10.3.2.104:6825/4620 10.2.2.104:6837/4620
exists,up ea12e5e8-d880-4f88-96d5-cfae6bc3f890
osd.80 up   in  weight 1 up_from 733 up_thru 810 down_at 722
last_clean_interval [716,721) 10.2.1.101:6818/3297
10.3.1.101:6812/3297 10.3.1.101:6813/3297 10.2.1.101:6819/3297
exists,up 61d7defb-d48d-40dd-b4cf-925caebc83e3
osd.81 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [667,742) 10.2.3.112:6801/3807
10.3.3.112:6800/3807 10.3.3.112:6803/3807 10.2.3.112:6804/3807
exists,up 040162fc-3b7e-48bb-8754-0817ec2449a1
osd.82 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [465,614) 10.2.1.107:6815/3078
10.3.1.107:6810/3078 10.3.1.107:6811/3078 10.2.1.107:6816/3078
exists,up 14601e93-8f09-48da-abb6-eeccf2d1c043
osd.83 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [711,755) 10.2.2.103:6821/3260
10.3.2.103:6814/3260 10.3.2.103:6815/3260 10.2.2.103:6822/3260
exists,up 45158a13-f8be-4a8e-963d-9b617b5d4507
osd.84 up   in  weight 1 up_from 772 up_thru 810 down_at 748
last_clean_interval [696,747) 10.2.2.109:6803/3001
10.3.2.109:6802/3001 10.3.2.109:6803/3001 10.2.2.109:6804/3001
exists,up fc50b1c6-cc19-4b81-8bc8-8e3d007f0ce4
osd.85 up   in  weight 1 up_from 641 up_thru 833 down_at 633
last_clean_interval [608,632) 10.2.3.106:6854/6342
10.3.3.106:6836/6342 10.3.3.106:6837/6342 10.2.3.106:6855/6342
exists,up 6494c679-0ad6-45cf-b179-38f554da46f3
osd.86 up   in  weight 1 up_from 774 up_thru 833 down_at 752
last_clean_interval [695,751) 10.2.2.110:6836/5200
10.3.2.110:6824/5200 10.3.2.110:6825/5200 10.2.2.110:6837/5200
exists,up 152c4c3f-38ae-4f87-bf55-d23be5bc01f5
osd.87 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [660,738) 10.2.3.111:6827/4413
10.3.3.111:6818/4413 10.3.3.111:6819/4413 10.2.3.111:6828/4413
exists,up 35401474-b0e3-47d2-9a46-e7a565d11543
osd.88 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [450,618) 10.2.1.108:6812/3006
10.3.1.108:6808/3006 10.3.1.108:6809/3006 10.2.1.108:6813/3006
exists,up 2cf7e048-b515-4093-8733-af82159bd0c8
osd.89 up   in  weight 1 up_from 648 up_thru 833 down_at 630
last_clean_interval [595,629) 10.2.3.105:6851/6100
10.3.3.105:6834/6100 10.3.3.105:6835/6100 10.2.3.105:6852/6100
exists,up 7a487064-7255-4c30-8cf0-a0c731d00bea
osd.90 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [719,726) 10.2.1.102:6812/3112
10.3.1.102:6808/3112 10.3.1.102:6809/3112 10.2.1.102:6813/3112
exists,up 9be24215-0832-4384-b28b-c49d39c58d5b
osd.91 up   in  weight 1 up_from 782 up_thru 833 down_at 760
last_clean_interval [713,759) 10.2.2.104:6845/5900
10.3.2.104:6830/5900 10.3.2.104:6831/5900 10.2.2.104:6846/5900
exists,up 203b80c5-2c4f-4250-a3f7-1c9b1954730e
osd.92 up   in  weight 1 up_from 735 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6848/6336
10.3.1.101:6832/6336 10.3.1.101:6833/6336 10.2.1.101:6849/6336
exists,up 109f9a06-039a-4aee-9347-52ba3bda74a3
osd.93 up   in  weight 1 up_from 627 up_thru 833 down_at 615
last_clean_interval [468,614) 10.2.1.107:6854/6395
10.3.1.107:6836/6395 10.3.1.107:6837/6395 10.2.1.107:6855/6395
exists,up 9c9507c6-4ea5-4f4f-a675-6145ff5f9d30
osd.94 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [668,743) 10.2.3.112:6851/7385
10.3.3.112:6834/7385 10.3.3.112:6835/7385 10.2.3.112:6852/7385
exists,up fdc854b9-9026-46de-8019-d7dc95380d65
osd.95 up   in  weight 1 up_from 780 up_thru 833 down_at 756
last_clean_interval [710,755) 10.2.2.103:6827/3819
10.3.2.103:6818/3819 10.3.2.103:6819/3819 10.2.2.103:6828/3819
exists,up 206b41d8-a15f-4633-961b-fac54e019dff
osd.96 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [692,747) 10.2.2.109:6815/3126
10.3.2.109:6810/3126 10.3.2.109:6811/3126 10.2.2.109:6816/3126
exists,up f2a67be3-01ac-4bdb-adc0-f2c80c077c0d
osd.97 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [608,632) 10.2.3.106:6809/2996
10.3.3.106:6806/2996 10.3.3.106:6807/2996 10.2.3.106:6810/2996
exists,up 500731d3-5d08-4031-9c10-3b0f79bc32aa
osd.98 up   in  weight 1 up_from 775 up_thru 833 down_at 752
last_clean_interval [693,751) 10.2.2.110:6842/5795
10.3.2.110:6828/5795 10.3.2.110:6829/5795 10.2.2.110:6843/5795
exists,up ac16d05d-3684-4ab3-9bea-531d97ee0d94
osd.99 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [661,738) 10.2.3.111:6803/4360
10.3.3.111:6802/4360 10.3.3.111:6803/4360 10.2.3.111:6804/4360
exists,up 1b3e1cce-45d7-4d12-8e58-4a582db5271b
osd.100 up   in  weight 1 up_from 624 up_thru 833 down_at 619
last_clean_interval [450,618) 10.2.1.108:6800/2724
10.3.1.108:6800/2724 10.3.1.108:6801/2724 10.2.1.108:6801/2724
exists,up 035502e6-9467-48e2-994c-6acf251d3d19
osd.101 up   in  weight 1 up_from 646 up_thru 833 down_at 630
last_clean_interval [594,629) 10.2.3.105:6839/4923
10.3.3.105:6826/4923 10.3.3.105:6827/4923 10.2.3.105:6840/4923
exists,up d065f777-7433-4817-9b4b-a95fa1987dc4
osd.102 up   in  weight 1 up_from 732 up_thru 833 down_at 722
last_clean_interval [719,721) 10.2.1.101:6803/2709
10.3.1.101:6802/2709 10.3.1.101:6803/2709 10.2.1.101:6804/2709
exists,up 84aef94d-a485-4b05-93b2-50bce1623f7e
osd.103 up   in  weight 1 up_from 783 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6854/6916
10.3.2.104:6836/6916 10.3.2.104:6837/6916 10.2.2.104:6855/6916
exists,up 75eb078d-bfdb-4e84-b52f-8c48e8d417dd
osd.104 up   in  weight 1 up_from 735 up_thru 833 down_at 727
last_clean_interval [716,726) 10.2.1.102:6833/4440
10.3.1.102:6822/4440 10.3.1.102:6823/4440 10.2.1.102:6834/4440
exists,up 67ec0cd8-64a6-4868-95a4-e0568ba0645e
osd.105 up   in  weight 1 up_from 625 up_thru 833 down_at 615
last_clean_interval [464,614) 10.2.1.107:6833/4418
10.3.1.107:6822/4418 10.3.1.107:6823/4418 10.2.1.107:6834/4418
exists,up 8c3c329e-bfaf-432d-97e0-a7719d708c0b
osd.106 up   in  weight 1 up_from 765 up_thru 833 down_at 744
last_clean_interval [667,742) 10.2.3.112:6824/3880
10.3.3.112:6816/3880 10.3.3.112:6817/3880 10.2.3.112:6825/3880
exists,up 2c3f5d03-909c-4966-aa6d-8d6a8edfa964
osd.107 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [712,755) 10.2.2.103:6806/2960
10.3.2.103:6804/2960 10.3.2.103:6805/2960 10.2.2.103:6807/2960
exists,up c2e25d4f-07a2-402b-83eb-2aebd70d6bc7
osd.108 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [694,747) 10.2.2.109:6821/3310
10.3.2.109:6814/3310 10.3.2.109:6815/3310 10.2.2.109:6822/3310
exists,up 2323f76d-de0a-4c2a-823a-2805675f900e
osd.109 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6803/2895
10.3.3.106:6802/2895 10.3.3.106:6803/2895 10.2.3.106:6804/2895
exists,up 21584a29-bcf7-4461-8867-6e6b337a4126
osd.110 up   in  weight 1 up_from 775 up_thru 833 down_at 752
last_clean_interval [693,751) 10.2.2.110:6839/5397
10.3.2.110:6826/5397 10.3.2.110:6827/5397 10.2.2.110:6840/5397
exists,up c180e193-f37f-43b9-9171-49deeee7d51b
osd.111 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [665,738) 10.2.3.111:6811/4378
10.3.3.111:6807/4378 10.3.3.111:6811/4378 10.2.3.111:6815/4378
exists,up 6c95ec0c-5383-4edf-a145-963303d1f804
osd.112 up   in  weight 1 up_from 645 up_thru 833 down_at 630
last_clean_interval [596,629) 10.2.3.105:6821/3552
10.3.3.105:6814/3552 10.3.3.105:6815/3552 10.2.3.105:6822/3552
exists,up b3644bf9-71a6-46eb-8850-6349752a8852
osd.113 up   in  weight 1 up_from 624 up_thru 833 down_at 619
last_clean_interval [454,618) 10.2.1.108:6803/2799
10.3.1.108:6802/2799 10.3.1.108:6803/2799 10.2.1.108:6804/2799
exists,up 2b6c0a5c-bf20-46d1-be66-fa01b8f8cf99
osd.114 up   in  weight 1 up_from 736 up_thru 833 down_at 722
last_clean_interval [717,721) 10.2.1.101:6854/6721
10.3.1.101:6836/6721 10.3.1.101:6837/6721 10.2.1.101:6855/6721
exists,up 375c9f34-1305-411d-8d1f-9767c073ad63
osd.115 up   in  weight 1 up_from 780 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6812/3063
10.3.2.104:6808/3063 10.3.2.104:6809/3063 10.2.2.104:6813/3063
exists,up 6beea134-f476-426b-bf31-c27d96b50e1a
osd.116 up   in  weight 1 up_from 766 up_thru 810 down_at 744
last_clean_interval [668,743) 10.2.3.112:6821/3879
10.3.3.112:6814/3879 10.3.3.112:6815/3879 10.2.3.112:6822/3879
exists,up 7880b830-5fe9-44d6-93f3-6884ab5e8e4f
osd.117 up   in  weight 1 up_from 735 up_thru 833 down_at 727
last_clean_interval [718,726) 10.2.1.102:6836/5134
10.3.1.102:6824/5134 10.3.1.102:6825/5134 10.2.1.102:6837/5134
exists,up b25b5589-18c6-4a3b-b09f-d69f7be89961
osd.118 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [471,614) 10.2.1.107:6818/3100
10.3.1.107:6812/3100 10.3.1.107:6813/3100 10.2.1.107:6819/3100
exists,up 802b2977-5a7e-4c36-a64e-0a28a5d91205
osd.119 up   in  weight 1 up_from 779 up_thru 810 down_at 756
last_clean_interval [709,755) 10.2.2.103:6803/2959
10.3.2.103:6802/2959 10.3.2.103:6803/2959 10.2.2.103:6804/2959
exists,up 13deeea5-dc85-41fe-89ea-4081bbadba2f
osd.120 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [694,747) 10.2.2.109:6806/3015
10.3.2.109:6804/3015 10.3.2.109:6805/3015 10.2.2.109:6807/3015
exists,up 83ce2668-e885-4c03-a64c-0aca75a182b5
osd.121 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6821/3102
10.3.3.106:6814/3102 10.3.3.106:6815/3102 10.2.3.106:6822/3102
exists,up 98a23c77-56b1-4ce1-8d54-870ef61845ae
osd.122 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [660,738) 10.2.3.111:6806/4367
10.3.3.111:6804/4367 10.3.3.111:6805/4367 10.2.3.111:6807/4367
exists,up 63af2474-6d2d-46bf-8c3e-6fde1cb1e3ef
osd.123 up   in  weight 1 up_from 775 up_thru 833 down_at 752
last_clean_interval [694,751) 10.2.2.110:6845/6171
10.3.2.110:6830/6171 10.3.2.110:6831/6171 10.2.2.110:6846/6171
exists,up a1772f6e-0067-41ec-ae9b-477f20788acb
osd.124 up   in  weight 1 up_from 644 up_thru 833 down_at 630
last_clean_interval [602,629) 10.2.3.105:6809/2913
10.3.3.105:6806/2913 10.3.3.105:6807/2913 10.2.3.105:6810/2913
exists,up b5eb3007-f17f-4b03-8aea-2fd9cc70685d
osd.125 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [451,618) 10.2.1.108:6806/2877
10.3.1.108:6804/2877 10.3.1.108:6805/2877 10.2.1.108:6807/2877
exists,up 6d7f771a-174d-4492-875c-a025574f3ade
osd.126 up   in  weight 1 up_from 779 up_thru 833 down_at 760
last_clean_interval [711,759) 10.2.2.104:6809/2977
10.3.2.104:6806/2977 10.3.2.104:6807/2977 10.2.2.104:6810/2977
exists,up 531a8bfe-65da-4431-b261-92e7e5f41ed4
osd.127 up   in  weight 1 up_from 732 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6800/2634
10.3.1.101:6800/2634 10.3.1.101:6801/2634 10.2.1.101:6801/2634
exists,up bf08cf79-7719-426e-890c-90b154528302
osd.128 up   in  weight 1 up_from 766 up_thru 810 down_at 744
last_clean_interval [670,743) 10.2.3.112:6818/3837
10.3.3.112:6812/3837 10.3.3.112:6813/3837 10.2.3.112:6819/3837
exists,up 1d1a4c8f-22c4-4fbe-ad1a-920213c54788
osd.129 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [466,614) 10.2.1.107:6800/2933
10.3.1.107:6800/2933 10.3.1.107:6801/2933 10.2.1.107:6801/2933
exists,up 1873ac3e-d52a-4030-a2e8-97d10a1436c6
osd.130 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [717,726) 10.2.1.102:6806/2972
10.3.1.102:6804/2972 10.3.1.102:6805/2972 10.2.1.102:6807/2972
exists,up a7688209-d8f6-46a0-902a-ec7b6061ffa8
osd.131 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [709,755) 10.2.2.103:6800/2956
10.3.2.103:6800/2956 10.3.2.103:6801/2956 10.2.2.103:6801/2956
exists,up 69a79457-b529-4d25-873d-215f4b2063ae
osd.132 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [692,747) 10.2.2.109:6800/2987
10.3.2.109:6800/2987 10.3.2.109:6801/2987 10.2.2.109:6801/2987
exists,up 24fcd073-8aa4-4c74-b44f-d86e007fc006
osd.133 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [608,632) 10.2.3.106:6800/2880
10.3.3.106:6800/2880 10.3.3.106:6801/2880 10.2.3.106:6801/2880
exists,up f9e3ef5d-c4be-4598-9cbd-9d12bed4da02
osd.134 up   in  weight 1 up_from 773 up_thru 833 down_at 752
last_clean_interval [695,751) 10.2.2.110:6821/3086
10.3.2.110:6814/3086 10.3.2.110:6815/3086 10.2.2.110:6822/3086
exists,up ab77c0b0-8443-4de1-aec8-ceaa09e77cd6
osd.135 up   in  weight 1 up_from 646 up_thru 833 down_at 630
last_clean_interval [594,629) 10.2.3.105:6836/4663
10.3.3.105:6824/4663 10.3.3.105:6825/4663 10.2.3.105:6837/4663
exists,up 8bae89d3-0106-4144-a2d1-4d99b3ce0547
osd.136 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [663,738) 10.2.3.111:6854/6301
10.3.3.111:6836/6301 10.3.3.111:6837/6301 10.2.3.111:6855/6301
exists,up 5016f910-01dd-4b1d-9d7d-5ceb31903720
osd.137 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [450,618) 10.2.1.108:6830/4218
10.3.1.108:6820/4218 10.3.1.108:6821/4218 10.2.1.108:6831/4218
exists,up c5e4c454-0307-4922-9c67-bb8212928a34
osd.138 up   in  weight 1 up_from 779 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6803/2945
10.3.2.104:6802/2945 10.3.2.104:6803/2945 10.2.2.104:6804/2945
exists,up 5dd89d1f-a035-48c4-8ca6-b99b3a0afea9
osd.139 up   in  weight 1 up_from 735 up_thru 833 down_at 722
last_clean_interval [715,721) 10.2.1.101:6842/5523
10.3.1.101:6828/5523 10.3.1.101:6829/5523 10.2.1.101:6843/5523
exists,up ff2bdb05-cb46-4ff6-9186-f70e19933337
osd.140 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [668,743) 10.2.3.112:6830/3902
10.3.3.112:6820/3902 10.3.3.112:6821/3902 10.2.3.112:6833/3902
exists,up 48432e47-fabe-45a0-961c-a5399de0bd4f
osd.141 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [717,726) 10.2.1.102:6824/3583
10.3.1.102:6816/3583 10.3.1.102:6817/3583 10.2.1.102:6825/3583
exists,up aed7ced3-ae30-4b4c-9e39-1f63968512e6
osd.142 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [469,614) 10.2.1.107:6803/2935
10.3.1.107:6802/2935 10.3.1.107:6803/2935 10.2.1.107:6804/2935
exists,up f578ffb8-371b-43ef-ac7e-106996c7ac46
osd.143 up   in  weight 1 up_from 781 up_thru 833 down_at 756
last_clean_interval [709,755) 10.2.2.103:6851/6191
10.3.2.103:6834/6191 10.3.2.103:6835/6191 10.2.2.103:6852/6191
exists,up 41ab004a-2465-4e16-bee3-523d97930fb5
osd.144 up   in  weight 1 up_from 639 up_thru 833 down_at 633
last_clean_interval [609,632) 10.2.3.106:6836/4432
10.3.3.106:6824/4432 10.3.3.106:6825/4432 10.2.3.106:6837/4432
exists,up 15d1035c-fdec-40e1-9aa9-6c7ff8bdb340
osd.145 up   in  weight 1 up_from 647 up_thru 833 down_at 630
last_clean_interval [595,629) 10.2.3.105:6848/5703
10.3.3.105:6832/5703 10.3.3.105:6833/5703 10.2.3.105:6849/5703
exists,up 347e9729-473f-4d71-b6d6-daf81bda42ac
osd.146 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [663,738) 10.2.3.111:6851/5826
10.3.3.111:6834/5826 10.3.3.111:6835/5826 10.2.3.111:6852/5826
exists,up b0b33b68-3198-47a8-b28d-efe71904ffbd
osd.147 up   in  weight 1 up_from 733 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6815/3137
10.3.1.101:6810/3137 10.3.1.101:6811/3137 10.2.1.101:6816/3137
exists,up 84cc173f-08c9-4e47-8695-ecfbec734621
osd.148 up   in  weight 1 up_from 737 up_thru 833 down_at 727
last_clean_interval [718,726) 10.2.1.102:6857/6980
10.3.1.102:6838/6980 10.3.1.102:6839/6980 10.2.1.102:6858/6980
exists,up 627c6077-f371-4431-afe1-ae04689d54d9
osd.149 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [451,618) 10.2.1.108:6833/4433
10.3.1.108:6822/4433 10.3.1.108:6823/4433 10.2.1.108:6834/4433
exists,up cb5900ae-15ca-409b-aa34-99481bc94aa2
osd.150 up   in  weight 1 up_from 780 up_thru 833 down_at 760
last_clean_interval [713,759) 10.2.2.104:6821/3212
10.3.2.104:6814/3212 10.3.2.104:6815/3212 10.2.2.104:6822/3212
exists,up 2fc21617-4b10-49ef-962b-71f9c9704042
osd.151 up   in  weight 1 up_from 627 up_thru 833 down_at 615
last_clean_interval [466,614) 10.2.1.107:6857/6911
10.3.1.107:6838/6911 10.3.1.107:6839/6911 10.2.1.107:6858/6911
exists,up 2d9bfc88-4813-463a-874b-7609c49afc50
osd.152 up   in  weight 1 up_from 766 up_thru 833 down_at 744
last_clean_interval [668,743) 10.2.3.112:6832/3885
10.3.3.112:6823/3885 10.3.3.112:6825/3885 10.2.3.112:6836/3885
exists,up e41e1246-0c2c-4349-8231-4ebd75accd13
osd.153 up   in  weight 1 up_from 781 up_thru 833 down_at 756
last_clean_interval [709,755) 10.2.2.103:6842/5438
10.3.2.103:6828/5438 10.3.2.103:6829/5438 10.2.2.103:6843/5438
exists,up e53b93c0-0312-4f9f-b9e2-67ac5729102d
osd.154 up   in  weight 1 up_from 639 up_thru 810 down_at 633
last_clean_interval [608,632) 10.2.3.106:6827/3568
10.3.3.106:6818/3568 10.3.3.106:6819/3568 10.2.3.106:6828/3568
exists,up 1b795ade-36eb-4a19-96be-8631949db184
osd.155 up   in  weight 1 up_from 646 up_thru 833 down_at 630
last_clean_interval [595,629) 10.2.3.105:6830/4270
10.3.3.105:6820/4270 10.3.3.105:6821/4270 10.2.3.105:6831/4270
exists,up 8f635a5b-a3d5-4a43-a6c9-d41fa08ce2e5
osd.156 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [660,738) 10.2.3.111:6824/4398
10.3.3.111:6816/4398 10.3.3.111:6817/4398 10.2.3.111:6825/4398
exists,up 7267a2b8-e8e1-47d7-b1ff-ec36adb6beba
osd.157 up   in  weight 1 up_from 735 up_thru 810 down_at 722
last_clean_interval [717,721) 10.2.1.101:6845/5751
10.3.1.101:6830/5751 10.3.1.101:6831/5751 10.2.1.101:6846/5751
exists,up 1e286a10-7c79-4e16-addb-2a0424c4ee4f
osd.158 up   in  weight 1 up_from 627 up_thru 833 down_at 619
last_clean_interval [455,618) 10.2.1.108:6842/5368
10.3.1.108:6828/5368 10.3.1.108:6829/5368 10.2.1.108:6843/5368
exists,up 510065f0-1f05-4ee0-959e-f9350d320f51
osd.159 up   in  weight 1 up_from 780 up_thru 833 down_at 760
last_clean_interval [711,759) 10.2.2.104:6800/2943
10.3.2.104:6800/2943 10.3.2.104:6801/2943 10.2.2.104:6801/2943
exists,up b10ee30e-96fd-4246-816e-24e45817cab0
osd.160 up   in  weight 1 up_from 737 up_thru 810 down_at 727
last_clean_interval [719,726) 10.2.1.102:6830/4221
10.3.1.102:6820/4221 10.3.1.102:6821/4221 10.2.1.102:6831/4221
exists,up 34e8f23a-c151-48ff-bcf6-020be21fe1dc
osd.161 up   in  weight 1 up_from 766 up_thru 833 down_at 744
last_clean_interval [672,743) 10.2.3.112:6812/3821
10.3.3.112:6808/3821 10.3.3.112:6809/3821 10.2.3.112:6813/3821
exists,up 818641a6-e89b-4f73-8bbd-cfd4a240e5d5
osd.162 up   in  weight 1 up_from 626 up_thru 833 down_at 615
last_clean_interval [468,614) 10.2.1.107:6845/5576
10.3.1.107:6830/5576 10.3.1.107:6831/5576 10.2.1.107:6846/5576
exists,up 4677d33f-2bb1-4d65-b169-55a7ad358927
osd.163 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [713,755) 10.2.2.103:6824/3487
10.3.2.103:6816/3487 10.3.2.103:6817/3487 10.2.2.103:6825/3487
exists,up ebbddba5-5ff6-4104-a810-1787397d6a2e
osd.164 up   in  weight 1 up_from 641 up_thru 833 down_at 633
last_clean_interval [609,632) 10.2.3.106:6857/6890
10.3.3.106:6838/6890 10.3.3.106:6839/6890 10.2.3.106:6858/6890
exists,up d1319db9-47cc-4784-954d-3c3e47d8baf5
osd.165 up   in  weight 1 up_from 644 up_thru 833 down_at 630
last_clean_interval [596,629) 10.2.3.105:6812/2998
10.3.3.105:6808/2998 10.3.3.105:6809/2998 10.2.3.105:6813/2998
exists,up 7c872ba3-fc65-49bc-85ad-191b748e7ccb
osd.166 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [662,738) 10.2.3.111:6842/4476
10.3.3.111:6828/4476 10.3.3.111:6829/4476 10.2.3.111:6843/4476
exists,up d8c569d9-ca1e-4fe7-ad8d-bfa4b20e0a0a
osd.167 up   in  weight 1 up_from 736 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6857/6948
10.3.1.101:6838/6948 10.3.1.101:6839/6948 10.2.1.101:6858/6948
exists,up 9b8f3ae0-26d6-4522-a600-7f847ae67ac0
osd.168 up   in  weight 1 up_from 734 up_thru 833 down_at 727
last_clean_interval [717,726) 10.2.1.102:6803/2959
10.3.1.102:6802/2959 10.3.1.102:6803/2959 10.2.1.102:6804/2959
exists,up 64603922-ed2f-42cd-9117-57ed4bbfcdae
osd.169 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [454,618) 10.2.1.108:6827/3987
10.3.1.108:6818/3987 10.3.1.108:6819/3987 10.2.1.108:6828/3987
exists,up 6c91f84b-09a7-4807-8cbc-7ef2a46dd5aa
osd.170 up   in  weight 1 up_from 782 up_thru 833 down_at 760
last_clean_interval [711,759) 10.2.2.104:6842/4977
10.3.2.104:6828/4977 10.3.2.104:6829/4977 10.2.2.104:6843/4977
exists,up fe354752-5537-41e2-943b-f53fd66876f7
osd.171 up   in  weight 1 up_from 625 up_thru 833 down_at 615
last_clean_interval [467,614) 10.2.1.107:6836/4553
10.3.1.107:6824/4553 10.3.1.107:6825/4553 10.2.1.107:6837/4553
exists,up 79e61ba5-929b-44bc-a49d-4a9f394d5b10
osd.172 up   in  weight 1 up_from 766 up_thru 833 down_at 744
last_clean_interval [671,743) 10.2.3.112:6815/3825
10.3.3.112:6810/3825 10.3.3.112:6811/3825 10.2.3.112:6816/3825
exists,up cd5e2f57-bef2-4012-8b27-280da053cc63
osd.173 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [709,755) 10.2.2.103:6809/2962
10.3.2.103:6806/2962 10.3.2.103:6807/2962 10.2.2.103:6810/2962
exists,up 1e57357f-cdb2-4ada-a769-5b80348007ad
osd.174 up   in  weight 1 up_from 637 up_thru 833 down_at 633
last_clean_interval [609,632) 10.2.3.106:6815/3006
10.3.3.106:6810/3006 10.3.3.106:6811/3006 10.2.3.106:6816/3006
exists,up e7679fd2-92bd-44b4-b066-192f78ef4e6d
osd.175 up   in  weight 1 up_from 646 up_thru 833 down_at 630
last_clean_interval [598,629) 10.2.3.105:6833/4515
10.3.3.105:6822/4515 10.3.3.105:6823/4515 10.2.3.105:6834/4515
exists,up e95a6b8d-38bd-42c6-b1f8-e003bf6ce3ef
osd.176 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [663,738) 10.2.3.111:6857/7030
10.3.3.111:6838/7030 10.3.3.111:6839/7030 10.2.3.111:6858/7030
exists,up cd5baafc-38ba-4f64-ad1a-cefcf85a8641
osd.177 up   in  weight 1 up_from 733 up_thru 833 down_at 722
last_clean_interval [719,721) 10.2.1.101:6821/3493
10.3.1.101:6814/3493 10.3.1.101:6815/3493 10.2.1.101:6822/3493
exists,up b27c404a-557c-4864-b8b1-feee49520f54
osd.178 up   in  weight 1 up_from 736 up_thru 833 down_at 727
last_clean_interval [719,726) 10.2.1.102:6842/5576
10.3.1.102:6828/5576 10.3.1.102:6829/5576 10.2.1.102:6843/5576
exists,up 63c0ff29-42d2-461f-b876-74142fb43f7e
osd.179 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [467,614) 10.2.1.107:6827/3769
10.3.1.107:6818/3769 10.3.1.107:6819/3769 10.2.1.107:6828/3769
exists,up b856af22-5af2-420b-ba6b-7b5e2bea9c63
osd.180 up   in  weight 1 up_from 780 up_thru 833 down_at 760
last_clean_interval [711,759) 10.2.2.104:6824/3478
10.3.2.104:6816/3478 10.3.2.104:6817/3478 10.2.2.104:6825/3478
exists,up 22e1ea6d-97dd-4919-85db-cd9bff5d7e4c
osd.181 up   in  weight 1 up_from 766 up_thru 810 down_at 744
last_clean_interval [669,743) 10.2.3.112:6839/4028
10.3.3.112:6826/4028 10.3.3.112:6827/4028 10.2.3.112:6840/4028
exists,up 077507b9-c287-4c8d-9205-877d23ac34aa
osd.182 up   in  weight 1 up_from 626 up_thru 833 down_at 619
last_clean_interval [453,618) 10.2.1.108:6839/4825
10.3.1.108:6826/4825 10.3.1.108:6827/4825 10.2.1.108:6840/4825
exists,up 51fb0261-b23d-4c93-bf11-d2cf6615de51
osd.183 up   in  weight 1 up_from 780 up_thru 833 down_at 756
last_clean_interval [712,755) 10.2.2.103:6833/4228
10.3.2.103:6822/4228 10.3.2.103:6823/4228 10.2.2.103:6834/4228
exists,up 46fec890-499e-4b80-b36b-0d281b506ce6
osd.184 up   in  weight 1 up_from 640 up_thru 833 down_at 633
last_clean_interval [607,632) 10.2.3.106:6845/5635
10.3.3.106:6830/5635 10.3.3.106:6831/5635 10.2.3.106:6846/5635
exists,up 945d9843-7133-4a3b-9fd7-d6b69daba9c8
osd.185 up   in  weight 1 up_from 643 up_thru 833 down_at 630
last_clean_interval [596,629) 10.2.3.105:6800/2617
10.3.3.105:6800/2617 10.3.3.105:6801/2617 10.2.3.105:6801/2617
exists,up c2b763aa-95de-434b-a5c3-32972bb33a9d
osd.186 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [663,738) 10.2.3.111:6830/4418
10.3.3.111:6820/4418 10.3.3.111:6821/4418 10.2.3.111:6831/4418
exists,up 0ccfa09d-273e-4ef6-809e-dbf96a20c44b
osd.187 up   in  weight 1 up_from 733 up_thru 833 down_at 727
last_clean_interval [720,726) 10.2.1.102:6800/2805
10.3.1.102:6800/2805 10.3.1.102:6801/2805 10.2.1.102:6801/2805
exists,up 16c8d09a-c305-47dd-bdc9-ba9398de59c1
osd.188 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [450,618) 10.2.1.108:6821/3384
10.3.1.108:6814/3384 10.3.1.108:6815/3384 10.2.1.108:6822/3384
exists,up 2d98439a-dd78-447d-8006-fc1a2f30cada
osd.189 up   in  weight 1 up_from 769 up_thru 810 down_at 744
last_clean_interval [670,743) 10.2.3.112:6827/3882
10.3.3.112:6818/3882 10.3.3.112:6819/3882 10.2.3.112:6828/3882
exists,up 3662ce46-99ad-4c22-b323-ca2ab51d36ab
osd.190 up   in  weight 1 up_from 785 up_thru 833 down_at 760
last_clean_interval [710,759) 10.2.2.104:6857/8883
10.3.2.104:6838/8883 10.3.2.104:6839/8883 10.2.2.104:6858/8883
exists,up 9dcf79e9-2645-4bf2-ad20-76f7da755b0a
osd.191 up   in  weight 1 up_from 734 up_thru 833 down_at 722
last_clean_interval [718,721) 10.2.1.101:6830/4348
10.3.1.101:6820/4348 10.3.1.101:6821/4348 10.2.1.101:6831/4348
exists,up d3924dcf-fd0c-4cc9-a85f-4534a71529ab
osd.192 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [465,614) 10.2.1.107:6806/2964
10.3.1.107:6804/2964 10.3.1.107:6805/2964 10.2.1.107:6807/2964
exists,up 1765d13c-5511-4f64-ad6a-baf3ed25f948
osd.193 up   in  weight 1 up_from 782 up_thru 833 down_at 756
last_clean_interval [709,755) 10.2.2.103:6854/6787
10.3.2.103:6836/6787 10.3.2.103:6837/6787 10.2.2.103:6855/6787
exists,up db1d062f-2298-4eb8-bcc1-b3f23a4c130a
osd.194 up   in  weight 1 up_from 638 up_thru 810 down_at 633
last_clean_interval [611,632) 10.2.3.106:6812/3002
10.3.3.106:6808/3002 10.3.3.106:6809/3002 10.2.3.106:6813/3002
exists,up fdc505e1-1034-42e4-b161-6b730bc4c0e0
osd.195 up   in  weight 1 up_from 644 up_thru 833 down_at 630
last_clean_interval [600,629) 10.2.3.105:6806/2794
10.3.3.105:6804/2794 10.3.3.105:6805/2794 10.2.3.105:6807/2794
exists,up 6b2d3689-723f-419a-92f3-af6a7c218bce
osd.196 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [662,738) 10.2.3.111:6833/4435
10.3.3.111:6822/4435 10.3.3.111:6823/4435 10.2.3.111:6834/4435
exists,up 15adb7b5-b472-4da0-b133-a0d1e35cba87
osd.197 up   in  weight 1 up_from 736 up_thru 833 down_at 727
last_clean_interval [720,726) 10.2.1.102:6839/5380
10.3.1.102:6826/5380 10.3.1.102:6827/5380 10.2.1.102:6840/5380
exists,up 9cbfd072-1054-4383-a796-55412dbe3417
osd.198 up   in  weight 1 up_from 734 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6827/4166
10.3.1.101:6818/4166 10.3.1.101:6819/4166 10.2.1.101:6828/4166
exists,up 0f95e905-0fa5-49df-a3a3-58542e5bb64a
osd.199 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [670,742) 10.2.3.112:6854/7503
10.3.3.112:6836/7503 10.3.3.112:6837/7503 10.2.3.112:6855/7503
exists,up 6fd4c03f-e938-4f31-8c1d-f91e589084ae
osd.200 up   in  weight 1 up_from 780 up_thru 833 down_at 760
last_clean_interval [712,759) 10.2.2.104:6818/3078
10.3.2.104:6812/3078 10.3.2.104:6813/3078 10.2.2.104:6819/3078
exists,up 30034130-8b62-4ef9-a26a-91e37e05d31b
osd.201 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [453,618) 10.2.1.108:6809/3004
10.3.1.108:6806/3004 10.3.1.108:6807/3004 10.2.1.108:6810/3004
exists,up 7ff2661b-e952-4da4-a6b0-ba746df1325c
osd.202 up   in  weight 1 up_from 626 up_thru 833 down_at 615
last_clean_interval [467,614) 10.2.1.107:6848/5858
10.3.1.107:6832/5858 10.3.1.107:6833/5858 10.2.1.107:6849/5858
exists,up e68135cb-4952-4f18-9ec8-b19991fd61d0
osd.203 up   in  weight 1 up_from 781 up_thru 833 down_at 756
last_clean_interval [708,755) 10.2.2.103:6845/5730
10.3.2.103:6830/5730 10.3.2.103:6831/5730 10.2.2.103:6846/5730
exists,up 00e0999c-7a30-4131-b8eb-df5324a701aa
osd.204 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6818/3027
10.3.3.106:6812/3027 10.3.3.106:6813/3027 10.2.3.106:6819/3027
exists,up 65425980-add5-486e-9653-9dfe5bbc9cd4
osd.205 up   in  weight 1 up_from 645 up_thru 833 down_at 630
last_clean_interval [596,629) 10.2.3.105:6815/3186
10.3.3.105:6810/3186 10.3.3.105:6811/3186 10.2.3.105:6816/3186
exists,up 893a19c6-0ccb-47f4-951c-f70b08217974
osd.206 up   in  weight 1 up_from 769 up_thru 833 down_at 739
last_clean_interval [665,738) 10.2.3.111:6836/4430
10.3.3.111:6824/4430 10.3.3.111:6825/4430 10.2.3.111:6837/4430
exists,up 171bde82-27c7-44b8-86da-23dc1a9cfbb5
osd.207 up   in  weight 1 up_from 734 up_thru 810 down_at 727
last_clean_interval [718,726) 10.2.1.102:6815/3133
10.3.1.102:6810/3133 10.3.1.102:6812/3133 10.2.1.102:6817/3133
exists,up 85bb699f-3afa-4ce4-ba44-68994fca09ff
osd.208 up   in  weight 1 up_from 780 up_thru 810 down_at 760
last_clean_interval [709,759) 10.2.2.104:6815/3075
10.3.2.104:6810/3075 10.3.2.104:6811/3075 10.2.2.104:6816/3075
exists,up a5ca98fb-bdb8-4e34-af80-a1c2f2c4077b
osd.209 up   in  weight 1 up_from 766 up_thru 833 down_at 744
last_clean_interval [668,742) 10.2.3.112:6842/4606
10.3.3.112:6828/4606 10.3.3.112:6829/4606 10.2.3.112:6843/4606
exists,up 2042b18d-8eb4-45a2-ab26-7b02a8b41d6c
osd.210 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [450,618) 10.2.1.108:6818/3175
10.3.1.108:6812/3175 10.3.1.108:6813/3175 10.2.1.108:6819/3175
exists,up dd916c66-988a-441e-9b85-99a07f1f8203
osd.211 up   in  weight 1 up_from 734 up_thru 833 down_at 722
last_clean_interval [718,721) 10.2.1.101:6833/4600
10.3.1.101:6822/4600 10.3.1.101:6823/4600 10.2.1.101:6834/4600
exists,up 61ca87ae-d265-49f3-905c-821fbb065b3c
osd.212 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [467,614) 10.2.1.107:6821/3249
10.3.1.107:6814/3249 10.3.1.107:6815/3249 10.2.1.107:6822/3249
exists,up cda980b5-c69a-44f9-82c8-4a5269c897a7
osd.213 up   in  weight 1 up_from 782 up_thru 833 down_at 756
last_clean_interval [710,755) 10.2.2.103:6857/7059
10.3.2.103:6838/7059 10.3.2.103:6839/7059 10.2.2.103:6858/7059
exists,up 64a7032f-67d8-4cf8-ae8d-27ca372da43f
osd.214 up   in  weight 1 up_from 638 up_thru 833 down_at 633
last_clean_interval [606,632) 10.2.3.106:6806/2983
10.3.3.106:6804/2983 10.3.3.106:6805/2983 10.2.3.106:6807/2983
exists,up 3d5ed6ee-9925-4ff0-8b66-fba838b5b1ca
osd.215 up   in  weight 1 up_from 645 up_thru 833 down_at 630
last_clean_interval [596,629) 10.2.3.105:6827/3927
10.3.3.105:6818/3927 10.3.3.105:6819/3927 10.2.3.105:6828/3927
exists,up 64d1c96c-e2a8-4cde-a3a3-c99d93cc4235
osd.216 up   in  weight 1 up_from 766 up_thru 833 down_at 739
last_clean_interval [664,738) 10.2.3.111:6839/4438
10.3.3.111:6826/4438 10.3.3.111:6827/4438 10.2.3.111:6840/4438
exists,up 2b01fe2f-c0f7-47bc-ae81-d75efd8caffd
osd.217 up   in  weight 1 up_from 737 up_thru 833 down_at 727
last_clean_interval [720,726) 10.2.1.102:6854/6702
10.3.1.102:6836/6702 10.3.1.102:6837/6702 10.2.1.102:6855/6702
exists,up d609f789-d526-4c70-9f97-984c3cceb960
osd.218 up   in  weight 1 up_from 733 up_thru 833 down_at 722
last_clean_interval [716,721) 10.2.1.101:6809/2999
10.3.1.101:6806/2999 10.3.1.101:6807/2999 10.2.1.101:6810/2999
exists,up e7efa69e-6eef-4a0e-a16f-304279aff297
osd.219 up   in  weight 1 up_from 782 up_thru 833 down_at 760
last_clean_interval [712,759) 10.2.2.104:6851/6245
10.3.2.104:6834/6245 10.3.2.104:6835/6245 10.2.2.104:6852/6245
exists,up fad72680-f9c4-4c42-8818-b1c9238d432d
osd.220 up   in  weight 1 up_from 769 up_thru 833 down_at 744
last_clean_interval [667,743) 10.2.3.112:6831/3892
10.3.3.112:6822/3892 10.3.3.112:6824/3892 10.2.3.112:6835/3892
exists,up 7eec9bed-ce71-4bea-bbdc-d156a2d08c08
osd.221 up   in  weight 1 up_from 779 up_thru 833 down_at 756
last_clean_interval [712,755) 10.2.2.103:6812/2963
10.3.2.103:6808/2963 10.3.2.103:6809/2963 10.2.2.103:6813/2963
exists,up 8c2dbd91-9660-48bc-9128-396e7603ab84
osd.222 up   in  weight 1 up_from 625 up_thru 833 down_at 619
last_clean_interval [456,618) 10.2.1.108:6815/3018
10.3.1.108:6810/3018 10.3.1.108:6811/3018 10.2.1.108:6816/3018
exists,up db0d0a07-3005-4e23-8f50-724c8568119c
osd.223 up   in  weight 1 up_from 624 up_thru 833 down_at 615
last_clean_interval [465,614) 10.2.1.107:6812/3049
10.3.1.107:6808/3049 10.3.1.107:6809/3049 10.2.1.107:6813/3049
exists,up 28ab05dd-db3c-4104-89a6-842c8e18d832
osd.226 up   in  weight 1 up_from 774 up_thru 833 down_at 748
last_clean_interval [693,747) 10.2.2.109:6848/6011
10.3.2.109:6832/6011 10.3.2.109:6833/6011 10.2.2.109:6849/6011
exists,up d1a81e58-5a32-4d73-b9b2-ed594c2fc857
osd.227 up   in  weight 1 up_from 774 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6845/5752
10.3.2.109:6830/5752 10.3.2.109:6831/5752 10.2.2.109:6846/5752
exists,up f1cca7f2-8bda-4f9e-b12a-f9632211f49b
osd.228 up   in  weight 1 up_from 775 up_thru 833 down_at 748
last_clean_interval [693,747) 10.2.2.109:6857/7079
10.3.2.109:6838/7079 10.3.2.109:6839/7079 10.2.2.109:6858/7079
exists,up 48fc07ac-436d-4831-b665-44000449a15b
osd.229 up   in  weight 1 up_from 775 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6854/6589
10.3.2.109:6836/6589 10.3.2.109:6837/6589 10.2.2.109:6855/6589
exists,up 4628d64d-5dfa-437e-95d8-824f1fdeb7f1
osd.230 up   in  weight 1 up_from 772 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6824/3548
10.3.2.109:6816/3548 10.3.2.109:6817/3548 10.2.2.109:6825/3548
exists,up 82985176-c9ad-41cc-9222-bf7aabd70d99
osd.231 up   in  weight 1 up_from 773 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6830/4234
10.3.2.109:6820/4234 10.3.2.109:6821/4234 10.2.2.109:6831/4234
exists,up df6a6f68-cdfc-45ef-b23e-11ba4c5953e0
osd.232 up   in  weight 1 up_from 771 up_thru 833 down_at 748
last_clean_interval [691,747) 10.2.2.109:6809/3024
10.3.2.109:6806/3024 10.3.2.109:6807/3024 10.2.2.109:6810/3024
exists,up 68ff4582-3607-4e3e-821e-c12f5e4734d1
osd.233 up   in  weight 1 up_from 774 up_thru 833 down_at 748
last_clean_interval [690,747) 10.2.2.109:6842/4915
10.3.2.109:6828/4915 10.3.2.109:6829/4915 10.2.2.109:6843/4915
exists,up 6ea4fcb4-45a2-44cf-97a1-903ea9795476
osd.234 up   in  weight 1 up_from 772 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6809/2925
10.3.2.110:6806/2925 10.3.2.110:6807/2925 10.2.2.110:6810/2925
exists,up 7c0ca822-841f-40f1-921e-ddb3d4613313
osd.235 up   in  weight 1 up_from 773 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6830/3901
10.3.2.110:6820/3901 10.3.2.110:6821/3901 10.2.2.110:6831/3901
exists,up b040d3f0-3ed6-4ad8-8dc9-d7e74ef4681f
osd.236 up   in  weight 1 up_from 772 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6806/2850
10.3.2.110:6804/2850 10.3.2.110:6805/2850 10.2.2.110:6807/2850
exists,up 3b46d8de-3447-45eb-ab9e-00a92c15fb99
osd.237 up   in  weight 1 up_from 773 up_thru 810 down_at 752
last_clean_interval [692,751) 10.2.2.110:6812/3055
10.3.2.110:6808/3055 10.3.2.110:6809/3055 10.2.2.110:6813/3055
exists,up 425901a5-7767-4bfb-af5f-31615247bcdb
osd.238 up   in  weight 1 up_from 772 up_thru 833 down_at 752
last_clean_interval [693,751) 10.2.2.110:6818/3073
10.3.2.110:6812/3073 10.3.2.110:6813/3073 10.2.2.110:6819/3073
exists,up cc496387-9d45-4c97-af67-f75cfbd70804
osd.239 up   in  weight 1 up_from 772 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6803/2770
10.3.2.110:6802/2770 10.3.2.110:6803/2770 10.2.2.110:6804/2770
exists,up 43087bd0-60b8-4d83-bdeb-b048e7ee57b6
osd.240 up   in  weight 1 up_from 772 up_thru 833 down_at 752
last_clean_interval [692,751) 10.2.2.110:6800/2756
10.3.2.110:6800/2756 10.3.2.110:6801/2756 10.2.2.110:6801/2756
exists,up abc35dfc-5e2b-4afc-aff5-2c5fb9d06c47
osd.241 up   in  weight 1 up_from 772 up_thru 810 down_at 752
last_clean_interval [692,751) 10.2.2.110:6815/3065
10.3.2.110:6810/3065 10.3.2.110:6811/3065 10.2.2.110:6816/3065
exists,up 2b944a94-1214-4ef1-8675-b717a815e219
blacklist 10.2.3.33:6800/5070 expires 2015-02-26 08:19:09.683066
blacklist 10.2.3.33:6800/5055 expires 2015-02-26 08:18:12.371958
blacklist 10.2.3.33:6800/5011 expires 2015-02-26 08:18:06.684058





On Thu, Feb 26, 2015 at 1:01 PM, John Spray <john.spray@xxxxxxxxxx> wrote:
On 26/02/2015 17:58, Wyllys Ingersoll wrote:
Yeah, I noticed that too, so I recreated both of those pools and it
still wont start. It crashes in a different place now, but still wont
start, even after running 'newfs'.  Attached is the debug log output
when I start ceph-mds

...
common/Thread.cc: In function 'int Thread::join(void**)' thread
7f0127612700 time 2015-02-26 07:55:09.734802
common/Thread.cc: 141: FAILED assert(status == 0)
   ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)
   1: (Thread::detach()+0) [0x8c30d0]
   2: (MonClient::shutdown()+0x50f) [0x881d5f]
   3: (MDS::suicide()+0xe5) [0x576285]
   4: (MDLog::handle_journaler_write_error(int)+0x5f) [0x7aab2f]
   5: (Context::complete(int)+0x9) [0x56d9a9]
   6: (Journaler::handle_write_error(int)+0x5e) [0x7b949e]
   7: (Journaler::_finish_write_head(int, Journaler::Header&,
Context*)+0x306) [0x7b9946]
   8: (Context::complete(int)+0x9) [0x56d9a9]
   9: (Objecter::check_op_pool_dne(Objecter::Op*)+0x214) [0x7ce6a4]
   10: (Objecter::C_Op_Map_Latest::finish(int)+0x124) [0x7cea04]
   11: (Context::complete(int)+0x9) [0x56d9a9]
   12: (Finisher::finisher_thread_entry()+0x1b8) [0x9aced8]
   13: (()+0x8182) [0x7f012db95182]
   14: (clone()+0x6d) [0x7f012c50befd]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2015-02-26 07:55:09.736256 7f0127612700 -1 common/Thread.cc: In
function 'int Thread::join(void**)' thread 7f0127612700 time
2015-02-26 07:55:09.734802
common/Thread.cc: 141: FAILED assert(status == 0)
That is still looking like a missing pool (the check_op_pool_dne frame in
the trace).  Can you post "ceph osd dump" and "ceph mds dump"?

John

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux