The OSDs are all there.
$ sudo ceph osd stat
osdmap e60609: 72 osds: 72 up, 72 in
an I have attached the result of ceph osd tree, and ceph osd dump commands.
I got some extra info about the network problem. A faulty network device has flooded the network eating up all the bandwidth so the OSDs were not able to properly communicate with each other. This has lasted for almost 1 day.
Thank you,
Laszlo
On 10.03.2017 12:19, Brad Hubbard wrote:
To me it looks like someone may have done an "rm" on these OSDs but
not removed them from the crushmap. This does not happen
automatically.
Do these OSDs show up in "ceph osd tree" and "ceph osd dump" ? If so,
paste the output.
Without knowing what exactly happened here it may be difficult to work
out how to proceed.
In order to go clean the primary needs to communicate with multiple
OSDs, some of which are marked DNE and seem to be uncontactable.
This seems to be more than a network issue (unless the outage is still
happening).
http://docs.ceph.com/docs/master/rados/operations/pg-states/?highlight=incomplete
On Fri, Mar 10, 2017 at 6:09 PM, Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx> wrote:
Hello,
I was informed that due to a networking issue the ceph cluster network was
affected. There was a huge packet loss, and network interfaces were
flipping. That's all I got.
This outage has lasted a longer period of time. So I assume that some OSD
may have been considered dead and the data from them has been moved away to
other PGs (this is what ceph is supposed to do if I'm correct). Probably
that was the point when the listed PGs have appeared into the picture.
From the query we can see this for one of those OSDs:
{
"peer": "14",
"pgid": "3.367",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": "[]",
"history": {
"epoch_created": 4,
"last_epoch_started": 54899,
"last_epoch_clean": 55143,
"last_epoch_split": 0,
"same_up_since": 60603,
"same_interval_since": 60603,
"same_primary_since": 60593,
"last_scrub": "2852'33528",
"last_scrub_stamp": "2017-02-26 02:36:55.210150",
"last_deep_scrub": "2852'16480",
"last_deep_scrub_stamp": "2017-02-21 00:14:08.866448",
"last_clean_scrub_stamp": "2017-02-26 02:36:55.210150"
},
"stats": {
"version": "0'0",
"reported_seq": "14",
"reported_epoch": "59779",
"state": "down+peering",
"last_fresh": "2017-02-27 16:30:16.230519",
"last_change": "2017-02-27 16:30:15.267995",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2017-02-27 16:30:16.230519",
"last_undegraded": "2017-02-27 16:30:16.230519",
"last_fullsized": "2017-02-27 16:30:16.230519",
"mapping_epoch": 60601,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 4,
"last_epoch_clean": 55143,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2852'33528",
"last_scrub_stamp": "2017-02-26 02:36:55.210150",
"last_deep_scrub": "2852'16480",
"last_deep_scrub_stamp": "2017-02-21 00:14:08.866448",
"last_clean_scrub_stamp": "2017-02-26 02:36:55.210150",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": "0",
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0
},
"up": [
28,
35,
2
],
"acting": [
28,
35,
2
],
"blocked_by": [],
"up_primary": 28,
"acting_primary": 28
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"current_last_stamp": "0.000000",
"current_info": {
"begin": "0.000000",
"end": "0.000000",
"version": "0'0",
"using_gmt": "1"
},
"history": []
}
},
Where can I read more about the meaning of each parameter, some of them have
quite self explanatory names, but not all (or probably we need a deeper
knowledge to understand them).
Isn't there any parameter that would say when was that OSD assigned to the
given PG? Also the stat_sum shows 0 for all its parameters. Why is it
blocking then?
Is there a way to tell the PG to forget about that OSD?
Thank you,
Laszlo
On 10.03.2017 03:05, Brad Hubbard wrote:
Can you explain more about what happened?
The query shows progress is blocked by the following OSDs.
"blocked_by": [
14,
17,
51,
58,
63,
64,
68,
70
],
Some of these OSDs are marked as "dne" (Does Not Exist).
peer": "17",
"dne": 1,
"peer": "51",
"dne": 1,
"peer": "58",
"dne": 1,
"peer": "64",
"dne": 1,
"peer": "70",
"dne": 1,
Can we get a complete background here please?
On Thu, Mar 9, 2017 at 10:53 PM, Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx>
wrote:
Hello,
After a major network outage our ceph cluster ended up with an inactive
PG:
# ceph health detail
HEALTH_WARN 1 pgs incomplete; 1 pgs stuck inactive; 1 pgs stuck unclean;
1
requests are blocked > 32 sec; 1 osds have slow requests
pg 3.367 is stuck inactive for 912263.766607, current state incomplete,
last
acting [28,35,2]
pg 3.367 is stuck unclean for 912263.766688, current state incomplete,
last
acting [28,35,2]
pg 3.367 is incomplete, acting [28,35,2]
1 ops are blocked > 268435 sec
1 ops are blocked > 268435 sec on osd.28
1 osds have slow requests
# ceph -s
cluster 6713d1b8-83da-11e6-aa79-525400d98c5a
health HEALTH_WARN
1 pgs incomplete
1 pgs stuck inactive
1 pgs stuck unclean
1 requests are blocked > 32 sec
monmap e3: 3 mons at
{tv-dl360-1=10.12.193.73:6789/0,tv-dl360-2=10.12.193.74:6789/0,tv-dl360-3=10.12.193.75:6789/0}
election epoch 72, quorum 0,1,2
tv-dl360-1,tv-dl360-2,tv-dl360-3
osdmap e60609: 72 osds: 72 up, 72 in
pgmap v3670252: 4864 pgs, 11 pools, 134 GB data, 23778 objects
490 GB used, 130 TB / 130 TB avail
4863 active+clean
1 incomplete
client io 0 B/s rd, 38465 B/s wr, 2 op/s
ceph pg repair doesn't change anything. What should I try to recover it?
Attached is the result of ceph pg query on the problem PG.
Thank you,
Laszlo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
epoch 60609
fsid 6713d1b8-83da-11e6-aa79-525400d98c5a
created 2016-10-17 11:28:35.501106
modified 2017-03-09 13:32:30.302087
flags
pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0
pool 1 'images' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2783 flags hashpspool stripe_width 0
removed_snaps [1~9,c~c2,cf~280]
pool 2 'instances' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 241 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 3 'volumes' replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 59778 flags hashpspool stripe_width 0
removed_snaps [1~13d]
pool 4 '.rgw.root' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 266 owner 18446744073709551615 flags hashpspool stripe_width 0
pool 5 '.rgw.control' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 270 flags hashpspool stripe_width 0
pool 6 '.rgw' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 272 flags hashpspool stripe_width 0
pool 7 '.rgw.gc' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 272 flags hashpspool stripe_width 0
pool 8 '.users.uid' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 279 flags hashpspool stripe_width 0
pool 9 '.rgw.buckets.index' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 285 flags hashpspool stripe_width 0
pool 10 '.rgw.buckets' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2846 flags hashpspool stripe_width 0
max_osd 72
osd.0 up in weight 1 up_from 60563 up_thru 60607 down_at 60556 last_clean_interval [59469,60562) 10.12.193.71:6826/2955 192.168.91.71:6808/1002955 192.168.91.71:6809/1002955 10.12.193.71:6816/1002955 exists,up bdf8c61d-83d8-4833-990c-3a5c0be404cf
osd.1 up in weight 1 up_from 60557 up_thru 60607 down_at 60556 last_clean_interval [59449,60556) 10.12.193.68:6815/1325 192.168.91.68:6801/1001325 192.168.91.68:6802/1001325 10.12.193.68:6805/1001325 exists,up a6acb440-3d13-48f9-abee-c67913201a2b
osd.2 up in weight 1 up_from 60607 up_thru 60607 down_at 60605 last_clean_interval [60603,60604) 10.12.193.72:6801/7265 192.168.91.72:6806/7265 192.168.91.72:6807/7265 10.12.193.72:6807/7265 exists,up 6a4db644-22c2-4a18-83d9-3c39c0a72966
osd.3 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [60376,60536) 10.12.193.70:6822/2913 192.168.91.70:6832/1002913 192.168.91.70:6833/1002913 10.12.193.70:6828/1002913 exists,up 1d282d68-5b07-43b1-8fa0-58c5444ae50f
osd.4 up in weight 1 up_from 60535 up_thru 60607 down_at 60534 last_clean_interval [59849,60534) 10.12.193.69:6810/21561 192.168.91.69:6826/1021561 192.168.91.69:6827/1021561 10.12.193.69:6825/1021561 exists,up 395394bc-91eb-4812-964b-329256b6eb67
osd.5 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [59442,60541) 10.12.193.67:6812/16762 192.168.91.67:6832/1016762 192.168.91.67:6833/1016762 10.12.193.67:6828/1016762 exists,up b5c925f3-17f7-4db8-ba0c-5badf702c0d0
osd.6 up in weight 1 up_from 60538 up_thru 60607 down_at 60534 last_clean_interval [59469,60537) 10.12.193.71:6803/2184 192.168.91.71:6828/1002184 192.168.91.71:6829/1002184 10.12.193.71:6802/1002184 exists,up 9427c51d-1303-40a8-952a-b1ebcaf20fda
osd.7 up in weight 1 up_from 60537 up_thru 60605 down_at 60534 last_clean_interval [59448,60535) 10.12.193.68:6824/1683 192.168.91.68:6828/1001683 192.168.91.68:6829/1001683 10.12.193.68:6812/1001683 exists,up ab9bfd4e-df2d-4fbd-bdb4-f8d2dd5154e7
osd.8 up in weight 1 up_from 60560 up_thru 60593 down_at 60556 last_clean_interval [59476,60559) 10.12.193.72:6826/6096 192.168.91.72:6803/1006096 192.168.91.72:6805/1006096 10.12.193.72:6804/1006096 exists,up 50e37b81-6eaf-4b0c-81a8-93cabc2a4e9d
osd.9 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [60386,60541) 10.12.193.70:6820/2864 192.168.91.70:6808/1002864 192.168.91.70:6835/1002864 10.12.193.70:6830/1002864 exists,up eb1d674b-a09a-4f57-807a-409ea2e3c597
osd.10 up in weight 1 up_from 60554 up_thru 60607 down_at 60553 last_clean_interval [59850,60553) 10.12.193.69:6812/21670 192.168.91.69:6822/1021670 192.168.91.69:6823/1021670 10.12.193.69:6823/1021670 exists,up 961437ac-0605-4e7a-bb47-65157c160241
osd.11 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [59443,60535) 10.12.193.67:6820/17254 192.168.91.67:6830/1017254 192.168.91.67:6831/1017254 10.12.193.67:6827/1017254 exists,up f17163d6-683f-486d-bd96-35b6615ecfd2
osd.12 up in weight 1 up_from 60538 up_thru 60593 down_at 60534 last_clean_interval [59470,60537) 10.12.193.71:6830/3144 192.168.91.71:6830/1003144 192.168.91.71:6832/1003144 10.12.193.71:6808/1003144 exists,up e7f2c05e-de6e-4805-aac5-d8ced7888679
osd.13 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [59449,60536) 10.12.193.68:6800/1062 192.168.91.68:6812/1001062 192.168.91.68:6834/1001062 10.12.193.68:6822/1001062 exists,up b9b40bc3-8ece-4b7a-aadf-95cd8a5a48de
osd.14 up in weight 1 up_from 60537 up_thru 60593 down_at 60534 last_clean_interval [59475,60536) 10.12.193.72:6808/5027 192.168.91.72:6831/1005027 192.168.91.72:6832/1005027 10.12.193.72:6822/1005027 exists,up bb437866-ead4-4663-b346-d0be19c6f6c1
osd.15 up in weight 1 up_from 60535 up_thru 60605 down_at 60534 last_clean_interval [60377,60534) 10.12.193.70:6816/2611 192.168.91.70:6824/1002611 192.168.91.70:6827/1002611 10.12.193.70:6825/1002611 exists,up 5aa55888-4c82-4718-872b-aa6559dc485c
osd.16 up in weight 1 up_from 60559 up_thru 60607 down_at 60556 last_clean_interval [59848,60557) 10.12.193.69:6802/21074 192.168.91.69:6800/1021074 192.168.91.69:6801/1021074 10.12.193.69:6801/1021074 exists,up d2b28ddc-285e-49a9-8bf8-8d4472faa617
osd.17 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59442,60535) 10.12.193.67:6814/16881 192.168.91.67:6826/1016881 192.168.91.67:6827/1016881 10.12.193.67:6825/1016881 exists,up 2b67b3d7-1274-4ce1-a9d3-6cc557d93254
osd.18 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [59469,60536) 10.12.193.71:6801/2188 192.168.91.71:6824/1002188 192.168.91.71:6825/1002188 10.12.193.71:6804/1002188 exists,up cef750c2-c5f0-49f8-b794-d3d4deb0274d
osd.19 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [59449,60541) 10.12.193.68:6828/1860 192.168.91.68:6835/1001860 192.168.91.68:6836/1001860 10.12.193.68:6830/1001860 exists,up bac0a6ea-89aa-43a7-a682-8f22da6f78c1
osd.20 up in weight 1 up_from 60536 up_thru 60593 down_at 60534 last_clean_interval [59475,60535) 10.12.193.72:6813/5331 192.168.91.72:6808/1005331 192.168.91.72:6815/1005331 10.12.193.72:6806/1005331 exists,up 04cceaba-ca17-482e-bc59-40ce6a203738
osd.21 up in weight 1 up_from 60545 up_thru 60607 down_at 60534 last_clean_interval [60367,60542) 10.12.193.70:6808/2199 192.168.91.70:6822/1002199 192.168.91.70:6834/1002199 10.12.193.70:6829/1002199 exists,up 7808e117-397b-472c-9670-4bdc483b6ec5
osd.22 up in weight 1 up_from 60548 up_thru 60607 down_at 60534 last_clean_interval [59851,60547) 10.12.193.69:6822/22136 192.168.91.69:6814/1022136 192.168.91.69:6815/1022136 10.12.193.69:6815/1022136 exists,up 449f717a-f9ec-4933-aaff-1d401ccfa957
osd.23 up in weight 1 up_from 60558 up_thru 60607 down_at 60556 last_clean_interval [59443,60557) 10.12.193.67:6818/17196 192.168.91.67:6800/1017196 192.168.91.67:6801/1017196 10.12.193.67:6801/1017196 exists,up e5e0c270-d5be-4ec9-a1b3-0fd02da51dda
osd.24 up in weight 1 up_from 60540 up_thru 60607 down_at 60534 last_clean_interval [59469,60539) 10.12.193.71:6822/2742 192.168.91.71:6836/1002742 192.168.91.71:6837/1002742 10.12.193.71:6812/1002742 exists,up afdca50a-8974-408c-841d-56f57a2acc9b
osd.25 up in weight 1 up_from 60547 up_thru 60607 down_at 60534 last_clean_interval [59449,60546) 10.12.193.68:6802/1052 192.168.91.68:6837/1001052 192.168.91.68:6838/1001052 10.12.193.68:6831/1001052 exists,up 23c66476-daa8-4cdc-bbfd-e58c4b38078c
osd.26 up in weight 1 up_from 60560 up_thru 60589 down_at 60556 last_clean_interval [59475,60559) 10.12.193.72:6805/4904 192.168.91.72:6809/1004904 192.168.91.72:6810/1004904 10.12.193.72:6809/1004904 exists,up 210e9831-b5ed-4c27-8ff1-89708fcff0e0
osd.27 up in weight 1 up_from 60554 up_thru 60607 down_at 60553 last_clean_interval [60373,60553) 10.12.193.70:6812/2407 192.168.91.70:6810/1002407 192.168.91.70:6841/1002407 10.12.193.70:6807/1002407 exists,up dfb407c8-9bd7-4ed9-bb82-3358b6d3dc5f
osd.28 up in weight 1 up_from 60593 up_thru 60607 down_at 60583 last_clean_interval [60576,60582) 10.12.193.69:6804/11178 192.168.91.69:6803/11178 192.168.91.69:6806/11178 10.12.193.69:6805/11178 exists,up c2adca59-6629-4ed8-bfe9-c43ce3a6d4b6
osd.29 up in weight 1 up_from 60552 up_thru 60607 down_at 60550 last_clean_interval [59441,60551) 10.12.193.67:6802/16057 192.168.91.67:6836/1016057 192.168.91.67:6837/1016057 10.12.193.67:6830/1016057 exists,up 4c88f2ec-faa2-4258-ae38-92b655b8b021
osd.30 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [59470,60541) 10.12.193.71:6828/3057 192.168.91.71:6838/1003057 192.168.91.71:6839/1003057 10.12.193.71:6818/1003057 exists,up 2d3b1780-c0c5-44a6-a7d6-9495275a607d
osd.31 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59448,60535) 10.12.193.68:6821/1581 192.168.91.68:6814/1001581 192.168.91.68:6816/1001581 10.12.193.68:6810/1001581 exists,up 13f17fba-b4af-433d-aea2-8439335f50dc
osd.32 up in weight 1 up_from 60537 up_thru 60593 down_at 60534 last_clean_interval [59476,60535) 10.12.193.72:6819/5743 192.168.91.72:6828/1005743 192.168.91.72:6830/1005743 10.12.193.72:6828/1005743 exists,up 91bf05a0-2218-4a3f-9c09-eaf88f1bd05f
osd.33 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [60363,60535) 10.12.193.70:6804/1995 192.168.91.70:6830/1001995 192.168.91.70:6831/1001995 10.12.193.70:6827/1001995 exists,up ec30c2a5-d225-4c99-b58f-9b5339a697c4
osd.34 up in weight 1 up_from 60543 up_thru 60607 down_at 60534 last_clean_interval [59849,60541) 10.12.193.69:6806/21324 192.168.91.69:6834/1021324 192.168.91.69:6835/1021324 10.12.193.69:6829/1021324 exists,up 1bfd424c-ddb3-4309-9f60-981c85224e37
osd.35 up in weight 1 up_from 60589 up_thru 60607 down_at 60585 last_clean_interval [60568,60584) 10.12.193.67:6805/11183 192.168.91.67:6804/11183 192.168.91.67:6805/11183 10.12.193.67:6806/11183 exists,up c612be90-28eb-42d7-93f2-a6a9bb6d8f17
osd.36 up in weight 1 up_from 60537 up_thru 60593 down_at 60534 last_clean_interval [59469,60536) 10.12.193.71:6817/2535 192.168.91.71:6826/1002535 192.168.91.71:6827/1002535 10.12.193.71:6800/1002535 exists,up ce8e5531-eec7-4dd0-a0bb-86f3b5699454
osd.37 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [59448,60536) 10.12.193.68:6809/1128 192.168.91.68:6830/1001128 192.168.91.68:6831/1001128 10.12.193.68:6816/1001128 exists,up 389d67d0-4732-4045-bafb-21dc5bde81b4
osd.38 up in weight 1 up_from 60537 up_thru 60593 down_at 60534 last_clean_interval [59476,60536) 10.12.193.72:6811/5295 192.168.91.72:6819/1005295 192.168.91.72:6826/1005295 10.12.193.72:6810/1005295 exists,up a688aab8-6efa-49ef-bc6f-f97786df3898
osd.39 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [60369,60541) 10.12.193.70:6806/2094 192.168.91.70:6818/1002094 192.168.91.70:6836/1002094 10.12.193.70:6833/1002094 exists,up 57f495b6-412b-482f-852d-e0ddfdac3e34
osd.40 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59850,60535) 10.12.193.69:6816/21940 192.168.91.69:6832/1021940 192.168.91.69:6833/1021940 10.12.193.69:6828/1021940 exists,up 08f745e0-b1d7-4e45-98a9-ebed7333fbb9
osd.41 up in weight 1 up_from 60557 up_thru 60607 down_at 60556 last_clean_interval [59443,60556) 10.12.193.67:6810/16634 192.168.91.67:6802/1016634 192.168.91.67:6803/1016634 10.12.193.67:6803/1016634 exists,up fb6f0ac3-d2f8-405c-8652-16f9e1077385
osd.42 up in weight 1 up_from 60560 up_thru 60607 down_at 60556 last_clean_interval [59468,60559) 10.12.193.71:6807/2186 192.168.91.71:6802/1002186 192.168.91.71:6803/1002186 10.12.193.71:6806/1002186 exists,up fa1ceca4-80a2-49a7-94cf-32728e8a49c9
osd.43 up in weight 1 up_from 60558 up_thru 60607 down_at 60556 last_clean_interval [59449,60557) 10.12.193.68:6803/1054 192.168.91.68:6810/1001054 192.168.91.68:6811/1001054 10.12.193.68:6811/1001054 exists,up db466c69-8f2d-4f57-99f9-361c871eb249
osd.44 up in weight 1 up_from 60542 up_thru 60593 down_at 60534 last_clean_interval [59474,60541) 10.12.193.72:6803/4855 192.168.91.72:6836/1004855 192.168.91.72:6837/1004855 10.12.193.72:6830/1004855 exists,up 177a09a0-fe35-45b2-8fb2-137cebe58b36
osd.45 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [60364,60536) 10.12.193.70:6802/1928 192.168.91.70:6828/1001928 192.168.91.70:6829/1001928 10.12.193.70:6826/1001928 exists,up a38276ab-6362-43d8-9643-946ae3c31568
osd.46 up in weight 1 up_from 60548 up_thru 60607 down_at 60534 last_clean_interval [59850,60547) 10.12.193.69:6814/21858 192.168.91.69:6804/1021858 192.168.91.69:6837/1021858 10.12.193.69:6830/1021858 exists,up b59d67b2-e9ea-457b-9ee8-dd1f28453c65
osd.47 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59441,60535) 10.12.193.67:6800/15914 192.168.91.67:6828/1015914 192.168.91.67:6829/1015914 10.12.193.67:6826/1015914 exists,up bd5541a9-7458-44d0-9fa8-5b319e59ad43
osd.48 up in weight 1 up_from 60539 up_thru 60607 down_at 60534 last_clean_interval [59469,60537) 10.12.193.71:6820/2629 192.168.91.71:6831/1002629 192.168.91.71:6833/1002629 10.12.193.71:6810/1002629 exists,up 74ee6c8a-6db8-4232-902e-b5e2d2a95a51
osd.49 up in weight 1 up_from 60560 up_thru 60607 down_at 60556 last_clean_interval [59449,60559) 10.12.193.68:6801/1059 192.168.91.68:6813/1001059 192.168.91.68:6818/1001059 10.12.193.68:6814/1001059 exists,up 73bf5889-e373-4d2b-9838-21df4890709c
osd.50 up in weight 1 up_from 60542 up_thru 60593 down_at 60534 last_clean_interval [59476,60541) 10.12.193.72:6817/5626 192.168.91.72:6840/1005626 192.168.91.72:6841/1005626 10.12.193.72:6831/1005626 exists,up ada64b06-b1dc-4f34-b375-489836aee510
osd.51 up in weight 1 up_from 60543 up_thru 60607 down_at 60534 last_clean_interval [60376,60542) 10.12.193.70:6810/2346 192.168.91.70:6839/1002346 192.168.91.70:6840/1002346 10.12.193.70:6809/1002346 exists,up 5f1b86a2-d0c3-493a-8bcf-dcbf4a054f47
osd.52 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59851,60535) 10.12.193.69:6820/22132 192.168.91.69:6830/1022132 192.168.91.69:6831/1022132 10.12.193.69:6827/1022132 exists,up c23ed2fb-4080-4bff-ae70-2cd43bc59544
osd.53 up in weight 1 up_from 60559 up_thru 60607 down_at 60556 last_clean_interval [59442,60558) 10.12.193.67:6808/16500 192.168.91.67:6814/1016500 192.168.91.67:6815/1016500 10.12.193.67:6815/1016500 exists,up d4b3f0af-949e-4fcb-a7a1-32d96afc3305
osd.54 up in weight 1 up_from 60559 up_thru 60607 down_at 60556 last_clean_interval [59469,60558) 10.12.193.71:6813/2253 192.168.91.71:6800/1002253 192.168.91.71:6801/1002253 10.12.193.71:6805/1002253 exists,up ea4077dd-f6d6-4843-bdf3-9cb2b6b13bee
osd.55 up in weight 1 up_from 60558 up_thru 60607 down_at 60556 last_clean_interval [59450,60557) 10.12.193.68:6826/1773 192.168.91.68:6807/1001773 192.168.91.68:6809/1001773 10.12.193.68:6808/1001773 exists,up 042c79f0-4a83-410c-a501-f522f62bf5dc
osd.56 up in weight 1 up_from 60542 up_thru 60593 down_at 60534 last_clean_interval [59476,60541) 10.12.193.72:6815/5524 192.168.91.72:6834/1005524 192.168.91.72:6835/1005524 10.12.193.72:6829/1005524 exists,up 01a00dce-0bd0-4068-bf5b-b93521a68208
osd.57 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [60369,60535) 10.12.193.70:6814/2509 192.168.91.70:6825/1002509 192.168.91.70:6826/1002509 10.12.193.70:6824/1002509 exists,up 05fbdc7d-002e-4f4d-b068-77253dfa4595
osd.58 up in weight 1 up_from 60559 up_thru 60607 down_at 60556 last_clean_interval [59850,60558) 10.12.193.69:6808/21451 192.168.91.69:6802/1021451 192.168.91.69:6805/1021451 10.12.193.69:6803/1021451 exists,up b2ed6957-9824-4f7c-8a25-bc8c1a64bfd0
osd.59 up in weight 1 up_from 60558 up_thru 60593 down_at 60556 last_clean_interval [59443,60557) 10.12.193.67:6816/16986 192.168.91.67:6806/1016986 192.168.91.67:6807/1016986 10.12.193.67:6807/1016986 exists,up 9579da6f-8b46-4465-b6b5-e3676f4dc3dc
osd.60 up in weight 1 up_from 60562 up_thru 60607 down_at 60556 last_clean_interval [59469,60561) 10.12.193.71:6815/2382 192.168.91.71:6804/1002382 192.168.91.71:6810/1002382 10.12.193.71:6819/1002382 exists,up e0f0bb3a-5dda-47d4-a6a9-21418cc24c17
osd.61 up in weight 1 up_from 60535 up_thru 60607 down_at 60534 last_clean_interval [59448,60534) 10.12.193.68:6819/1501 192.168.91.68:6800/1001501 192.168.91.68:6806/1001501 10.12.193.68:6804/1001501 exists,up dd60e730-9bab-42ce-a5d6-9db6d25006ac
osd.62 up in weight 1 up_from 60536 up_thru 60593 down_at 60534 last_clean_interval [59476,60535) 10.12.193.72:6824/6054 192.168.91.72:6800/1006054 192.168.91.72:6804/1006054 10.12.193.72:6800/1006054 exists,up 910f92c4-45bb-4d7f-984d-9b1f8109a963
osd.63 up in weight 1 up_from 60543 up_thru 60607 down_at 60534 last_clean_interval [60311,60542) 10.12.193.70:6800/965 192.168.91.70:6837/1000965 192.168.91.70:6838/1000965 10.12.193.70:6832/1000965 exists,up 4af592fc-029a-4afe-be31-c2d6b728ddbb
osd.64 up in weight 1 up_from 60535 up_thru 60607 down_at 60534 last_clean_interval [59769,60534) 10.12.193.69:6800/18864 192.168.91.69:6824/1018864 192.168.91.69:6825/1018864 10.12.193.69:6824/1018864 exists,up 37451657-c9f7-4d05-a47f-c6ba35f47b26
osd.65 up in weight 1 up_from 60560 up_thru 60607 down_at 60556 last_clean_interval [59442,60559) 10.12.193.67:6804/16207 192.168.91.67:6812/1016207 192.168.91.67:6813/1016207 10.12.193.67:6813/1016207 exists,up 32aa2e4f-8a00-4320-bf2f-6c107b36adee
osd.66 up in weight 1 up_from 60539 up_thru 60607 down_at 60534 last_clean_interval [59469,60538) 10.12.193.71:6824/2844 192.168.91.71:6834/1002844 192.168.91.71:6835/1002844 10.12.193.71:6811/1002844 exists,up 55771d00-0ca9-4453-a3af-e1e72520ef87
osd.67 up in weight 1 up_from 60537 up_thru 60607 down_at 60534 last_clean_interval [59449,60536) 10.12.193.68:6813/1252 192.168.91.68:6832/1001252 192.168.91.68:6833/1001252 10.12.193.68:6818/1001252 exists,up b1b29cab-7e44-4651-b142-dd542f8855f4
osd.68 up in weight 1 up_from 60558 up_thru 60589 down_at 60556 last_clean_interval [59477,60556) 10.12.193.72:6821/5915 192.168.91.72:6801/1005915 192.168.91.72:6802/1005915 10.12.193.72:6802/1005915 exists,up 3d5924a7-1c74-48c1-9fae-62825a66df52
osd.69 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59851,60535) 10.12.193.69:6818/22106 192.168.91.69:6828/1022106 192.168.91.69:6829/1022106 10.12.193.69:6826/1022106 exists,up a7ecc3a7-ecaa-428e-8317-14db07825f9e
osd.70 up in weight 1 up_from 60542 up_thru 60607 down_at 60534 last_clean_interval [60371,60541) 10.12.193.70:6818/2686 192.168.91.70:6800/1002686 192.168.91.70:6806/1002686 10.12.193.70:6831/1002686 exists,up aadd793b-7a53-49ea-948a-0a6f20481f0c
osd.71 up in weight 1 up_from 60536 up_thru 60607 down_at 60534 last_clean_interval [59443,60535) 10.12.193.67:6822/17348 192.168.91.67:6824/1017348 192.168.91.67:6825/1017348 10.12.193.67:6824/1017348 exists,up 0b37aeb9-a92f-449c-8e3c-99fbcf5bd8b8
$ sudo ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 131.03943 root default
-2 21.83990 host tv-c2-al02
0 1.81999 osd.0 up 1.00000 1.00000
6 1.81999 osd.6 up 1.00000 1.00000
12 1.81999 osd.12 up 1.00000 1.00000
18 1.81999 osd.18 up 1.00000 1.00000
24 1.81999 osd.24 up 1.00000 1.00000
30 1.81999 osd.30 up 1.00000 1.00000
36 1.81999 osd.36 up 1.00000 1.00000
42 1.81999 osd.42 up 1.00000 1.00000
48 1.81999 osd.48 up 1.00000 1.00000
54 1.81999 osd.54 up 1.00000 1.00000
60 1.81999 osd.60 up 1.00000 1.00000
66 1.81999 osd.66 up 1.00000 1.00000
-3 21.83990 host tv-c1-al02
1 1.81999 osd.1 up 1.00000 1.00000
7 1.81999 osd.7 up 1.00000 1.00000
13 1.81999 osd.13 up 1.00000 1.00000
19 1.81999 osd.19 up 1.00000 1.00000
25 1.81999 osd.25 up 1.00000 1.00000
31 1.81999 osd.31 up 1.00000 1.00000
37 1.81999 osd.37 up 1.00000 1.00000
43 1.81999 osd.43 up 1.00000 1.00000
49 1.81999 osd.49 up 1.00000 1.00000
55 1.81999 osd.55 up 1.00000 1.00000
61 1.81999 osd.61 up 1.00000 1.00000
67 1.81999 osd.67 up 1.00000 1.00000
-4 21.83990 host tv-c2-al03
2 1.81999 osd.2 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000
14 1.81999 osd.14 up 1.00000 1.00000
20 1.81999 osd.20 up 1.00000 1.00000
26 1.81999 osd.26 up 1.00000 1.00000
32 1.81999 osd.32 up 1.00000 1.00000
38 1.81999 osd.38 up 1.00000 1.00000
44 1.81999 osd.44 up 1.00000 1.00000
50 1.81999 osd.50 up 1.00000 1.00000
56 1.81999 osd.56 up 1.00000 1.00000
62 1.81999 osd.62 up 1.00000 1.00000
68 1.81999 osd.68 up 1.00000 1.00000
-5 21.83990 host tv-c2-al01
3 1.81999 osd.3 up 1.00000 1.00000
9 1.81999 osd.9 up 1.00000 1.00000
15 1.81999 osd.15 up 1.00000 1.00000
21 1.81999 osd.21 up 1.00000 1.00000
27 1.81999 osd.27 up 1.00000 1.00000
33 1.81999 osd.33 up 1.00000 1.00000
39 1.81999 osd.39 up 1.00000 1.00000
45 1.81999 osd.45 up 1.00000 1.00000
51 1.81999 osd.51 up 1.00000 1.00000
57 1.81999 osd.57 up 1.00000 1.00000
63 1.81999 osd.63 up 1.00000 1.00000
70 1.81999 osd.70 up 1.00000 1.00000
-6 21.83990 host tv-c1-al03
4 1.81999 osd.4 up 1.00000 1.00000
10 1.81999 osd.10 up 1.00000 1.00000
16 1.81999 osd.16 up 1.00000 1.00000
22 1.81999 osd.22 up 1.00000 1.00000
28 1.81999 osd.28 up 1.00000 1.00000
34 1.81999 osd.34 up 1.00000 1.00000
40 1.81999 osd.40 up 1.00000 1.00000
46 1.81999 osd.46 up 1.00000 1.00000
52 1.81999 osd.52 up 1.00000 1.00000
58 1.81999 osd.58 up 1.00000 1.00000
64 1.81999 osd.64 up 1.00000 1.00000
69 1.81999 osd.69 up 1.00000 1.00000
-7 21.83990 host tv-c1-al01
5 1.81999 osd.5 up 1.00000 1.00000
11 1.81999 osd.11 up 1.00000 1.00000
17 1.81999 osd.17 up 1.00000 1.00000
23 1.81999 osd.23 up 1.00000 1.00000
29 1.81999 osd.29 up 1.00000 1.00000
35 1.81999 osd.35 up 1.00000 1.00000
41 1.81999 osd.41 up 1.00000 1.00000
47 1.81999 osd.47 up 1.00000 1.00000
53 1.81999 osd.53 up 1.00000 1.00000
59 1.81999 osd.59 up 1.00000 1.00000
65 1.81999 osd.65 up 1.00000 1.00000
71 1.81999 osd.71 up 1.00000 1.00000
$
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
