Hi !
We have upgraded our Ceph cluster (3 Mon Servers, 9 OSD Servers, 190
OSDs total) From 10.2.10 to Ceph 12.2.4 and then to 12.2.5.
(A mixture of Ubuntu 14 and 16 with the Repos from
https://download.ceph.com/debian-luminous/)
Now we have the Problem that One ODS is crashing again and again
(approx. once per day). systemd restarts it.
We could now propably identify the problem. It looks like one placement
group (5.9b) causes the crash.
It seems like it doesnt matter if it is running on a filestore or a
bluestore osd.
We could even break it down to some RBDs that were in this pool.
They are already deleted, but it looks like there are some objects on
the osd left, but we cant delete them :
rados -p rbd ls > radosrbdls.txt
echo radosrbdls.txt | grep -vE "($(rados -p rbd ls | grep rbd_header |
grep -o "\.[0-9a-f]*" | sed -e :a -e '$!N; s/\n/|/; ta' -e
's/\./\\./g'))" | grep -E '(rbd_data|journal|rbd_object_map)'
rbd_data.112913b238e1f29.0000000000000e3f
rbd_data.112913b238e1f29.00000000000009d2
rbd_data.112913b238e1f29.0000000000000ba3
rados -p rbd rm rbd_data.112913b238e1f29.0000000000000e3f
error removing rbd>rbd_data.112913b238e1f29.0000000000000e3f: (2) No
such file or directory
rados -p rbd rm rbd_data.112913b238e1f29.00000000000009d2
error removing rbd>rbd_data.112913b238e1f29.00000000000009d2: (2) No
such file or directory
rados -p rbd rm rbd_data.112913b238e1f29.0000000000000ba3
error removing rbd>rbd_data.112913b238e1f29.0000000000000ba3: (2) No
such file or directory
In the "current" directory of the osd there are a lot more files with
this rbd prefix.
Is there any chance to delete these obviously orpahed stuff before the
pg becomes healthy ?
(it is running now at only 2 of 3 osds)
What else could cause such a crash ?
We attatch (hopefully all) of the relevant logs.
-103> 2018-05-14 13:01:50.514850 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.49:0/2866 conn(0x55a13fd0d000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=453 cs=1 l=1). rx osd.60 seq 2720 0x55a13e7bac00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.511610) v4
-102> 2018-05-14 13:01:50.514878 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.60 10.7.2.49:0/2866 2720 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.511610) v4 ==== 2004+0+0 (1134770966 0 0) 0x55a13e7bac00 con 0x55a13fd0d000
-101> 2018-05-14 13:01:50.514896 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.49:0/2866 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.511610) v4 -- 0x55a13fd27200 con 0
-100> 2018-05-14 13:01:50.525876 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.144:0/2988 conn(0x55a13f2dd000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=865 cs=1 l=1). rx osd.179 seq 2652 0x55a13e442600 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.531899) v4
-99> 2018-05-14 13:01:50.525902 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.179 10.7.2.144:0/2988 2652 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.531899) v4 ==== 2004+0+0 (3454691771 0 0) 0x55a13e442600 con 0x55a13f2dd000
-98> 2018-05-14 13:01:50.525917 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.144:0/2988 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.531899) v4 -- 0x55a13fd27200 con 0
-97> 2018-05-14 13:01:50.526649 7f389914d700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.24:0/2988 conn(0x55a13f2de800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=869 cs=1 l=1). rx osd.179 seq 2652 0x55a17bd8a200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.531899) v4
-96> 2018-05-14 13:01:50.526675 7f389914d700 1 -- 10.0.0.28:6801/139719 <== osd.179 10.0.0.24:0/2988 2652 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.531899) v4 ==== 2004+0+0 (3454691771 0 0) 0x55a17bd8a200 con 0x55a13f2de800
-95> 2018-05-14 13:01:50.526688 7f389914d700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.24:0/2988 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.531899) v4 -- 0x55a13e43ec00 con 0
-94> 2018-05-14 13:01:50.546508 7f389994e700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.50:6802/2519 conn(0x55a13e724000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18716 cs=1 l=0). rx osd.47 seq 4894 0x55a13ec9d000 MOSDScrubReserve(3.111 REQUEST e502962) v1
-93> 2018-05-14 13:01:50.546537 7f389994e700 1 -- 10.7.2.141:6800/139719 <== osd.47 10.7.2.50:6802/2519 4894 ==== MOSDScrubReserve(3.111 REQUEST e502962) v1 ==== 43+0+0 (327031511 0 0) 0x55a13ec9d000 con 0x55a13e724000
-92> 2018-05-14 13:01:50.546655 7f3883138700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.50:6802/2519 -- MOSDScrubReserve(3.111 REJECT e502962) v1 -- 0x55a13e8fd200 con 0
-91> 2018-05-14 13:01:50.547685 7f389994e700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.50:6802/2519 conn(0x55a13e724000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18716 cs=1 l=0). rx osd.47 seq 4895 0x55a13e8fd200 MOSDScrubReserve(3.111 RELEASE e502962) v1
-90> 2018-05-14 13:01:50.547714 7f389994e700 1 -- 10.7.2.141:6800/139719 <== osd.47 10.7.2.50:6802/2519 4895 ==== MOSDScrubReserve(3.111 RELEASE e502962) v1 ==== 43+0+0 (3697533141 0 0) 0x55a13e8fd200 con 0x55a13e724000
-89> 2018-05-14 13:01:50.548003 7f389914d700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.142:0/4345 conn(0x55a11fe57800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=457 cs=1 l=1). rx osd.140 seq 2757 0x55a13ec7ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.544179) v4
-88> 2018-05-14 13:01:50.548049 7f389914d700 1 -- 10.7.2.141:6801/139719 <== osd.140 10.7.2.142:0/4345 2757 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.544179) v4 ==== 2004+0+0 (469525880 0 0) 0x55a13ec7ec00 con 0x55a11fe57800
-87> 2018-05-14 13:01:50.548068 7f389994e700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.29:0/4345 conn(0x55a11fe59000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=457 cs=1 l=1). rx osd.140 seq 2757 0x55a15e34b200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.544179) v4
-86> 2018-05-14 13:01:50.548087 7f389914d700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.142:0/4345 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.544179) v4 -- 0x55a13e43ec00 con 0
-85> 2018-05-14 13:01:50.548089 7f389994e700 1 -- 10.0.0.28:6801/139719 <== osd.140 10.0.0.29:0/4345 2757 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.544179) v4 ==== 2004+0+0 (469525880 0 0) 0x55a15e34b200 con 0x55a11fe59000
-84> 2018-05-14 13:01:50.548172 7f389994e700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.29:0/4345 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.544179) v4 -- 0x55a13ec1a800 con 0
-83> 2018-05-14 13:01:50.551423 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130308 0x55a15d9c0d80 MOSDPGPush(21.4a4 502962/502817 [PushOp(21:253ffff8:::default.19540791.4_videos%2fpics%2f53848640d0d399b961016648%2f7.jpg:head, version: 385405'281502, data_included: [0~106010], data_size: 106010, omap_header_size: 0, omap_entries_size: 0, attrset_size: 13, recovery_info: ObjectRecoveryInfo(21:253ffff8:::default.19540791.4_videos%2fpics%2f53848640d0d399b961016648%2f7.jpg:head@385405'281502, size: 106010, copy_subset: [0~106010], clone_subset: {}, snapset: 0=[]:[]), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106010, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))]) v3
-82> 2018-05-14 13:01:50.551469 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130308 ==== MOSDPGPush(21.4a4 502962/502817 [PushOp(21:253ffff8:::default.19540791.4_videos%2fpics%2f53848640d0d399b961016648%2f7.jpg:head, version: 385405'281502, data_included: [0~106010], data_size: 106010, omap_header_size: 0, omap_entries_size: 0, attrset_size: 13, recovery_info: ObjectRecoveryInfo(21:253ffff8:::default.19540791.4_videos%2fpics%2f53848640d0d399b961016648%2f7.jpg:head@385405'281502, size: 106010, copy_subset: [0~106010], clone_subset: {}, snapset: 0=[]:[]), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106010, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))]) v3 ==== 108276+0+0 (1366903846 0 0) 0x55a15d9c0d80 con 0x55a13e942800
-81> 2018-05-14 13:01:50.551544 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130309 0x55a176cfcd00 pg_backfill(progress 21.4a4 e 502962/502962 lb 21:253fffdd:::default.20758044.3_logs%2f2017-10-18-04-51-55-0C184FE5F481DDFF:head) v3
-80> 2018-05-14 13:01:50.551571 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130309 ==== pg_backfill(progress 21.4a4 e 502962/502962 lb 21:253fffdd:::default.20758044.3_logs%2f2017-10-18-04-51-55-0C184FE5F481DDFF:head) v3 ==== 978+0+0 (285648651 0 0) 0x55a176cfcd00 con 0x55a13e942800
-79> 2018-05-14 13:01:50.552853 7f388b949700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.144:6808/62146 -- pg_trim(21.4a4 to 502845'320604 e502962) v2 -- 0x55a140516d80 con 0
-78> 2018-05-14 13:01:50.552927 7f388b949700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.144:6808/62146 -- MOSDPGPushReply(21.4a4 502962/502817 [PushReplyOp(21:253ffff8:::default.19540791.4_videos%2fpics%2f53848640d0d399b961016648%2f7.jpg:head)]) v3 -- 0x55a164c86480 con 0
-77> 2018-05-14 13:01:50.624973 7f389994e700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.15:0/3266 conn(0x55a1407b2800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=875 cs=1 l=1). rx osd.48 seq 2772 0x55a17bd8ce00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.621641) v4
-76> 2018-05-14 13:01:50.625022 7f389994e700 1 -- 10.0.0.28:6801/139719 <== osd.48 10.0.0.15:0/3266 2772 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.621641) v4 ==== 2004+0+0 (1695103482 0 0) 0x55a17bd8ce00 con 0x55a1407b2800
-75> 2018-05-14 13:01:50.625053 7f389994e700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.15:0/3266 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.621641) v4 -- 0x55a13ec1a800 con 0
-74> 2018-05-14 13:01:50.624973 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.49:0/3266 conn(0x55a14093e000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=874 cs=1 l=1). rx osd.48 seq 2772 0x55a13e7bac00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.621641) v4
-73> 2018-05-14 13:01:50.625149 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.48 10.7.2.49:0/3266 2772 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.621641) v4 ==== 2004+0+0 (1695103482 0 0) 0x55a13e7bac00 con 0x55a14093e000
-72> 2018-05-14 13:01:50.625173 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.49:0/3266 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.621641) v4 -- 0x55a13fd27200 con 0
-71> 2018-05-14 13:01:50.633021 7f389914d700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.24:0/4019 conn(0x55a13f0b1000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=404 cs=1 l=1). rx osd.184 seq 2768 0x55a17bd8a200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.638813) v4
-70> 2018-05-14 13:01:50.633047 7f389914d700 1 -- 10.0.0.28:6801/139719 <== osd.184 10.0.0.24:0/4019 2768 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.638813) v4 ==== 2004+0+0 (1220405549 0 0) 0x55a17bd8a200 con 0x55a13f0b1000
-69> 2018-05-14 13:01:50.633079 7f389914d700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.24:0/4019 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.638813) v4 -- 0x55a13e43ec00 con 0
-68> 2018-05-14 13:01:50.633253 7f389994e700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.144:0/4019 conn(0x55a13f0b2800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=404 cs=1 l=1). rx osd.184 seq 2768 0x55a15e34b200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.638813) v4
-67> 2018-05-14 13:01:50.633277 7f389994e700 1 -- 10.7.2.141:6801/139719 <== osd.184 10.7.2.144:0/4019 2768 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.638813) v4 ==== 2004+0+0 (1220405549 0 0) 0x55a15e34b200 con 0x55a13f0b2800
-66> 2018-05-14 13:01:50.633296 7f389994e700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.144:0/4019 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.638813) v4 -- 0x55a13ec1a800 con 0
-65> 2018-05-14 13:01:50.634153 7f389894c700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.170:0/2476 conn(0x55a13ff82000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=818 cs=1 l=1). rx osd.53 seq 2756 0x55a13e442600 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631916) v4
-64> 2018-05-14 13:01:50.634187 7f389894c700 1 -- 10.0.0.28:6801/139719 <== osd.53 10.0.0.170:0/2476 2756 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631916) v4 ==== 2004+0+0 (1547211059 0 0) 0x55a13e442600 con 0x55a13ff82000
-63> 2018-05-14 13:01:50.634228 7f389894c700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.170:0/2476 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.631916) v4 -- 0x55a13fd27200 con 0
-62> 2018-05-14 13:01:50.634336 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.140:0/2476 conn(0x55a1405a1800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=818 cs=1 l=1). rx osd.53 seq 2756 0x55a13e7bac00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631916) v4
-61> 2018-05-14 13:01:50.634362 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.53 10.7.2.140:0/2476 2756 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631916) v4 ==== 2004+0+0 (1547211059 0 0) 0x55a13e7bac00 con 0x55a1405a1800
-60> 2018-05-14 13:01:50.634383 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.140:0/2476 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.631916) v4 -- 0x55a13fd27200 con 0
-59> 2018-05-14 13:01:50.634678 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.142:0/3802 conn(0x55a13fdc0800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=861 cs=1 l=1). rx osd.153 seq 2703 0x55a13e442600 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631623) v4
-58> 2018-05-14 13:01:50.634707 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.153 10.7.2.142:0/3802 2703 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631623) v4 ==== 2004+0+0 (2902405628 0 0) 0x55a13e442600 con 0x55a13fdc0800
-57> 2018-05-14 13:01:50.634731 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.142:0/3802 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.631623) v4 -- 0x55a13fd27200 con 0
-56> 2018-05-14 13:01:50.635437 7f389994e700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.29:0/3802 conn(0x55a13fdbf000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=861 cs=1 l=1). rx osd.153 seq 2703 0x55a17bd8ce00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631623) v4
-55> 2018-05-14 13:01:50.635463 7f389994e700 1 -- 10.0.0.28:6801/139719 <== osd.153 10.0.0.29:0/3802 2703 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.631623) v4 ==== 2004+0+0 (2902405628 0 0) 0x55a17bd8ce00 con 0x55a13fdbf000
-54> 2018-05-14 13:01:50.635478 7f389994e700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.29:0/3802 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.631623) v4 -- 0x55a13ec1a800 con 0
-53> 2018-05-14 13:01:50.653750 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130310 0x55a149eccd00 pg_backfill(finish 21.4a4 e 502962/502962 lb MAX) v3
-52> 2018-05-14 13:01:50.653782 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130310 ==== pg_backfill(finish 21.4a4 e 502962/502962 lb MAX) v3 ==== 918+0+0 (3026729080 0 0) 0x55a149eccd00 con 0x55a13e942800
-51> 2018-05-14 13:01:50.653840 7f3883138700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.144:6808/62146 -- pg_backfill(finish_ack 21.4a4 e 502962/502962 lb MIN) v3 -- 0x55a176cfcd00 con 0
-50> 2018-05-14 13:01:50.653987 7f388513c700 5 osd.130 pg_epoch: 502962 pg[21.4a4( v 502845'320604 (448209'318935,502845'320604] local-lis/les=499194/499196 n=65064 ec=95070/95070 lis/c 499194/502813 les/c/f 499196/502814/0 502817/502817/500353) [189,24,130]/[189,24,59] r=-1 lpr=502817 pi=[494803,502817)/2 luod=0'0 crt=502845'320604 active+remapped] exit Started/ReplicaActive/RepRecovering 7676.644217 418 0.001469
-49> 2018-05-14 13:01:50.654050 7f388513c700 5 osd.130 pg_epoch: 502962 pg[21.4a4( v 502845'320604 (448209'318935,502845'320604] local-lis/les=499194/499196 n=65064 ec=95070/95070 lis/c 499194/502813 les/c/f 499196/502814/0 502817/502817/500353) [189,24,130]/[189,24,59] r=-1 lpr=502817 pi=[494803,502817)/2 luod=0'0 crt=502845'320604 active+remapped] enter Started/ReplicaActive/RepNotRecovering
-48> 2018-05-14 13:01:50.654277 7f388593d700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.49:6836/3920 -- MBackfillReserve GRANT pgid: 5.9b, query_epoch: 502962 v3 -- 0x55a11fd90800 con 0
-47> 2018-05-14 13:01:50.654309 7f388593d700 5 osd.130 pg_epoch: 502962 pg[5.9b( v 502962'20360820 (502724'20350805,502962'20360820] lb 5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56c06 (bitwise) local-lis/les=498152/498153 n=9786 ec=874/874 lis/c 498152/459426 les/c/f 498153/459427/0 502817/502821/493821) [19,166,130]/[19,166] r=-1 lpr=502821 pi=[459426,502821)/10 luod=0'0 crt=502962'20360820 lcod 502962'20360819 active+remapped] exit Started/ReplicaActive/RepWaitBackfillReserved 7676.643632 418 0.001391
-46> 2018-05-14 13:01:50.654345 7f388593d700 5 osd.130 pg_epoch: 502962 pg[5.9b( v 502962'20360820 (502724'20350805,502962'20360820] lb 5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56c06 (bitwise) local-lis/les=498152/498153 n=9786 ec=874/874 lis/c 498152/459426 les/c/f 498153/459427/0 502817/502821/493821) [19,166,130]/[19,166] r=-1 lpr=502821 pi=[459426,502821)/10 luod=0'0 crt=502962'20360820 lcod 502962'20360819 active+remapped] enter Started/ReplicaActive/RepRecovering
-45> 2018-05-14 13:01:50.665766 7f389894c700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.26:0/3432 conn(0x55a1400fe000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=889 cs=1 l=1). rx osd.95 seq 2713 0x55a13e7bac00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.661313) v4
-44> 2018-05-14 13:01:50.665792 7f389894c700 1 -- 10.0.0.28:6801/139719 <== osd.95 10.0.0.26:0/3432 2713 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.661313) v4 ==== 2004+0+0 (973149088 0 0) 0x55a13e7bac00 con 0x55a1400fe000
-43> 2018-05-14 13:01:50.665821 7f389894c700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.26:0/3432 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.661313) v4 -- 0x55a13fd27200 con 0
-42> 2018-05-14 13:01:50.666433 7f389994e700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.51:0/3432 conn(0x55a1400ec800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=889 cs=1 l=1). rx osd.95 seq 2713 0x55a15e34b200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.661313) v4
-41> 2018-05-14 13:01:50.666458 7f389994e700 1 -- 10.7.2.141:6801/139719 <== osd.95 10.7.2.51:0/3432 2713 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.661313) v4 ==== 2004+0+0 (973149088 0 0) 0x55a15e34b200 con 0x55a1400ec800
-40> 2018-05-14 13:01:50.666483 7f389994e700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.51:0/3432 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.661313) v4 -- 0x55a13ec1a800 con 0
-39> 2018-05-14 13:01:50.692423 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.142:6814/3713 conn(0x55a13e99f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19483 cs=3 l=0). rx osd.147 seq 3758 0x55a11fd90800 MOSDScrubReserve(10.1a REQUEST e502962) v1
-38> 2018-05-14 13:01:50.692453 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.147 10.7.2.142:6814/3713 3758 ==== MOSDScrubReserve(10.1a REQUEST e502962) v1 ==== 43+0+0 (3677106971 0 0) 0x55a11fd90800 con 0x55a13e99f800
-37> 2018-05-14 13:01:50.692568 7f388413a700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.142:6814/3713 -- MOSDScrubReserve(10.1a REJECT e502962) v1 -- 0x55a13e466600 con 0
-36> 2018-05-14 13:01:50.693195 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.142:6814/3713 conn(0x55a13e99f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19483 cs=3 l=0). rx osd.147 seq 3759 0x55a13e466600 MOSDScrubReserve(10.1a RELEASE e502962) v1
-35> 2018-05-14 13:01:50.693218 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.147 10.7.2.142:6814/3713 3759 ==== MOSDScrubReserve(10.1a RELEASE e502962) v1 ==== 43+0+0 (338807065 0 0) 0x55a13e466600 con 0x55a13e99f800
-34> 2018-05-14 13:01:50.727964 7f389914d700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.51:0/3076 conn(0x55a13f040800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=419 cs=1 l=1). rx osd.87 seq 2729 0x55a13ec7ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.724488) v4
-33> 2018-05-14 13:01:50.727999 7f389914d700 1 -- 10.7.2.141:6801/139719 <== osd.87 10.7.2.51:0/3076 2729 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.724488) v4 ==== 2004+0+0 (1301168773 0 0) 0x55a13ec7ec00 con 0x55a13f040800
-32> 2018-05-14 13:01:50.728018 7f389914d700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.51:0/3076 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.724488) v4 -- 0x55a13e43ec00 con 0
-31> 2018-05-14 13:01:50.728388 7f389994e700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.26:0/3076 conn(0x55a13f068000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=419 cs=1 l=1). rx osd.87 seq 2729 0x55a17bd8ce00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.724488) v4
-30> 2018-05-14 13:01:50.728419 7f389994e700 1 -- 10.0.0.28:6801/139719 <== osd.87 10.0.0.26:0/3076 2729 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.724488) v4 ==== 2004+0+0 (1301168773 0 0) 0x55a17bd8ce00 con 0x55a13f068000
-29> 2018-05-14 13:01:50.728439 7f389994e700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.26:0/3076 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.724488) v4 -- 0x55a13ec1a800 con 0
-28> 2018-05-14 13:01:50.758526 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130311 0x55a164c86480 pg_trim(21.4a4 to 454747'319104 e502962) v2
-27> 2018-05-14 13:01:50.758628 7f388f1c9700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130311 ==== pg_trim(21.4a4 to 454747'319104 e502962) v2 ==== 34+0+0 (1705713896 0 0) 0x55a164c86480 con 0x55a13e942800
-26> 2018-05-14 13:01:50.759553 7f388f1c9700 5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: 448209'318936,448209'318937,448209'318938,448209'318939,448209'318940,448209'318941,448209'318942,448209'318943,448209'318944,448209'318945,448209'318946,448209'318947,448209'318948,448209'318949,448209'318950,448209'318951,448209'318952,448209'318953,448209'318954,448209'318955,448209'318956,448209'318957,448209'318958,448209'318959,448209'318960,448209'318961,448209'318962,448209'318963,448209'318964,448209'318965,448209'318966,448209'318967,448209'318968,448209'318969,448209'318970,448209'318971,448238'318972,448340'318973,448488'318974,448488'318975,448488'318976,448490'318977,448490'318978,448625'318979,448625'318980,448625'318981,448625'318982,448640'318983,448652'318984,448658'318985,448660'318986,448660'318987,448764'318988,448764'318989,448899'318990,449033'318991,449033'318992,449033'318993,449033'318994,449061'318995,449064'318996,449066'318997,449169'318998,449169'318999,449314'319000,449314'319001,449314'319002,449317'319003,449317'319004,449317'319005,449332'319006,449355'319007,449456'319008,449456'319009,449456'319010,449456'319011,449456'319012,449679'319013,449781'319014,449781'319015,449784'319016,449784'319017,449917'319018,449920'319019,450060'319020,450060'319021,450060'319022,450060'319023,450074'319024,450091'319025,450196'319026,450196'319027,450212'319028,450212'319029,450353'319030,450353'319031,450355'319032,450367'319033,450494'319034,450494'319035,450494'319036,450494'319037,450494'319038,450494'319039,450494'319040,450494'319041,450494'319042,450494'319043,450494'319044,450494'319045,450494'319046,450494'319047,450494'319048,450523'319049,450529'319050,450633'319051,450636'319052,450791'319053,450935'319054,450988'319055,451088'319056,451088'319057,451088'319058,451088'319059,451088'319060,451088'319061,451088'319062,451088'319063,451088'319064,451088'319065,451105'319066,451244'319067,451244'319068,451686'319069,451686'319070,451829'319071,451841'319072,451854'319073,451962'319074,452102'319075,452102'319076,452102'319077,452116'319078,452245'319079,452304'319080,452563'319081,453060'319082,453424'319083,453424'319084,453433'319085,453433'319086,453439'319087,453439'319088,453683'319089,453683'319090,453685'319091,453685'319092,453687'319093,453961'319094,453985'319095,454168'319096,454170'319097,454363'319098,454471'319099,454475'319100,454475'319101,454747'319102,454747'319103,454747'319104, trimmed_dups: , clear_divergent_priors: 0
-25> 2018-05-14 13:01:50.759685 7f389914d700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.143:0/3927 conn(0x55a1409b1000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=784 cs=1 l=1). rx osd.54 seq 2752 0x55a13ec7ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.764622) v4
-24> 2018-05-14 13:01:50.759709 7f389914d700 1 -- 10.7.2.141:6801/139719 <== osd.54 10.7.2.143:0/3927 2752 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.764622) v4 ==== 2004+0+0 (1348842237 0 0) 0x55a13ec7ec00 con 0x55a1409b1000
-23> 2018-05-14 13:01:50.759733 7f389914d700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.143:0/3927 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.764622) v4 -- 0x55a13e43ec00 con 0
-22> 2018-05-14 13:01:50.759845 7f389894c700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.25:0/3927 conn(0x55a1409af800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=786 cs=1 l=1). rx osd.54 seq 2752 0x55a13e442600 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.764622) v4
-21> 2018-05-14 13:01:50.759869 7f389894c700 1 -- 10.0.0.28:6801/139719 <== osd.54 10.0.0.25:0/3927 2752 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.764622) v4 ==== 2004+0+0 (1348842237 0 0) 0x55a13e442600 con 0x55a1409af800
-20> 2018-05-14 13:01:50.759886 7f389894c700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.25:0/3927 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.764622) v4 -- 0x55a13fd27200 con 0
-19> 2018-05-14 13:01:50.763522 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130312 0x55a17bd8a200 pg_info((query:502962 sent:502962 21.4a4( v 502845'320604 (454747'319104,502845'320604] local-lis/les=502817/502819 n=65064 ec=95070/95070 lis/c 502817/502817 les/c/f 502819/502962/0 502817/502817/500353))=(empty) epoch 502962) v5
-18> 2018-05-14 13:01:50.763566 7f388f1c9700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130312 ==== pg_info((query:502962 sent:502962 21.4a4( v 502845'320604 (454747'319104,502845'320604] local-lis/les=502817/502819 n=65064 ec=95070/95070 lis/c 502817/502817 les/c/f 502819/502962/0 502817/502817/500353))=(empty) epoch 502962) v5 ==== 922+0+0 (813456022 0 0) 0x55a17bd8a200 con 0x55a13e942800
-17> 2018-05-14 13:01:50.790100 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.49:6836/3920 conn(0x55a13e727000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19104 cs=1 l=0). rx osd.19 seq 5704 0x55a145c3a700 pg_scan(get_digest 5.9b 5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56c06-MIN e 502962/502962) v2
-16> 2018-05-14 13:01:50.790130 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.19 10.7.2.49:6836/3920 5704 ==== pg_scan(get_digest 5.9b 5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56c06-MIN e 502962/502962) v2 ==== 160+0+0 (1020111130 0 0) 0x55a145c3a700 con 0x55a13e727000
-15> 2018-05-14 13:01:50.802285 7f388493b700 1 -- 10.7.2.141:6800/139719 --> 10.7.2.49:6836/3920 -- pg_scan(digest 5.9b 5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56c06-MAX e 502962/502962) v2 -- 0x55a1445fba00 con 0
-14> 2018-05-14 13:01:50.869039 7f389914d700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.24:0/3767 conn(0x55a13f25d000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=818 cs=1 l=1). rx osd.159 seq 2702 0x55a13ec7ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.874380) v4
-13> 2018-05-14 13:01:50.869084 7f389994e700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.144:0/3767 conn(0x55a13f25e800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=816 cs=1 l=1). rx osd.159 seq 2702 0x55a15e34b200 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.874380) v4
-12> 2018-05-14 13:01:50.869098 7f389914d700 1 -- 10.0.0.28:6801/139719 <== osd.159 10.0.0.24:0/3767 2702 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.874380) v4 ==== 2004+0+0 (2085408752 0 0) 0x55a13ec7ec00 con 0x55a13f25d000
-11> 2018-05-14 13:01:50.869119 7f389994e700 1 -- 10.7.2.141:6801/139719 <== osd.159 10.7.2.144:0/3767 2702 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.874380) v4 ==== 2004+0+0 (2085408752 0 0) 0x55a15e34b200 con 0x55a13f25e800
-10> 2018-05-14 13:01:50.869169 7f389914d700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.24:0/3767 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.874380) v4 -- 0x55a13e401000 con 0
-9> 2018-05-14 13:01:50.869236 7f389994e700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.144:0/3767 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.874380) v4 -- 0x55a13ec1a800 con 0
-8> 2018-05-14 13:01:50.878179 7f389894c700 5 -- 10.0.0.28:6801/139719 >> 10.0.1.199:0/2453 conn(0x55a13e5e2000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=453 cs=1 l=1). rx osd.26 seq 2770 0x55a13e7bac00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.875542) v4
-7> 2018-05-14 13:01:50.878203 7f389894c700 1 -- 10.0.0.28:6801/139719 <== osd.26 10.0.1.199:0/2453 2770 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.875542) v4 ==== 2004+0+0 (3124665699 0 0) 0x55a13e7bac00 con 0x55a13e5e2000
-6> 2018-05-14 13:01:50.878217 7f389894c700 1 -- 10.0.0.28:6801/139719 --> 10.0.1.199:0/2453 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.875542) v4 -- 0x55a13fd27200 con 0
-5> 2018-05-14 13:01:50.878700 7f389894c700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.50:0/2453 conn(0x55a13ef37000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=453 cs=1 l=1). rx osd.26 seq 2770 0x55a13e442600 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.875542) v4
-4> 2018-05-14 13:01:50.878722 7f389894c700 1 -- 10.7.2.141:6801/139719 <== osd.26 10.7.2.50:0/2453 2770 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.875542) v4 ==== 2004+0+0 (3124665699 0 0) 0x55a13e442600 con 0x55a13ef37000
-3> 2018-05-14 13:01:50.878734 7f389894c700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.50:0/2453 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.875542) v4 -- 0x55a13fd27200 con 0
-2> 2018-05-14 13:01:50.906581 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.49:6836/3920 conn(0x55a13e727000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19104 cs=1 l=0). rx osd.19 seq 5705 0x55a140516d80 MOSDPGPush(5.9b 502962/502821 [PushOp(5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56cca, version: 472938'18842085, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56cca@472938'18842085, size: 4194304, copy_subset: [], clone_subset: {5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56b45=[]}, snapset: 5b03a=[]:{56244=[56244],566c5=[566c5],56b45=[56b45],56c06=[56c06],56d84=[56d84],56f05=[56f05,56e42],56fc3=[56fc3],57081=[57081],5713e=[5713e],571fa=[571fa],572b9=[572b9],57377=[57377],57434=[57434],574f3=[574f3],575b1=[575b1],5766f=[5766f],5772d=[5772d],577ec=[577ec],578a9=[578a9],57966=[57966],57a25=[57a25],57ae2=[57ae2],57ba0=[57ba0],57c5e=[57c5e],57d1d=[57d1d],57ddb=[57ddb],57e99=[57e99],57f56=[57f56],58015=[58015],580d3=[580d3],58190=[58190],5824e=[5824e],5830d=[5830d],583ca=[583ca],58488=[58488],58547=[58547],58604=[58604],586c2=[586c2],58782=[58782],5883d=[5883d],588fb=[588fb],589b8=[589b8],58a74=[58a74],58b31=[58b31],58bf0=[58bf0],58cae=[58cae],58d6c=[58d6c],58e2a=[58e2a],58ee7=[58ee7],58fa5=[58fa5],59062=[59062],59112=[59112],591ce=[591ce],5928e=[5928e],59349=[59349],59407=[59407],594c6=[594c6],59584=[59584],59643=[59643],59700=[59700],597bd=[597bd],5987a=[5987a],59939=[59939],599f6=[599f6],59ab5=[59ab5],59b72=[59b72],59c31=[59c31],59cef=[59cef],59dac=[59dac],59e6a=[59e6a],59f26=[59f26],59fe6=[59fe6],5a0a4=[5a0a4],5a162=[5a162],5a221=[5a221],5a2e0=[5a2e0],5a39e=[5a39e],5a45a=[5a45a],5a518=[5a518],5a5d7=[5a5d7],5a695=[5a695],5a752=[5a752],5a810=[5a810],5a8cf=[5a8cf],5a98d=[5a98d],5aa4b=[5aa4b],5ab0a=[5ab0a],5abc7=[5abc7],5ac85=[5ac85],5ad43=[5ad43],5ae01=[5ae01],5aebf=[5aebf],5af7c=[5af7c],5b03a=[5b03a]}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))]) v3
-1> 2018-05-14 13:01:50.906665 7f389914d700 1 -- 10.7.2.141:6800/139719 <== osd.19 10.7.2.49:6836/3920 5705 ==== MOSDPGPush(5.9b 502962/502821 [PushOp(5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56cca, version: 472938'18842085, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56cca@472938'18842085, size: 4194304, copy_subset: [], clone_subset: {5:d97d84eb:::rbd_data.112913b238e1f29.0000000000000ba3:56b45=[]}, snapset: 5b03a=[]:{56244=[56244],566c5=[566c5],56b45=[56b45],56c06=[56c06],56d84=[56d84],56f05=[56f05,56e42],56fc3=[56fc3],57081=[57081],5713e=[5713e],571fa=[571fa],572b9=[572b9],57377=[57377],57434=[57434],574f3=[574f3],575b1=[575b1],5766f=[5766f],5772d=[5772d],577ec=[577ec],578a9=[578a9],57966=[57966],57a25=[57a25],57ae2=[57ae2],57ba0=[57ba0],57c5e=[57c5e],57d1d=[57d1d],57ddb=[57ddb],57e99=[57e99],57f56=[57f56],58015=[58015],580d3=[580d3],58190=[58190],5824e=[5824e],5830d=[5830d],583ca=[583ca],58488=[58488],58547=[58547],58604=[58604],586c2=[586c2],58782=[58782],5883d=[5883d],588fb=[588fb],589b8=[589b8],58a74=[58a74],58b31=[58b31],58bf0=[58bf0],58cae=[58cae],58d6c=[58d6c],58e2a=[58e2a],58ee7=[58ee7],58fa5=[58fa5],59062=[59062],59112=[59112],591ce=[591ce],5928e=[5928e],59349=[59349],59407=[59407],594c6=[594c6],59584=[59584],59643=[59643],59700=[59700],597bd=[597bd],5987a=[5987a],59939=[59939],599f6=[599f6],59ab5=[59ab5],59b72=[59b72],59c31=[59c31],59cef=[59cef],59dac=[59dac],59e6a=[59e6a],59f26=[59f26],59fe6=[59fe6],5a0a4=[5a0a4],5a162=[5a162],5a221=[5a221],5a2e0=[5a2e0],5a39e=[5a39e],5a45a=[5a45a],5a518=[5a518],5a5d7=[5a5d7],5a695=[5a695],5a752=[5a752],5a810=[5a810],5a8cf=[5a8cf],5a98d=[5a98d],5aa4b=[5aa4b],5ab0a=[5ab0a],5abc7=[5abc7],5ac85=[5ac85],5ad43=[5ad43],5ae01=[5ae01],5aebf=[5aebf],5af7c=[5af7c],5b03a=[5b03a]}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))]) v3 ==== 20509+0+0 (761983086 0 0) 0x55a140516d80 con 0x55a13e727000
0> 2018-05-14 13:01:50.914106 7f388493b700 -1 /build/ceph-12.2.5/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::on_local_recover(const hobject_t&, const ObjectRecoveryInfo&, ObjectContextRef, bool, ObjectStore::Transaction*)' thread 7f388493b700 time 2018-05-14 13:01:50.907217
/build/ceph-12.2.5/src/osd/PrimaryLogPG.cc: 358: FAILED assert(p != recovery_info.ss.clone_snaps.end())
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55a113785a02]
2: (PrimaryLogPG::on_local_recover(hobject_t const&, ObjectRecoveryInfo const&, std::shared_ptr<ObjectContext>, bool, ObjectStore::Transaction*)+0xd63) [0x55a1132f3873]
3: (ReplicatedBackend::handle_push(pg_shard_t, PushOp const&, PushReplyOp*, ObjectStore::Transaction*)+0x2da) [0x55a1134855ca]
4: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x12e) [0x55a1134857fe]
5: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2c1) [0x55a113494d71]
6: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x55a1133a0440]
7: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x543) [0x55a113304853]
8: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x55a11317b539]
9: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x55a113424f37]
10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x55a1131a9847]
11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x55a11378a7f4]
12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a11378d830]
13: (()+0x76ba) [0x7f389ccc36ba]
14: (clone()+0x6d) [0x7f389bd3a41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.130.log
--- end dump of recent events ---
2018-05-14 13:01:50.958594 7f388493b700 -1 *** Caught signal (Aborted) **
in thread 7f388493b700 thread_name:tp_osd_tp
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
1: (()+0xa7cab4) [0x55a113742ab4]
2: (()+0x11390) [0x7f389cccd390]
3: (gsignal()+0x38) [0x7f389bc68428]
4: (abort()+0x16a) [0x7f389bc6a02a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55a113785b8e]
6: (PrimaryLogPG::on_local_recover(hobject_t const&, ObjectRecoveryInfo const&, std::shared_ptr<ObjectContext>, bool, ObjectStore::Transaction*)+0xd63) [0x55a1132f3873]
7: (ReplicatedBackend::handle_push(pg_shard_t, PushOp const&, PushReplyOp*, ObjectStore::Transaction*)+0x2da) [0x55a1134855ca]
8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x12e) [0x55a1134857fe]
9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2c1) [0x55a113494d71]
10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x55a1133a0440]
11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x543) [0x55a113304853]
12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x55a11317b539]
13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x55a113424f37]
14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x55a1131a9847]
15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x55a11378a7f4]
16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a11378d830]
17: (()+0x76ba) [0x7f389ccc36ba]
18: (clone()+0x6d) [0x7f389bd3a41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-17> 2018-05-14 13:01:50.927479 7f389914d700 5 -- 10.7.2.141:6801/139719 >> 10.7.2.142:0/4853 conn(0x55a13f0cc800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=854 cs=1 l=1). rx osd.151 seq 2719 0x55a13ec7ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.924366) v4
-16> 2018-05-14 13:01:50.927520 7f389914d700 1 -- 10.7.2.141:6801/139719 <== osd.151 10.7.2.142:0/4853 2719 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.924366) v4 ==== 2004+0+0 (1599265769 0 0) 0x55a13ec7ec00 con 0x55a13f0cc800
-15> 2018-05-14 13:01:50.927539 7f389914d700 1 -- 10.7.2.141:6801/139719 --> 10.7.2.142:0/4853 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.924366) v4 -- 0x55a13e401000 con 0
-14> 2018-05-14 13:01:50.928421 7f389914d700 5 -- 10.0.0.28:6801/139719 >> 10.0.0.29:0/4853 conn(0x55a13f16e800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=854 cs=1 l=1). rx osd.151 seq 2719 0x55a13e43ec00 osd_ping(ping e502962 stamp 2018-05-14 13:01:50.924366) v4
-13> 2018-05-14 13:01:50.928442 7f389914d700 1 -- 10.0.0.28:6801/139719 <== osd.151 10.0.0.29:0/4853 2719 ==== osd_ping(ping e502962 stamp 2018-05-14 13:01:50.924366) v4 ==== 2004+0+0 (1599265769 0 0) 0x55a13e43ec00 con 0x55a13f16e800
-12> 2018-05-14 13:01:50.928458 7f389914d700 1 -- 10.0.0.28:6801/139719 --> 10.0.0.29:0/4853 -- osd_ping(ping_reply e502962 stamp 2018-05-14 13:01:50.924366) v4 -- 0x55a13e401000 con 0
-11> 2018-05-14 13:01:50.948542 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130313 0x55a13ec7ec00 MBackfillReserve REQUEST pgid: 7.ed, query_epoch: 502963, prio: 100 v3
-10> 2018-05-14 13:01:50.948640 7f388f1c9700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130313 ==== MBackfillReserve REQUEST pgid: 7.ed, query_epoch: 502963, prio: 100 v3 ==== 30+0+0 (2880574370 0 0) 0x55a13ec7ec00 con 0x55a13e942800
-9> 2018-05-14 13:01:50.948676 7f388f1c9700 10 monclient: _renew_subs
-8> 2018-05-14 13:01:50.948682 7f388f1c9700 10 monclient: _send_mon_message to mon.ceph-m-03 at 10.0.1.108:6789/0
-7> 2018-05-14 13:01:50.948687 7f388f1c9700 1 -- 10.0.0.28:6800/139719 --> 10.0.1.108:6789/0 -- mon_subscribe({osdmap=502963}) v2 -- 0x55a164c86480 con 0
-6> 2018-05-14 13:01:50.949118 7f389894c700 5 -- 10.0.0.28:6800/139719 >> 10.0.1.108:6789/0 conn(0x55a13e327800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5324381 cs=1 l=1). rx mon.2 seq 75 0x55a154fc8f00 osd_map(502963..502963 src has 472968..502963) v3
-5> 2018-05-14 13:01:50.949225 7f38901cb700 1 -- 10.0.0.28:6800/139719 <== mon.2 10.0.1.108:6789/0 75 ==== osd_map(502963..502963 src has 472968..502963) v3 ==== 273+0+0 (3394664797 0 0) 0x55a154fc8f00 con 0x55a13e327800
-4> 2018-05-14 13:01:50.949332 7f38901cb700 3 osd.130 502962 handle_osd_map epochs [502963,502963], i have 502962, src has [472968,502963]
-3> 2018-05-14 13:01:50.954765 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130314 0x55a14a8aff80 pg_query(21.4a4 epoch 502963) v4
-2> 2018-05-14 13:01:50.954828 7f389914d700 5 -- 10.7.2.141:6800/139719 >> 10.7.2.144:6808/62146 conn(0x55a13e942800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=5699 cs=1 l=0). rx osd.189 seq 130315 0x55a14d4b8f80 osd_map(502963..502963 src has 472968..502963) v3
-1> 2018-05-14 13:01:50.954821 7f388f1c9700 1 -- 10.7.2.141:6800/139719 <== osd.189 10.7.2.144:6808/62146 130314 ==== pg_query(21.4a4 epoch 502963) v4 ==== 162+0+0 (3726042250 0 0) 0x55a14a8aff80 con 0x55a13e942800
0> 2018-05-14 13:01:50.958594 7f388493b700 -1 *** Caught signal (Aborted) **
in thread 7f388493b700 thread_name:tp_osd_tp
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
1: (()+0xa7cab4) [0x55a113742ab4]
2: (()+0x11390) [0x7f389cccd390]
3: (gsignal()+0x38) [0x7f389bc68428]
4: (abort()+0x16a) [0x7f389bc6a02a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55a113785b8e]
6: (PrimaryLogPG::on_local_recover(hobject_t const&, ObjectRecoveryInfo const&, std::shared_ptr<ObjectContext>, bool, ObjectStore::Transaction*)+0xd63) [0x55a1132f3873]
7: (ReplicatedBackend::handle_push(pg_shard_t, PushOp const&, PushReplyOp*, ObjectStore::Transaction*)+0x2da) [0x55a1134855ca]
8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x12e) [0x55a1134857fe]
9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2c1) [0x55a113494d71]
10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x55a1133a0440]
11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x543) [0x55a113304853]
12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x55a11317b539]
13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x55a113424f37]
14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1047) [0x55a1131a9847]
15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x55a11378a7f4]
16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a11378d830]
17: (()+0x76ba) [0x7f389ccc36ba]
18: (clone()+0x6d) [0x7f389bd3a41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.130.log
--- end dump of recent events ---
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-66-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Mon May 14 08:45:51 CEST 2018
System load: 0.68 Processes: 210
Usage of /: 1.8% of 456.37GB Users logged in: 1
Memory usage: 15% IP address for p1p1: 10.7.2.60
Swap usage: 0% IP address for p1p2: 10.0.0.176
Graph this data and manage this system at:
https://landscape.canonical.com/
138 packages can be updated.
111 updates are security updates.
New release '16.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Your Hardware Enablement Stack (HWE) is supported until April 2019.
Last login: Mon May 14 08:45:51 2018 from 192.168.192.2
root@ceph-m-01:~# cat /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-luminous/ trusty main
root@ceph-m-01:~# dpkg -l | grep ceph
ii ceph 12.2.5-1trusty amd64 distributed storage and file system
ii ceph-base 12.2.5-1trusty amd64 common ceph daemon libraries and management tools
ii ceph-common 12.2.5-1trusty amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mds 12.2.5-1trusty amd64 metadata server for the ceph distributed file system
ii ceph-mgr 12.2.5-1trusty amd64 manager for the ceph distributed storage system
ii ceph-mon 12.2.5-1trusty amd64 monitor server for the ceph storage system
ii ceph-osd 12.2.5-1trusty amd64 OSD server for the ceph storage system
ii libcephfs1 10.2.10-1trusty amd64 Ceph distributed file system client library
ii libcephfs2 12.2.5-1trusty amd64 Ceph distributed file system client library
ii python-ceph 12.2.5-1trusty amd64 Meta-package for python libraries for the Ceph libraries
ii python-cephfs 12.2.5-1trusty amd64 Python 2 libraries for the Ceph libcephfs library
root@ceph-m-01:~# ceph versions
{
"mon": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 3
},
"mgr": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 3
},
"osd": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 192
},
"mds": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 3
},
"rgw": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 4
},
"rgw-nfs": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 1
},
"overall": {
"ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)": 206
}
}
root@ceph-m-01:~# ceph features
{
"mon": {
"group": {
"features": "0x1ffddff8eea4fffb",
"release": "luminous",
"num": 3
}
},
"mds": {
"group": {
"features": "0x1ffddff8eea4fffb",
"release": "luminous",
"num": 3
}
},
"osd": {
"group": {
"features": "0x1ffddff8eea4fffb",
"release": "luminous",
"num": 192
}
},
"client": {
"group": {
"features": "0x106b84a842a42",
"release": "hammer",
"num": 50
},
"group": {
"features": "0x107b84a842aca",
"release": "hammer",
"num": 77
},
"group": {
"features": "0xc1dff8eea4fffb",
"release": "hammer",
"num": 18
},
"group": {
"features": "0x7fddff8ee84bffb",
"release": "jewel",
"num": 17
},
"group": {
"features": "0x1ffddff8eea4fffb",
"release": "luminous",
"num": 165
}
}
}
root@ceph-m-01:~#
root@ceph-m-01:~# ceph -s
cluster:
id: f58a140c-a239-496d-9bef-28144dff4b27
health: HEALTH_WARN
noout flag(s) set
151643/445157614 objects misplaced (0.034%)
Degraded data redundancy: 215/445157614 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
services:
mon: 3 daemons, quorum ceph-m-01,ceph-m-02,ceph-m-03
mgr: ceph-m-03(active), standbys: ceph-m-02, ceph-m-01
mds: cephfs-1/1/1 up {0=ceph-m-01=up:active}, 2 up:standby
osd: 192 osds: 192 up, 190 in; 9 remapped pgs
flags noout
rgw: 4 daemons active
rgw-nfs: 1 daemon active
data:
pools: 20 pools, 9680 pgs
objects: 141M objects, 102120 GB
usage: 299 TB used, 195 TB / 495 TB avail
pgs: 215/445157614 objects degraded (0.000%)
151643/445157614 objects misplaced (0.034%)
9652 active+clean
18 active+clean+scrubbing+deep
7 active+remapped+backfill_wait
1 active+clean+scrubbing
1 active+undersized+degraded+remapped+backfill_wait
1 active+remapped+backfilling
io:
client: 131 MB/s rd, 15444 kB/s wr, 475 op/s rd, 981 op/s wr
recovery: 2010 kB/s, 7 objects/s
root@ceph-m-01:~# ceph health detail
HEALTH_WARN noout flag(s) set; 151601/445157614 objects misplaced (0.034%); Degraded data redundancy: 215/445157614 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
OSDMAP_FLAGS noout flag(s) set
OBJECT_MISPLACED 151601/445157614 objects misplaced (0.034%)
PG_DEGRADED Degraded data redundancy: 215/445157614 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
pg 5.9b is stuck undersized for 7987.201465, current state active+undersized+degraded+remapped+backfill_wait, last acting [19,166]
root@ceph-m-01:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME
-9 527.66998 - 249T 155T 97037G 0 0 - root spin
-14 0 - 0 0 0 0 0 - host ceph-mini-01-spin
-15 0 - 0 0 0 0 0 - host ceph-mini-02-03-spin
-2 47.59999 - 48304G 28670G 19634G 59.35 0.97 - host ceph-s-01-spin
0 hdd 1.81000 1.00000 1857G 775G 1081G 41.78 0.68 66 osd.0
1 hdd 1.81000 1.00000 1857G 1201G 656G 64.67 1.05 96 osd.1
2 hdd 1.81000 1.00000 1857G 1171G 685G 63.07 1.03 89 osd.2
3 hdd 1.81000 1.00000 1857G 936G 920G 50.41 0.82 66 osd.3
4 hdd 1.59999 1.00000 1857G 1283G 573G 69.12 1.13 83 osd.4
5 hdd 1.81000 1.00000 1857G 1007G 849G 54.24 0.88 86 osd.5
6 hdd 1.81000 1.00000 1857G 1286G 571G 69.25 1.13 89 osd.6
7 hdd 1.81000 1.00000 1857G 865G 992G 46.58 0.76 70 osd.7
12 hdd 1.81000 1.00000 1857G 1108G 748G 59.70 0.97 88 osd.12
13 hdd 1.59999 1.00000 1857G 1371G 486G 73.83 1.20 91 osd.13
16 hdd 1.81000 1.00000 1857G 1030G 826G 55.49 0.90 86 osd.16
21 hdd 1.81000 1.00000 1857G 1044G 812G 56.23 0.92 99 osd.21
25 hdd 4.00000 1.00000 3719G 2331G 1387G 62.69 1.02 185 osd.25
26 hdd 1.59999 1.00000 1857G 941G 915G 50.70 0.83 81 osd.26
30 hdd 1.81000 1.00000 1857G 1143G 713G 61.59 1.00 89 osd.30
31 hdd 2.79999 1.00000 2788G 1894G 893G 67.96 1.11 158 osd.31
33 hdd 2.79999 1.00000 2788G 1518G 1269G 54.45 0.89 140 osd.33
34 hdd 1.57999 1.00000 1857G 1132G 724G 60.99 0.99 97 osd.34
38 hdd 1.81000 1.00000 1857G 955G 902G 51.43 0.84 77 osd.38
39 hdd 1.89999 1.00000 1857G 1098G 758G 59.18 0.96 97 osd.39
45 hdd 4.00000 1.00000 3719G 2260G 1459G 60.77 0.99 172 osd.45
47 hdd 4.00000 1.00000 3719G 2312G 1406G 62.17 1.01 172 osd.47
-3 60.00000 - 59482G 34884G 24597G 58.65 0.96 - host ceph-s-02-spin
42 hdd 2.79999 1.00000 2788G 1352G 1436G 48.49 0.79 129 osd.42
49 hdd 2.79999 1.00000 2788G 1555G 1232G 55.79 0.91 127 osd.49
51 hdd 2.79999 1.00000 2788G 1306G 1481G 46.86 0.76 115 osd.51
65 hdd 4.00000 1.00000 3719G 2144G 1574G 57.67 0.94 179 osd.65
68 hdd 2.79999 1.00000 2788G 1216G 1572G 43.62 0.71 106 osd.68
73 hdd 2.79999 1.00000 2788G 1625G 1162G 58.29 0.95 137 osd.73
82 hdd 2.79999 1.00000 2788G 1881G 906G 67.49 1.10 141 osd.82
87 hdd 2.79999 1.00000 2788G 1546G 1242G 55.45 0.90 119 osd.87
88 hdd 2.79999 1.00000 2788G 1853G 934G 66.47 1.08 140 osd.88
89 hdd 2.79999 1.00000 2788G 1919G 868G 68.84 1.12 147 osd.89
91 hdd 2.79999 1.00000 2788G 2130G 657G 76.42 1.25 144 osd.91
92 hdd 2.79999 1.00000 2788G 1503G 1285G 53.91 0.88 112 osd.92
95 hdd 2.79999 1.00000 2788G 1789G 999G 64.17 1.05 142 osd.95
96 hdd 2.79999 1.00000 2788G 1734G 1054G 62.20 1.01 131 osd.96
97 hdd 2.79999 1.00000 2788G 1657G 1130G 59.46 0.97 126 osd.97
98 hdd 2.79999 1.00000 2788G 1359G 1428G 48.75 0.79 114 osd.98
99 hdd 2.79999 1.00000 2788G 1524G 1264G 54.66 0.89 124 osd.99
100 hdd 2.79999 1.00000 2788G 1695G 1092G 60.81 0.99 123 osd.100
101 hdd 2.79999 1.00000 2788G 1503G 1284G 53.94 0.88 139 osd.101
102 hdd 2.79999 1.00000 2788G 1784G 1003G 64.00 1.04 128 osd.102
171 hdd 2.79999 1.00000 2788G 1802G 985G 64.65 1.05 121 osd.171
-4 65.01999 - 57620G 38570G 19050G 66.94 1.09 - host ceph-s-03-spin
17 hdd 3.20000 1.00000 2788G 1503G 1284G 53.92 0.88 129 osd.17
19 hdd 3.20000 0.95001 2788G 2171G 616G 77.89 1.27 166 osd.19
22 hdd 3.20000 1.00000 2788G 2021G 766G 72.50 1.18 146 osd.22
24 hdd 3.20000 1.00000 2788G 1868G 919G 67.03 1.09 146 osd.24
27 hdd 3.20000 0.95001 2788G 2062G 725G 73.99 1.21 158 osd.27
29 hdd 3.20000 1.00000 2788G 1634G 1154G 58.61 0.96 144 osd.29
32 hdd 3.20000 0.95001 2788G 2078G 709G 74.56 1.22 154 osd.32
35 hdd 3.20000 1.00000 2788G 2023G 764G 72.59 1.18 160 osd.35
37 hdd 3.20000 1.00000 2788G 1942G 845G 69.68 1.14 144 osd.37
40 hdd 3.20000 1.00000 2788G 1930G 857G 69.23 1.13 147 osd.40
43 hdd 3.20000 1.00000 2788G 1849G 938G 66.33 1.08 149 osd.43
46 hdd 3.20000 1.00000 2788G 1597G 1190G 57.30 0.93 129 osd.46
48 hdd 1.81000 1.00000 1857G 756G 1101G 40.71 0.66 68 osd.48
50 hdd 3.20000 1.00000 2788G 2011G 776G 72.14 1.18 148 osd.50
52 hdd 3.20000 1.00000 2788G 1954G 833G 70.09 1.14 150 osd.52
56 hdd 1.81000 1.00000 1857G 1202G 654G 64.76 1.06 87 osd.56
58 hdd 3.20000 1.00000 2788G 1900G 887G 68.17 1.11 159 osd.58
60 hdd 3.20000 1.00000 2788G 1890G 897G 67.79 1.10 137 osd.60
63 hdd 3.20000 1.00000 2788G 1960G 827G 70.33 1.15 143 osd.63
78 hdd 4.00000 1.00000 3719G 2468G 1250G 66.38 1.08 193 osd.78
105 hdd 3.00000 1.00000 2788G 1739G 1048G 62.40 1.02 131 osd.105
-10 36.04999 - 38078G 22323G 15755G 58.62 0.96 - host ceph-s-04-spin
8 hdd 0.89999 1.00000 926G 444G 481G 48.03 0.78 40 osd.8
9 hdd 0.89999 1.00000 926G 562G 363G 60.77 0.99 48 osd.9
10 hdd 3.00000 1.00000 2788G 1568G 1219G 56.26 0.92 137 osd.10
53 hdd 1.64999 1.00000 1857G 919G 937G 49.53 0.81 76 osd.53
55 hdd 1.64999 1.00000 1857G 1021G 835G 55.00 0.90 79 osd.55
66 hdd 1.64999 1.00000 1857G 1069G 787G 57.58 0.94 77 osd.66
67 hdd 1.64999 1.00000 1857G 1098G 758G 59.15 0.96 78 osd.67
69 hdd 1.64999 1.00000 1857G 1032G 825G 55.57 0.91 88 osd.69
70 hdd 1.64999 1.00000 1857G 846G 1010G 45.57 0.74 70 osd.70
72 hdd 1.64999 1.00000 1857G 1213G 643G 65.34 1.07 84 osd.72
74 hdd 1.64999 1.00000 1857G 950G 906G 51.20 0.83 73 osd.74
75 hdd 1.64999 1.00000 1857G 1101G 756G 59.29 0.97 79 osd.75
76 hdd 4.00000 1.00000 3719G 2559G 1159G 68.81 1.12 201 osd.76
77 hdd 1.64999 1.00000 1857G 1006G 850G 54.21 0.88 83 osd.77
79 hdd 1.64999 1.00000 1857G 854G 1002G 46.02 0.75 65 osd.79
80 hdd 1.64999 1.00000 1857G 1229G 627G 66.22 1.08 82 osd.80
81 hdd 1.64999 1.00000 1857G 1024G 832G 55.17 0.90 90 osd.81
172 hdd 4.00000 1.00000 3719G 2815G 903G 75.70 1.23 205 osd.172
173 hdd 1.79999 1.00000 1857G 1003G 853G 54.04 0.88 81 osd.173
-12 39.00000 - 39004G 23681G 15323G 60.71 0.99 - host ceph-s-05-spin
14 hdd 1.89999 1.00000 1857G 1229G 627G 66.19 1.08 94 osd.14
15 hdd 1.89999 1.00000 1857G 1016G 840G 54.75 0.89 83 osd.15
18 hdd 2.79999 1.00000 2788G 2048G 739G 73.48 1.20 159 osd.18
20 hdd 2.79999 1.00000 2788G 1769G 1018G 63.48 1.03 142 osd.20
36 hdd 1.89999 1.00000 1857G 1018G 838G 54.86 0.89 90 osd.36
107 hdd 1.89999 1.00000 1857G 959G 897G 51.68 0.84 78 osd.107
109 hdd 1.89999 0.84999 1857G 1195G 661G 64.38 1.05 90 osd.109
110 hdd 1.89999 1.00000 1857G 1048G 808G 56.45 0.92 86 osd.110
111 hdd 1.89999 1.00000 1857G 969G 887G 52.23 0.85 82 osd.111
112 hdd 1.89999 1.00000 1857G 1125G 731G 60.59 0.99 87 osd.112
113 hdd 1.89999 1.00000 1857G 927G 929G 49.96 0.81 79 osd.113
114 hdd 1.89999 1.00000 1857G 1083G 773G 58.34 0.95 84 osd.114
115 hdd 1.89999 1.00000 1857G 1186G 670G 63.89 1.04 92 osd.115
116 hdd 1.89999 1.00000 1857G 1054G 802G 56.80 0.93 81 osd.116
120 hdd 1.89999 1.00000 1857G 1113G 743G 59.97 0.98 81 osd.120
121 hdd 1.89999 0.95001 1857G 1394G 462G 75.12 1.22 101 osd.121
123 hdd 1.89999 1.00000 1857G 1043G 813G 56.21 0.92 82 osd.123
124 hdd 1.89999 1.00000 1857G 1144G 712G 61.65 1.00 85 osd.124
125 hdd 1.89999 1.00000 1857G 1200G 656G 64.63 1.05 86 osd.125
126 hdd 1.89999 1.00000 1857G 1148G 708G 61.85 1.01 90 osd.126
-16 88.00000 - 74390G 49054G 25336G 65.94 1.07 - host ceph-s-06-spin
57 hdd 4.00000 0.85001 3719G 2829G 889G 76.08 1.24 206 osd.57
59 hdd 4.00000 1.00000 3719G 2449G 1270G 65.85 1.07 218 osd.59
61 hdd 4.00000 1.00000 3719G 2541G 1177G 68.35 1.11 198 osd.61
71 hdd 4.00000 0.95000 3719G 2489G 1230G 66.92 1.09 183 osd.71
83 hdd 4.00000 1.00000 3719G 2365G 1354G 63.59 1.04 172 osd.83
90 hdd 4.00000 0.95000 3719G 2455G 1264G 66.01 1.08 202 osd.90
93 hdd 4.00000 1.00000 3719G 2398G 1320G 64.49 1.05 191 osd.93
94 hdd 4.00000 1.00000 3719G 2312G 1407G 62.16 1.01 194 osd.94
103 hdd 4.00000 1.00000 3719G 2489G 1229G 66.95 1.09 198 osd.103
104 hdd 4.00000 0 0 0 0 0 0 0 osd.104
106 hdd 4.00000 1.00000 3719G 2461G 1257G 66.18 1.08 183 osd.106
108 hdd 4.00000 0.95000 3719G 2590G 1128G 69.65 1.14 192 osd.108
117 hdd 4.00000 1.00000 3719G 2487G 1231G 66.89 1.09 201 osd.117
119 hdd 4.00000 0 0 0 0 0 0 0 osd.119
122 hdd 4.00000 1.00000 3719G 2431G 1287G 65.37 1.07 195 osd.122
127 hdd 4.00000 1.00000 3719G 2406G 1312G 64.70 1.05 173 osd.127
128 hdd 4.00000 1.00000 3719G 2282G 1436G 61.38 1.00 174 osd.128
129 hdd 4.00000 1.00000 3719G 2211G 1507G 59.45 0.97 161 osd.129
130 hdd 4.00000 0.89999 3726G 2490G 1235G 66.83 1.09 180 osd.130
131 hdd 4.00000 1.00000 3719G 2405G 1314G 64.67 1.05 172 osd.131
132 hdd 4.00000 0.95000 3719G 2615G 1103G 70.33 1.15 187 osd.132
133 hdd 4.00000 0.90001 3719G 2341G 1377G 62.97 1.03 189 osd.133
-17 88.00000 - 81822G 47599G 34222G 58.17 0.95 - host ceph-s-07-spin
134 hdd 4.00000 1.00000 3719G 2193G 1525G 58.98 0.96 175 osd.134
135 hdd 4.00000 1.00000 3719G 2115G 1603G 56.88 0.93 165 osd.135
136 hdd 4.00000 1.00000 3719G 2211G 1507G 59.46 0.97 175 osd.136
137 hdd 4.00000 1.00000 3719G 2284G 1434G 61.43 1.00 173 osd.137
138 hdd 4.00000 1.00000 3719G 2519G 1199G 67.74 1.10 197 osd.138
139 hdd 4.00000 1.00000 3719G 2199G 1519G 59.14 0.96 179 osd.139
140 hdd 4.00000 1.00000 3719G 1946G 1772G 52.33 0.85 164 osd.140
141 hdd 4.00000 1.00000 3719G 2137G 1581G 57.48 0.94 165 osd.141
142 hdd 4.00000 1.00000 3719G 2094G 1625G 56.31 0.92 155 osd.142
143 hdd 4.00000 1.00000 3719G 2322G 1397G 62.44 1.02 189 osd.143
144 hdd 4.00000 1.00000 3719G 2151G 1567G 57.85 0.94 167 osd.144
145 hdd 4.00000 1.00000 3719G 2021G 1697G 54.36 0.89 154 osd.145
146 hdd 4.00000 1.00000 3719G 2172G 1546G 58.42 0.95 176 osd.146
147 hdd 4.00000 1.00000 3719G 2127G 1591G 57.22 0.93 157 osd.147
148 hdd 4.00000 1.00000 3719G 2449G 1269G 65.86 1.07 189 osd.148
149 hdd 4.00000 1.00000 3719G 1904G 1814G 51.22 0.83 158 osd.149
150 hdd 4.00000 1.00000 3719G 2260G 1458G 60.78 0.99 183 osd.150
151 hdd 4.00000 1.00000 3719G 2133G 1585G 57.36 0.93 162 osd.151
152 hdd 4.00000 1.00000 3719G 2260G 1459G 60.77 0.99 187 osd.152
153 hdd 4.00000 1.00000 3719G 2384G 1334G 64.12 1.05 187 osd.153
154 hdd 4.00000 1.00000 3719G 1990G 1728G 53.52 0.87 162 osd.154
155 hdd 4.00000 1.00000 3719G 1717G 2002G 46.17 0.75 151 osd.155
-18 32.00000 - 32514G 18834G 13679G 57.93 0.94 - host ceph-s-08-spin
11 hdd 2.00000 1.00000 1863G 1417G 445G 76.07 1.24 94 osd.11
41 hdd 1.79999 1.00000 1857G 1020G 836G 54.95 0.90 82 osd.41
44 hdd 1.79999 1.00000 1857G 1082G 774G 58.29 0.95 86 osd.44
54 hdd 3.00000 1.00000 2794G 1748G 1046G 62.55 1.02 145 osd.54
62 hdd 1.79999 1.00000 1857G 1100G 756G 59.28 0.97 90 osd.62
85 hdd 1.79999 1.00000 1857G 874G 982G 47.12 0.77 85 osd.85
156 hdd 1.79999 1.00000 1857G 804G 1052G 43.32 0.71 78 osd.156
158 hdd 1.79999 1.00000 1857G 1017G 839G 54.77 0.89 74 osd.158
160 hdd 1.79999 1.00000 1857G 1329G 527G 71.60 1.17 94 osd.160
161 hdd 1.79999 1.00000 1857G 1023G 833G 55.13 0.90 79 osd.161
162 hdd 1.79999 1.00000 1857G 1018G 838G 54.83 0.89 90 osd.162
163 hdd 1.79999 1.00000 1857G 1033G 824G 55.63 0.91 77 osd.163
164 hdd 1.79999 0.79999 1857G 1099G 757G 59.20 0.96 84 osd.164
165 hdd 1.79999 1.00000 1857G 1049G 807G 56.53 0.92 85 osd.165
166 hdd 1.79999 1.00000 1857G 907G 949G 48.86 0.80 73 osd.166
168 hdd 1.79999 1.00000 1857G 1256G 600G 67.68 1.10 95 osd.168
170 hdd 1.79999 0.89999 1857G 1051G 806G 56.60 0.92 80 osd.170
-19 72.00000 - 67036G 43237G 23799G 64.50 1.05 - host ceph-s-09-spin
84 hdd 3.00000 1.00000 2794G 1540G 1253G 55.13 0.90 121 osd.84
118 hdd 3.00000 1.00000 2788G 2085G 702G 74.80 1.22 143 osd.118
157 hdd 3.00000 1.00000 2794G 1700G 1093G 60.86 0.99 135 osd.157
159 hdd 3.00000 1.00000 2794G 1664G 1130G 59.56 0.97 138 osd.159
167 hdd 3.00000 1.00000 2794G 1910G 883G 68.37 1.11 147 osd.167
169 hdd 3.00000 1.00000 2788G 1657G 1130G 59.46 0.97 131 osd.169
174 hdd 3.00000 1.00000 2794G 1926G 867G 68.94 1.12 136 osd.174
175 hdd 3.00000 1.00000 2788G 1458G 1329G 52.30 0.85 130 osd.175
176 hdd 3.00000 1.00000 2788G 1681G 1106G 60.32 0.98 141 osd.176
177 hdd 3.00000 1.00000 2794G 1823G 970G 65.25 1.06 148 osd.177
178 hdd 3.00000 1.00000 2788G 1532G 1255G 54.96 0.90 131 osd.178
179 hdd 3.00000 1.00000 2794G 1752G 1042G 62.69 1.02 133 osd.179
180 hdd 3.00000 1.00000 2794G 1899G 894G 67.98 1.11 139 osd.180
181 hdd 3.00000 1.00000 2794G 1770G 1023G 63.36 1.03 148 osd.181
182 hdd 3.00000 1.00000 2794G 1912G 882G 68.43 1.12 140 osd.182
183 hdd 3.00000 1.00000 2794G 2007G 786G 71.85 1.17 155 osd.183
184 hdd 3.00000 1.00000 2794G 1723G 1070G 61.68 1.01 132 osd.184
185 hdd 3.00000 1.00000 2794G 1634G 1160G 58.48 0.95 115 osd.185
186 hdd 3.00000 1.00000 2794G 1955G 838G 69.98 1.14 146 osd.186
187 hdd 3.00000 0.95001 2794G 1883G 911G 67.39 1.10 126 osd.187
188 hdd 3.00000 1.00000 2794G 1717G 1076G 61.47 1.00 127 osd.188
189 hdd 3.00000 1.00000 2794G 1954G 840G 69.93 1.14 140 osd.189
190 hdd 3.00000 1.00000 2794G 1940G 854G 69.44 1.13 145 osd.190
191 hdd 3.00000 1.00000 2794G 2103G 690G 75.28 1.23 145 osd.191
-8 1.83997 - 1886G 12667M 1874G 0 0 - root ssd
-5 0 - 0 0 0 0 0 - host ceph-s-01-ssd
-6 0.45999 - 471G 3032M 468G 0.63 0.01 - host ceph-s-02-ssd
28 hdd 0.45999 1.00000 471G 3032M 468G 0.63 0.01 792 osd.28
-7 0.45999 - 471G 3364M 468G 0.70 0.01 - host ceph-s-03-ssd
64 hdd 0.45999 1.00000 471G 3364M 468G 0.70 0.01 779 osd.64
-11 0.45999 - 471G 3200M 468G 0.66 0.01 - host ceph-s-04-ssd
86 hdd 0.45999 1.00000 471G 3200M 468G 0.66 0.01 757 osd.86
-13 0.45999 - 471G 3069M 468G 0.64 0.01 - host ceph-s-05-ssd
23 hdd 0.45999 1.00000 471G 3069M 468G 0.64 0.01 744 osd.23
-1 529.50995 - 249T 155T 97037G 0 0 - root default
-14 0 - 0 0 0 0 0 - host ceph-mini-01-spin
-15 0 - 0 0 0 0 0 - host ceph-mini-02-03-spin
-2 47.59999 - 48304G 28670G 19634G 59.35 0.97 - host ceph-s-01-spin
0 hdd 1.81000 1.00000 1857G 775G 1081G 41.78 0.68 66 osd.0
1 hdd 1.81000 1.00000 1857G 1201G 656G 64.67 1.05 96 osd.1
2 hdd 1.81000 1.00000 1857G 1171G 685G 63.07 1.03 89 osd.2
3 hdd 1.81000 1.00000 1857G 936G 920G 50.41 0.82 66 osd.3
4 hdd 1.59999 1.00000 1857G 1283G 573G 69.12 1.13 83 osd.4
5 hdd 1.81000 1.00000 1857G 1007G 849G 54.24 0.88 86 osd.5
6 hdd 1.81000 1.00000 1857G 1286G 571G 69.25 1.13 89 osd.6
7 hdd 1.81000 1.00000 1857G 865G 992G 46.58 0.76 70 osd.7
12 hdd 1.81000 1.00000 1857G 1108G 748G 59.70 0.97 88 osd.12
13 hdd 1.59999 1.00000 1857G 1371G 486G 73.83 1.20 91 osd.13
16 hdd 1.81000 1.00000 1857G 1030G 826G 55.49 0.90 86 osd.16
21 hdd 1.81000 1.00000 1857G 1044G 812G 56.23 0.92 99 osd.21
25 hdd 4.00000 1.00000 3719G 2331G 1387G 62.69 1.02 185 osd.25
26 hdd 1.59999 1.00000 1857G 941G 915G 50.70 0.83 81 osd.26
30 hdd 1.81000 1.00000 1857G 1143G 713G 61.59 1.00 89 osd.30
31 hdd 2.79999 1.00000 2788G 1894G 893G 67.96 1.11 158 osd.31
33 hdd 2.79999 1.00000 2788G 1518G 1269G 54.45 0.89 140 osd.33
34 hdd 1.57999 1.00000 1857G 1132G 724G 60.99 0.99 97 osd.34
38 hdd 1.81000 1.00000 1857G 955G 902G 51.43 0.84 77 osd.38
39 hdd 1.89999 1.00000 1857G 1098G 758G 59.18 0.96 97 osd.39
45 hdd 4.00000 1.00000 3719G 2260G 1459G 60.77 0.99 172 osd.45
47 hdd 4.00000 1.00000 3719G 2312G 1406G 62.17 1.01 172 osd.47
-5 0 - 0 0 0 0 0 - host ceph-s-01-ssd
-3 60.00000 - 59482G 34884G 24597G 58.65 0.96 - host ceph-s-02-spin
42 hdd 2.79999 1.00000 2788G 1352G 1436G 48.49 0.79 129 osd.42
49 hdd 2.79999 1.00000 2788G 1555G 1232G 55.79 0.91 127 osd.49
51 hdd 2.79999 1.00000 2788G 1306G 1481G 46.86 0.76 115 osd.51
65 hdd 4.00000 1.00000 3719G 2144G 1574G 57.67 0.94 179 osd.65
68 hdd 2.79999 1.00000 2788G 1216G 1572G 43.62 0.71 106 osd.68
73 hdd 2.79999 1.00000 2788G 1625G 1162G 58.29 0.95 137 osd.73
82 hdd 2.79999 1.00000 2788G 1881G 906G 67.49 1.10 141 osd.82
87 hdd 2.79999 1.00000 2788G 1546G 1242G 55.45 0.90 119 osd.87
88 hdd 2.79999 1.00000 2788G 1853G 934G 66.47 1.08 140 osd.88
89 hdd 2.79999 1.00000 2788G 1919G 868G 68.84 1.12 147 osd.89
91 hdd 2.79999 1.00000 2788G 2130G 657G 76.42 1.25 144 osd.91
92 hdd 2.79999 1.00000 2788G 1503G 1285G 53.91 0.88 112 osd.92
95 hdd 2.79999 1.00000 2788G 1789G 999G 64.17 1.05 142 osd.95
96 hdd 2.79999 1.00000 2788G 1734G 1054G 62.20 1.01 131 osd.96
97 hdd 2.79999 1.00000 2788G 1657G 1130G 59.46 0.97 126 osd.97
98 hdd 2.79999 1.00000 2788G 1359G 1428G 48.75 0.79 114 osd.98
99 hdd 2.79999 1.00000 2788G 1524G 1264G 54.66 0.89 124 osd.99
100 hdd 2.79999 1.00000 2788G 1695G 1092G 60.81 0.99 123 osd.100
101 hdd 2.79999 1.00000 2788G 1503G 1284G 53.94 0.88 139 osd.101
102 hdd 2.79999 1.00000 2788G 1784G 1003G 64.00 1.04 128 osd.102
171 hdd 2.79999 1.00000 2788G 1802G 985G 64.65 1.05 121 osd.171
-6 0.45999 - 471G 3032M 468G 0.63 0.01 - host ceph-s-02-ssd
28 hdd 0.45999 1.00000 471G 3032M 468G 0.63 0.01 792 osd.28
-4 65.01999 - 57620G 38570G 19050G 66.94 1.09 - host ceph-s-03-spin
17 hdd 3.20000 1.00000 2788G 1503G 1284G 53.92 0.88 129 osd.17
19 hdd 3.20000 0.95001 2788G 2171G 616G 77.89 1.27 166 osd.19
22 hdd 3.20000 1.00000 2788G 2021G 766G 72.50 1.18 146 osd.22
24 hdd 3.20000 1.00000 2788G 1868G 919G 67.03 1.09 146 osd.24
27 hdd 3.20000 0.95001 2788G 2062G 725G 73.99 1.21 158 osd.27
29 hdd 3.20000 1.00000 2788G 1634G 1154G 58.61 0.96 144 osd.29
32 hdd 3.20000 0.95001 2788G 2078G 709G 74.56 1.22 154 osd.32
35 hdd 3.20000 1.00000 2788G 2023G 764G 72.59 1.18 160 osd.35
37 hdd 3.20000 1.00000 2788G 1942G 845G 69.68 1.14 144 osd.37
40 hdd 3.20000 1.00000 2788G 1930G 857G 69.23 1.13 147 osd.40
43 hdd 3.20000 1.00000 2788G 1849G 938G 66.33 1.08 149 osd.43
46 hdd 3.20000 1.00000 2788G 1597G 1190G 57.30 0.93 129 osd.46
48 hdd 1.81000 1.00000 1857G 756G 1101G 40.71 0.66 68 osd.48
50 hdd 3.20000 1.00000 2788G 2011G 776G 72.14 1.18 148 osd.50
52 hdd 3.20000 1.00000 2788G 1954G 833G 70.09 1.14 150 osd.52
56 hdd 1.81000 1.00000 1857G 1202G 654G 64.76 1.06 87 osd.56
58 hdd 3.20000 1.00000 2788G 1900G 887G 68.17 1.11 159 osd.58
60 hdd 3.20000 1.00000 2788G 1890G 897G 67.79 1.10 137 osd.60
63 hdd 3.20000 1.00000 2788G 1960G 827G 70.33 1.15 143 osd.63
78 hdd 4.00000 1.00000 3719G 2468G 1250G 66.38 1.08 193 osd.78
105 hdd 3.00000 1.00000 2788G 1739G 1048G 62.40 1.02 131 osd.105
-7 0.45999 - 471G 3364M 468G 0.70 0.01 - host ceph-s-03-ssd
64 hdd 0.45999 1.00000 471G 3364M 468G 0.70 0.01 779 osd.64
-10 36.04999 - 38078G 22323G 15755G 58.62 0.96 - host ceph-s-04-spin
8 hdd 0.89999 1.00000 926G 444G 481G 48.03 0.78 40 osd.8
9 hdd 0.89999 1.00000 926G 562G 363G 60.77 0.99 48 osd.9
10 hdd 3.00000 1.00000 2788G 1568G 1219G 56.26 0.92 137 osd.10
53 hdd 1.64999 1.00000 1857G 919G 937G 49.53 0.81 76 osd.53
55 hdd 1.64999 1.00000 1857G 1021G 835G 55.00 0.90 79 osd.55
66 hdd 1.64999 1.00000 1857G 1069G 787G 57.58 0.94 77 osd.66
67 hdd 1.64999 1.00000 1857G 1098G 758G 59.15 0.96 78 osd.67
69 hdd 1.64999 1.00000 1857G 1032G 825G 55.57 0.91 88 osd.69
70 hdd 1.64999 1.00000 1857G 846G 1010G 45.57 0.74 70 osd.70
72 hdd 1.64999 1.00000 1857G 1213G 643G 65.34 1.07 84 osd.72
74 hdd 1.64999 1.00000 1857G 950G 906G 51.20 0.83 73 osd.74
75 hdd 1.64999 1.00000 1857G 1101G 756G 59.29 0.97 79 osd.75
76 hdd 4.00000 1.00000 3719G 2559G 1159G 68.81 1.12 201 osd.76
77 hdd 1.64999 1.00000 1857G 1006G 850G 54.21 0.88 83 osd.77
79 hdd 1.64999 1.00000 1857G 854G 1002G 46.02 0.75 65 osd.79
80 hdd 1.64999 1.00000 1857G 1229G 627G 66.22 1.08 82 osd.80
81 hdd 1.64999 1.00000 1857G 1024G 832G 55.17 0.90 90 osd.81
172 hdd 4.00000 1.00000 3719G 2815G 903G 75.70 1.23 205 osd.172
173 hdd 1.79999 1.00000 1857G 1003G 853G 54.04 0.88 81 osd.173
-11 0.45999 - 471G 3200M 468G 0.66 0.01 - host ceph-s-04-ssd
86 hdd 0.45999 1.00000 471G 3200M 468G 0.66 0.01 757 osd.86
-12 39.00000 - 39004G 23681G 15323G 60.71 0.99 - host ceph-s-05-spin
14 hdd 1.89999 1.00000 1857G 1229G 627G 66.19 1.08 94 osd.14
15 hdd 1.89999 1.00000 1857G 1016G 840G 54.75 0.89 83 osd.15
18 hdd 2.79999 1.00000 2788G 2048G 739G 73.48 1.20 159 osd.18
20 hdd 2.79999 1.00000 2788G 1769G 1018G 63.48 1.03 142 osd.20
36 hdd 1.89999 1.00000 1857G 1018G 838G 54.86 0.89 90 osd.36
107 hdd 1.89999 1.00000 1857G 959G 897G 51.68 0.84 78 osd.107
109 hdd 1.89999 0.84999 1857G 1195G 661G 64.38 1.05 90 osd.109
110 hdd 1.89999 1.00000 1857G 1048G 808G 56.45 0.92 86 osd.110
111 hdd 1.89999 1.00000 1857G 969G 887G 52.23 0.85 82 osd.111
112 hdd 1.89999 1.00000 1857G 1125G 731G 60.59 0.99 87 osd.112
113 hdd 1.89999 1.00000 1857G 927G 929G 49.96 0.81 79 osd.113
114 hdd 1.89999 1.00000 1857G 1083G 773G 58.34 0.95 84 osd.114
115 hdd 1.89999 1.00000 1857G 1186G 670G 63.89 1.04 92 osd.115
116 hdd 1.89999 1.00000 1857G 1054G 802G 56.80 0.93 81 osd.116
120 hdd 1.89999 1.00000 1857G 1113G 743G 59.97 0.98 81 osd.120
121 hdd 1.89999 0.95001 1857G 1394G 462G 75.12 1.22 101 osd.121
123 hdd 1.89999 1.00000 1857G 1043G 813G 56.21 0.92 82 osd.123
124 hdd 1.89999 1.00000 1857G 1144G 712G 61.65 1.00 85 osd.124
125 hdd 1.89999 1.00000 1857G 1200G 656G 64.63 1.05 86 osd.125
126 hdd 1.89999 1.00000 1857G 1148G 708G 61.85 1.01 90 osd.126
-13 0.45999 - 471G 3069M 468G 0.64 0.01 - host ceph-s-05-ssd
23 hdd 0.45999 1.00000 471G 3069M 468G 0.64 0.01 744 osd.23
-16 88.00000 - 74390G 49054G 25336G 65.94 1.07 - host ceph-s-06-spin
57 hdd 4.00000 0.85001 3719G 2829G 889G 76.08 1.24 206 osd.57
59 hdd 4.00000 1.00000 3719G 2449G 1270G 65.85 1.07 218 osd.59
61 hdd 4.00000 1.00000 3719G 2541G 1177G 68.35 1.11 198 osd.61
71 hdd 4.00000 0.95000 3719G 2489G 1230G 66.92 1.09 183 osd.71
83 hdd 4.00000 1.00000 3719G 2365G 1354G 63.59 1.04 172 osd.83
90 hdd 4.00000 0.95000 3719G 2455G 1264G 66.01 1.08 202 osd.90
93 hdd 4.00000 1.00000 3719G 2398G 1320G 64.49 1.05 191 osd.93
94 hdd 4.00000 1.00000 3719G 2312G 1407G 62.16 1.01 194 osd.94
103 hdd 4.00000 1.00000 3719G 2489G 1229G 66.95 1.09 198 osd.103
104 hdd 4.00000 0 0 0 0 0 0 0 osd.104
106 hdd 4.00000 1.00000 3719G 2461G 1257G 66.18 1.08 183 osd.106
108 hdd 4.00000 0.95000 3719G 2590G 1128G 69.65 1.14 192 osd.108
117 hdd 4.00000 1.00000 3719G 2487G 1231G 66.89 1.09 201 osd.117
119 hdd 4.00000 0 0 0 0 0 0 0 osd.119
122 hdd 4.00000 1.00000 3719G 2431G 1287G 65.37 1.07 195 osd.122
127 hdd 4.00000 1.00000 3719G 2406G 1312G 64.70 1.05 173 osd.127
128 hdd 4.00000 1.00000 3719G 2282G 1436G 61.38 1.00 174 osd.128
129 hdd 4.00000 1.00000 3719G 2211G 1507G 59.45 0.97 161 osd.129
130 hdd 4.00000 0.89999 3726G 2490G 1235G 66.83 1.09 180 osd.130
131 hdd 4.00000 1.00000 3719G 2405G 1314G 64.67 1.05 172 osd.131
132 hdd 4.00000 0.95000 3719G 2615G 1103G 70.33 1.15 187 osd.132
133 hdd 4.00000 0.90001 3719G 2341G 1377G 62.97 1.03 189 osd.133
-17 88.00000 - 81822G 47599G 34222G 58.17 0.95 - host ceph-s-07-spin
134 hdd 4.00000 1.00000 3719G 2193G 1525G 58.98 0.96 175 osd.134
135 hdd 4.00000 1.00000 3719G 2115G 1603G 56.88 0.93 165 osd.135
136 hdd 4.00000 1.00000 3719G 2211G 1507G 59.46 0.97 175 osd.136
137 hdd 4.00000 1.00000 3719G 2284G 1434G 61.43 1.00 173 osd.137
138 hdd 4.00000 1.00000 3719G 2519G 1199G 67.74 1.10 197 osd.138
139 hdd 4.00000 1.00000 3719G 2199G 1519G 59.14 0.96 179 osd.139
140 hdd 4.00000 1.00000 3719G 1946G 1772G 52.33 0.85 164 osd.140
141 hdd 4.00000 1.00000 3719G 2137G 1581G 57.48 0.94 165 osd.141
142 hdd 4.00000 1.00000 3719G 2094G 1625G 56.31 0.92 155 osd.142
143 hdd 4.00000 1.00000 3719G 2322G 1397G 62.44 1.02 189 osd.143
144 hdd 4.00000 1.00000 3719G 2151G 1567G 57.85 0.94 167 osd.144
145 hdd 4.00000 1.00000 3719G 2021G 1697G 54.36 0.89 154 osd.145
146 hdd 4.00000 1.00000 3719G 2172G 1546G 58.42 0.95 176 osd.146
147 hdd 4.00000 1.00000 3719G 2127G 1591G 57.22 0.93 157 osd.147
148 hdd 4.00000 1.00000 3719G 2449G 1269G 65.86 1.07 189 osd.148
149 hdd 4.00000 1.00000 3719G 1904G 1814G 51.22 0.83 158 osd.149
150 hdd 4.00000 1.00000 3719G 2260G 1458G 60.78 0.99 183 osd.150
151 hdd 4.00000 1.00000 3719G 2133G 1585G 57.36 0.93 162 osd.151
152 hdd 4.00000 1.00000 3719G 2260G 1459G 60.77 0.99 187 osd.152
153 hdd 4.00000 1.00000 3719G 2384G 1334G 64.12 1.05 187 osd.153
154 hdd 4.00000 1.00000 3719G 1990G 1728G 53.52 0.87 162 osd.154
155 hdd 4.00000 1.00000 3719G 1717G 2002G 46.17 0.75 151 osd.155
-18 32.00000 - 32514G 18834G 13679G 57.93 0.94 - host ceph-s-08-spin
11 hdd 2.00000 1.00000 1863G 1417G 445G 76.07 1.24 94 osd.11
41 hdd 1.79999 1.00000 1857G 1020G 836G 54.95 0.90 82 osd.41
44 hdd 1.79999 1.00000 1857G 1082G 774G 58.29 0.95 86 osd.44
54 hdd 3.00000 1.00000 2794G 1748G 1046G 62.55 1.02 145 osd.54
62 hdd 1.79999 1.00000 1857G 1100G 756G 59.28 0.97 90 osd.62
85 hdd 1.79999 1.00000 1857G 874G 982G 47.12 0.77 85 osd.85
156 hdd 1.79999 1.00000 1857G 804G 1052G 43.32 0.71 78 osd.156
158 hdd 1.79999 1.00000 1857G 1017G 839G 54.77 0.89 74 osd.158
160 hdd 1.79999 1.00000 1857G 1329G 527G 71.60 1.17 94 osd.160
161 hdd 1.79999 1.00000 1857G 1023G 833G 55.13 0.90 79 osd.161
162 hdd 1.79999 1.00000 1857G 1018G 838G 54.83 0.89 90 osd.162
163 hdd 1.79999 1.00000 1857G 1033G 824G 55.63 0.91 77 osd.163
164 hdd 1.79999 0.79999 1857G 1099G 757G 59.20 0.96 84 osd.164
165 hdd 1.79999 1.00000 1857G 1049G 807G 56.53 0.92 85 osd.165
166 hdd 1.79999 1.00000 1857G 907G 949G 48.86 0.80 73 osd.166
168 hdd 1.79999 1.00000 1857G 1256G 600G 67.68 1.10 95 osd.168
170 hdd 1.79999 0.89999 1857G 1051G 806G 56.60 0.92 80 osd.170
-19 72.00000 - 67036G 43237G 23799G 64.50 1.05 - host ceph-s-09-spin
84 hdd 3.00000 1.00000 2794G 1540G 1253G 55.13 0.90 121 osd.84
118 hdd 3.00000 1.00000 2788G 2085G 702G 74.80 1.22 143 osd.118
157 hdd 3.00000 1.00000 2794G 1700G 1093G 60.86 0.99 135 osd.157
159 hdd 3.00000 1.00000 2794G 1664G 1130G 59.56 0.97 138 osd.159
167 hdd 3.00000 1.00000 2794G 1910G 883G 68.37 1.11 147 osd.167
169 hdd 3.00000 1.00000 2788G 1657G 1130G 59.46 0.97 131 osd.169
174 hdd 3.00000 1.00000 2794G 1926G 867G 68.94 1.12 136 osd.174
175 hdd 3.00000 1.00000 2788G 1458G 1329G 52.30 0.85 130 osd.175
176 hdd 3.00000 1.00000 2788G 1681G 1106G 60.32 0.98 141 osd.176
177 hdd 3.00000 1.00000 2794G 1823G 970G 65.25 1.06 148 osd.177
178 hdd 3.00000 1.00000 2788G 1532G 1255G 54.96 0.90 131 osd.178
179 hdd 3.00000 1.00000 2794G 1752G 1042G 62.69 1.02 133 osd.179
180 hdd 3.00000 1.00000 2794G 1899G 894G 67.98 1.11 139 osd.180
181 hdd 3.00000 1.00000 2794G 1770G 1023G 63.36 1.03 148 osd.181
182 hdd 3.00000 1.00000 2794G 1912G 882G 68.43 1.12 140 osd.182
183 hdd 3.00000 1.00000 2794G 2007G 786G 71.85 1.17 155 osd.183
184 hdd 3.00000 1.00000 2794G 1723G 1070G 61.68 1.01 132 osd.184
185 hdd 3.00000 1.00000 2794G 1634G 1160G 58.48 0.95 115 osd.185
186 hdd 3.00000 1.00000 2794G 1955G 838G 69.98 1.14 146 osd.186
187 hdd 3.00000 0.95001 2794G 1883G 911G 67.39 1.10 126 osd.187
188 hdd 3.00000 1.00000 2794G 1717G 1076G 61.47 1.00 127 osd.188
189 hdd 3.00000 1.00000 2794G 1954G 840G 69.93 1.14 140 osd.189
190 hdd 3.00000 1.00000 2794G 1940G 854G 69.44 1.13 145 osd.190
191 hdd 3.00000 1.00000 2794G 2103G 690G 75.28 1.23 145 osd.191
TOTAL 495T 299T 195T 61.36
MIN/MAX VAR: 0.01/1.27 STDDEV: 11.66
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
