The fullratio was ignored, that's why that happenned most likely. I
can't delete pgs, because it's only kb's worth of space - the osd is
40gb, 39.8 gb is taken up by omap - that's why i can't move/extract. Any
clue on how to compress/move away the omap dir?
On 27/08/18 12:34, Paul Emmerich wrote:
Don't ever let an OSD run 100% full, that's usually bad news.
Two ways to salvage this:
1. You can try to extract the PGs with ceph-objectstore-tool and
inject them into another OSD; Ceph will find them and recover
2. You seem to be using Filestore, so you should easily be able to
just delete a whole PG on the full OSD's file system to make space
(preferably one that is already recovered and active+clean even
without the dead OSD)
Paul
2018-08-27 10:44 GMT+02:00 Josef Zelenka <josef.zelenka@xxxxxxxxxxxxxxxx>:
Hi, i've had a very ugly thing happen to me over the weekend. Some of my
OSDs in a root that handles metadata pools overflowed to 100% disk usage due
to omap size(even though i had 97% full ratio, which is odd) and refused to
start. There were some pgs on those OSDs that went away with them. I have
tried compacting the omap, moving files away etc, but nothing - i can't
export the pgs, i get errors like this:
2018-08-27 04:42:33.436182 7fcb53382580 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1535359353436170, "job": 1, "event": "recovery_started",
"log_files": [5504, 5507]}
2018-08-27 04:42:33.436194 7fcb53382580 4 rocksdb:
[/build/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #5504
mode 2
2018-08-27 04:42:35.422502 7fcb53382580 4 rocksdb:
[/build/ceph-12.2.5/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all
background work
2018-08-27 04:42:35.431613 7fcb53382580 4 rocksdb:
[/build/ceph-12.2.5/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2018-08-27 04:42:35.431716 7fcb53382580 -1 rocksdb: IO error: No space left
on device/var/lib/ceph/osd/ceph-5//current/omap/005507.sst: No space left on
device
Mount failed with '(1) Operation not permitted'
2018-08-27 04:42:35.432945 7fcb53382580 -1
filestore(/var/lib/ceph/osd/ceph-5/) mount(1723): Error initializing rocksdb
:
I decided to take the loss and mark the osds as lost and remove them from
the cluster, however, it left 4 pgs hanging in incomplete + inactive state,
which apparently prevents my radosgw from starting. Is there another way to
export/import the pgs into their new osds/recreate them? I'm running
Luminous 12.2.5 on Ubuntu 16.04.
Thanks
Josef
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
