Hi Mark,
I doubt read-only mode would help here.
Log replay is required to build a consistent store state and one can't
bypass it. And looks like your drive/controller still detect some errors
while reading.
For the second issue this PR might help (you'll be able to disable csum
verification and hopefully OSD will start) once it's merged
(https://github.com/ceph/ceph/pull/26247). For now IMO the only way to
go for you is to have a custom build with this patch manually applied.
Thanks,
Igor
On 7/9/2019 5:02 PM, Mark Lehrer wrote:
My main question is this - is there a way to stop any replay or
journaling during OSD startup and bring up the pool/fs in read-only
mode?
Here is a description of what I'm seeing. I have a Luminous cluster
with CephFS and 16 8TB SSDs, using size=3.
I had a problem with one of my SAS controllers, and now I have at
least 3 OSDs that refuse to start. The hardware appears to be fine
now.
I have my essential data backed up, but there are a few files that I
wouldn't mind saving so I want to use this as disaster recovery
practice.
The two problems I am seeing are:
1) On two of OSDs, there is a startup replay error after successfully
replaying quite a few blocks:
2019-07-06 16:08:05.281063 7f6baec66e40 10 bluefs _replay 0x1543000:
stop: uuid c366a2d6-e221-98b3-59fe-0f324c9dac8e != super.uuid
263428d5-8963-4339-8815-92ab6067e7a4
2019-07-06 16:08:05.281064 7f6baec66e40 10 bluefs _replay log file
size was 0x1543000
2019-07-06 16:08:05.281085 7f6baec66e40 -1 bluefs _replay file with
link count 0: file(ino 1485 size 0x15f4c43 mtime 2019-07-04
20:39:39.387601 bdev 1 allocated 1600000 extents
[1:0x35771500000+100000,1:0x35771600000+100000,1:0x35771700000+100000,1:0x35771c00000+100000,1:0x35771d00000+100000,1:0x35772200000+100000,1:0x35772300000+100000,1:0x35772800000+100000,1:0x35772900000+100000,1:0x35772a00000+100000,1:0x35772b00000+100000,1:0x35772c00000+100000,1:0x35772d00000+100000,1:0x35772e00000+100000,1:0x35773300000+100000,1:0x35773400000+100000,1:0x35773500000+100000,1:0x35773600000+100000,1:0x35773700000+100000,1:0x35773800000+100000,1:0x35773900000+100000,1:0x35773a00000+100000])
2019-07-06 16:08:05.281093 7f6baec66e40 -1 bluefs mount failed to
replay log: (5) Input/output error
2) The following error happens on at least two OSDs:
2019-07-06 15:58:46.621008 7fdcee030e40 -1
bluestore(/var/lib/ceph/osd/ceph-74) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0x147db0c5, expected 0x8f052c9,
device location [0x10000~1000], logical extent 0x0~1000, object
#-1:7b3f43c4:::osd_superblock:0#
The system was archiving some unimportant files when it went down, so
I really don't care about any of the recent writes.
What are my recovery options here? I was thinking that turning off
replaying and running in read-only mode would be feasible, but maybe
there are better options?
Thanks,
Mark
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
