Re: osd crashing and rocksdb corruption

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks again for your reactivity and your advices. You saved our lives !

We reactivate recovery/backfilling/rebalancing and it starts the recovery. We now have to wait to see how it will evolve.

Last question : We noticed (a few days ago and it still occurs) that after ~1h the recovery was stopping (Recovery Throughput drop to 0). We could restart it by restarting the ceph-mgr.target on the active manager... for another ~1 hour ! It's strange because I cannot see any crash or relevant info in the logs ! Moreover the ceph crash command hangs and no way to get output. Maybe it's because of the huge number of failures on the osds ! Do you think that this two problems could be related to the osd crashing ? I will continue to investigate and maybe open a new different thread on this topic.
F.

Le 30/04/2020 à 10:57, Igor Fedotov a écrit :
I created the following ticket and PR to track/fix the issue with incomplete large writes when bluefs_buffereed_io=1.

https://tracker.ceph.com/issues/45337

https://github.com/ceph/ceph/pull/34836


But In fact setting bluefs_buffered_io to false is the mainstream for now, see https://github.com/ceph/ceph/pull/34224

Francois, you can proceed with OSD.21 &.49 I can reproduce the issue locally hence no much need in them now.

Still investigating what's happening with OSD.8...

As for reactivating recovery/backfill/rebalancing - I can say for sure whether it's safe or not.


Thanks,

Igor


On 4/30/2020 1:39 AM, Francois Legrand wrote:
Hello,
We set bluefs_buffered_io to false for the whole cluster except 2 osd (21 and 49) for which we decided to keep the value to true for future experiments/troubleshooting as you asked. We then restarted all the 25 downs osd and they started... except one (number 8) which still continue to crash with the same kind of errors. I tryed a fsck on this osd which ended by a success. I set the debug to 20 and recorded the logs. You will find the logs there if you want to have a look : https://we.tl/t-GDvvvi2Gmm Now we plan to reactivate the recovery, backfill and rebalancing if you think it's safe.
F.

Le 29/04/2020 à 16:45, Igor Fedotov a écrit :
So the crash seems to be caused by the same issue - big (and presumably incomplete) write and subsequent read failure.

I've managed to repro this locally. So bluefs_buffered_io seems to be a remedy for now.

But additionally I can observe multiple slow ops indications in this new log and I think they cause those big writes.

And I presume some RGW-triggered ops are in flight - bucket resharding or removal or something..

I've seen multiple reports about this stuff causing OSD slowdown, high memory utilization and  finally huge reads and/or writes from RocksDB.

Don't know how to deal with this at the moment...

Thanks,

Igor


On 4/29/2020 5:33 PM, Francois Legrand wrote:
Here are the logs of the newly crashed osd.
F.

Le 29/04/2020 à 16:21, Igor Fedotov a écrit :
Sounds interesting - could you please share the crash log for these new OSDs? They presumably suffer from another issue. At least that first crash is caused by something else.


"bluefs buffered io" can be injected on the fly but I expect it to help when OSD isn't starting up only.


On 4/29/2020 5:17 PM, Francois Legrand wrote:
Ok we will try that.
Indeed, restarting osd.5 triggered the falling down of two other osds in the cluster. Thus we will set bluefs buffered io = false for all osds and force bluefs buffered io = true for one of the downs osds. Is that modification needs to use injectargs or changing it in the configuration is enougth to have it applied on the fly ?
F.

Le 29/04/2020 à 15:56, Igor Fedotov a écrit :
That's bluefs buffered io = false which did the trick.

It modified write path and this presumably has fixed large write(s).

Trying to reproduce locally but please preserve at least one failing OSD (i.e. do not start it with the disabled buffered io) for future experiments/troubleshooting for a while if possible.


Thanks,

Igor



On 4/29/2020 4:50 PM, Francois Legrand wrote:
Hi,
It seems much better with theses options. The osd is now up since 10mn without crashing (before it was rebooting after ~1mn).
F.

Le 29/04/2020 à 15:16, Igor Fedotov a écrit :
Hi Francois,

I'll write a more thorough response a bit later.

Meanwhile could you please try OSD startup with the following settings now:

debug-bluefs abd debug-bdev = 20

bluefs sync write = false

bluefs buffered io = false


Thanks,

Igor


On 4/29/2020 3:35 PM, Francois Legrand wrote:
Hi Igor,
Here is what we did :
First, as other osd were falling down, we stopped all operations with
ceph osd set norecover
ceph osd set norebalance
ceph osd set nobackfill
ceph osd set pause

to avoid other crashs !

Then we moved to your recommandations (still testing on osd 5):
in /etc/ceph/ceph.conf we added:
[osd.5]
    debug bluefs = 20
    debug bdev = 20

We ran :
ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-5 -l /var/log/ceph/bluestore-tool-fsck-osd-5.log --log-level 20 > /var/log/ceph/bluestore-tool-fsck-osd-5.out 2>&1

it ended with fsck success

It seems that the default value for bluefs sync write is false (https://github.com/ceph/ceph/blob/v14.2.8/src/common/options.cc), thus
we changed /etc/ceph/ceph.conf to :
[osd.5]
    debug bluefs = 20
    debug bdev = 20
    bluefs sync write = true

and restarted the osd. It crashed !
We tryed to change explicitely bluefs sync write = false and restarted... same result !

The logs are here : https://we.tl/t-HMiFDu22XH

Moreover, we have a rados gateway pool with hundreds of 4GB files. Can this be the origin of the problem ?
Do you thing that removing this pool can help ?

Thanks again for your expertise.
F.


Le 28/04/2020 à 18:52, Igor Fedotov a écrit :
Short update - please treat bluefs_sync_write parameter instead of bdev-aio. Changing the latter isn't supported in fact.

On 4/28/2020 7:35 PM, Igor Fedotov wrote:
Francious,

here are some observations got from your log.

1) Rocksdb reports error on the following .sst file:

   -35> 2020-04-28 15:23:47.612 7f4856e82a80 -1 rocksdb: Corruption: Bad table magic number: expected 986351839
0377041911, found 12950032858166034944 in db/068269.sst

2) which relates to BlueFS ino 53361:

  -50> 2020-04-28 15:23:45.103 7f4856e82a80 10 bluefs open_for_read db/068269.sst (random)    -49> 2020-04-28 15:23:45.103 7f4856e82a80 10 bluefs open_for_read h 0x557914fb80b0 on file(ino 53361 size 0x c496f919 mtime 2020-04-28 15:23:39.827515 bdev 1 allocated c4970000 extents [1:0x383db280000~c4970000])

3) and failed read happens to the end (0xc496f8e4~35, last 0x35 bytes) of this huge(3+GB) file:

 -44> 2020-04-28 15:23:47.514 7f4856e82a80 10 bluefs _read_random h 0x557914fb80b0 0xc496f8e4~35 from file(in o 53361 size 0xc496f919 mtime 2020-04-28 15:23:39.827515 bdev 1 allocated c4970000 extents [1:0x383db280000~c49
70000])
   -43> 2020-04-28 15:23:47.514 7f4856e82a80 20 bluefs _read_random left 0x71c 0xc496f8e4~35    -42> 2020-04-28 15:23:47.514 7f4856e82a80 20 bluefs _read_random got 53

4) This .sst file was created from scratch shortly before with a single-shot 3+GB write:

 -88> 2020-04-28 15:23:35.661 7f4856e82a80 10 bluefs open_for_write db/068269.sst    -87> 2020-04-28 15:23:35.661 7f4856e82a80 20 bluefs open_for_write mapping db/068269.sst to bdev 1    -86> 2020-04-28 15:23:35.662 7f4856e82a80 10 bluefs open_for_write h 0x5579145e7a40 on file(ino 53361 size 0 x0 mtime 2020-04-28 15:23:35.663142 bdev 1 allocated 0 extents [])    -85> 2020-04-28 15:23:39.826 7f4856e82a80 10 bluefs _flush 0x5579145e7a40 0x0~c496f919 to file(ino 53361 siz e 0x0 mtime 2020-04-28 15:23:35.663142 bdev 1 allocated 0 extents [])

5) Presumably RocksDB creates this file in an attempt to recover/compact/process another existing file (ino 52405) which is pretty large as well

Please find multiple earlier reads, the last one:

   -92> 2020-04-28 15:23:29.857 7f4856e82a80 10 bluefs _read h 0x5579147286e0 0xc6788000~8000 from file(ino 524 05 size 0xc67888a0 mtime 2020-04-25 13:34:55.325699 bdev 0 allocated c6790000 extents [1:0x381c8220000~10000,1:

The rationales for binding these two files are pretty uncommon file sizes.


So you have 3+GB single-shot BlueFS write and immediate read from the end of the written extent which returns unexpected magic.

It's well known in the software world that large (2+GB) data processing implementations tend to be error-prone. And Ceph is not an exception. Here is a couple of recent examples which are pretty close to your case:

https://github.com/ceph/ceph/commit/4d33114a40d5ae0d541c36175977ca22789a3b88

https://github.com/ceph/ceph/commit/207806abaa91259d9bb726c2555e7e21ac541527

Although they are already fixed in Nautilus 14.2.8 there might be others present along the write path (including H/W firmware).

The good news are that failure happens on a newly written file (remember invalid magic is read at the end(!) of this new large file shortly(!) after the write). So looks like data hasn't yet landed to a disk. And hopefully original data are untouched.

You can try to run ceph-bluestore-tool's fsck command on the broken OSD. This will open DB in read-only and hopefully proves the idea that the issue is on the write path rather than already corrupted data.

Also you might want to set bdev-aio to false (or true if it's already at false) and try OSD startup once more - this will go via a bit different write path and may be provide a workaround.

Also please collect debug logs for OSD startup (with both current and updated bdev-aio parameter) and --debug-bdev/debug-bluefs set to 20. You can omit --debug-bluestore increase for now to reduce log size.


Thanks,

Igor


On 4/28/2020 5:16 PM, Francois Legrand wrote:
Here is the output of ceph-bluestore-tool bluefs-bdev-sizes
inferring bluefs devices from bluestore path
 slot 1 /var/lib/ceph/osd/ceph-5/block -> /dev/dm-17
1 : device size 0x746c0000000 : own 0x[37e1eb00000~4a82900000] = 0x4a82900000 : using 0x5bc780000(23 GiB)


the result of the debug-bluestore (and debug-bluefs) set to 20 for osd.5
is at the following address (28MB).

https://wetransfer.com/downloads/a193ab15ab5e2395fe2462c963507a7f20200428141355/5da2ebf0d33750a5fde85bf662cf0e6d20200428141415/55849f?utm_campaign=WT_email_tracking&utm_content=general&utm_medium=download_button&utm_source=notify_recipient_email

Thanks for your help.
F.

Le 28/04/2020 à 13:33, Igor Fedotov a écrit :
Hi Francois,


Could you please share OSD startup log with debug-bluestore (and debug-bluefs) set to 20.

Also please run ceph-bluestore-tool's bluefs-bdev-sizes command and share the output.

Thanks,

Igor


On 4/28/2020 12:55 AM, Francois Legrand wrote:
Hi all,

*** Short version ***
Is there a way to repair a rocksdb from errors "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ?


*** Long version ***
We operate a nautilus ceph cluster (with 100 disks of 8TB in 6 servers + 4 mons/mgr + 3 mds). We recently (Monday 20) upgraded from 14.2.7 to 14.2.8. This triggered a rebalancing of some data. Two days later (Wednesday 22) we had a very short power outage. Only one of the osd servers went down (and unfortunately died). This triggered a reconstruction of the losts osds. Operations went fine until Saturday 25 where some osds in the 5 remaining servers started to crash apparently with no reasons. We tryed to restart them, but they crashed again. We ended with 18 osd down (+ 16 in the dead server so 34 osd downs out of 100).
Looking at the logs we found for all the crashed osd :

-237> 2020-04-25 16:32:51.835 7f1f45527a80 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2729370997 in db/181355.sst offset 18446744073709551615 size 18446744073709551615

and

2020-04-25 16:05:47.251 7fcbd1e46a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db erroring opening db:

We also noticed that the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" was present few days before the crash.
We also have some osd with this error but still up.

We tryed to repair with :
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-3 repair
But no success (it ends with _open_db erroring opening db).

Thus does somebody have an idea to fix this or at least know if it's possible to repair and correct the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ! Thanks for your help (we are desperate because we will loose datas and are fighting to save something) !!!
F.

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux