Re: Dirty data loss after cache disk error recovery

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



I call that a necro-bump... ;-)

Am Mi., 6. Sept. 2023 um 22:33 Uhr schrieb Eric Wheeler
> On Fri, 7 May 2021, Kai Krakow wrote:
> > > Adding a new "stop" error action IMHO doesn't make things better. When
> > > the cache device is disconnected, it is always risky that some caching
> > > data or meta data is not updated onto cache device. Permit the cache
> > > device to be re-attached to the backing device may introduce "silent
> > > data loss" which might be worse....  It was the reason why I didn't add
> > > new error action for the device failure handling patch set.
> >
> > But we are actually now seeing silent data loss: The system f'ed up
> > somehow, needed a hard reset, and after reboot the bcache device was
> > accessible in cache mode "none" (because they have been unregistered
> > before, and because udev just detected it and you can use bcache
> > without an attached cache in "none" mode), completely hiding the fact
> > that we lost dirty write-back data, it's even not quite obvious that
> > /dev/bcache0 now is detached, cache mode none, but accessible
> > nevertheless. To me, this is quite clearly "silent data loss",
> > especially since the unregister action threw the dirty data away.
> >
> > So this:
> >
> > > Permit the cache
> > > device to be re-attached to the backing device may introduce "silent
> > > data loss" which might be worse....
> >
> > is actually the situation we are facing currently: Device has been
> > unregistered, after reboot, udev detects it has clean backing device
> > without cache association, using cache mode none, and it is readable
> > and writable just fine: It essentially permitted access to the stale
> > backing device (tho, it didn't re-attach as you outlined, but that's
> > more or less the same situation).
> >
> > Maybe devices that become disassociated from a cache due to IO errors
> > but have dirty data should go to a caching mode "stale", and bcache
> > should refuse to access such devices or throw away their dirty data
> > until I decide to force them back online into the cache set or force
> > discard the dirty data. Then at least I would discover that something
> > went badly wrong. Otherwise, I may not detect that dirty data wasn't
> > written. In the best case, that makes my FS unmountable, in the worst
> > case, some file data is simply lost (aka silent data loss), besides
> > both situations are the worst-case scenario anyways.
> >
> > The whole situation probably comes from udev auto-registering bcache
> > backing devices again, and bcache has no record of why the device was
> > unregistered - it looks clean after such a situation.


> I think we hit this same issue from 2021. Here is that original thread from 2021:
> Kai, did you end up with a good patch for this? We are running a 5.15
> kernel with the many backported bcache commits that Coly suggested here:

I'm currently running 6.1 with bcache on mdraid1 and device-level
write caching disabled. I didn't see this ever occur again.

BUT: Between that time and now I eventually also replaced my faulty
RAM which had a few rare bit-flips.

> Based on the thread from Kai (from 2021), I think we need to restore from
> backup. While the root of the problem may be hardware related, bcache
> should be more gracefully than unplugging the cache.

Yes, it may be hardware-related and you should probably confirm your
RAM working properly.

Currently, I'm running with no bcache patches on LTS 6.1, only some
btrfs patches:

Especially the allocation-hint patches provide better speedups for
meta data than bcache could ever do. With these patches, you could
dedicate a small amount of two SSD partitions (on different drivers)
to a btrfs metadata raid1, and use the remainder of the SSDs as a
bcache mdraid1. Then just don't use writeback caching but writearound
or writethrough instead. Most btrfs performance issues come from slow
metadata which can be much better improved by allocator-hints than by

But as written above, I had bad RAM, and meanwhile upgraded to kernel
6.1, and had no issues since with bcache even on power loss.

> Coly, is there already a patch to prevent complete dirty cache loss?

This is probably still an issue. The cache attachment MUST NEVER EVER
automatically degrade to "none" which it did for my fail-cases I had
back then. I don't know if this has changed meanwhile. But because
bcache explicitly does not honor write-barriers from upstream writes
for its own writeback (which is okay because it guarantees to write
back all data anyways and give a consistent view to upstream FS -
well, unless it has to handle write errors), the backed filesystem is
guaranteed to be effed up in that case, and allowing it to mount and
write because bcache silently has fallen back to "none" will only make
the matter worse.

(HINT: I never used brbd personally, most of the following is
theoretical thinking without real-world experience)

I see that you're using drbd? Did it fail due to networking issues?
I'm pretty sure it should be robust in that case but maybe bcache
cannot handle the situation? Does brbd have a write log to replay
writes after network connection loss? It looks like it doesn't and
thus bcache exploded.

Anyways, since your backing device seems to be on drbd, using metadata
allocation hinting is probably no option. You could of course still
use drbd with bcache for metadata hinted partitions, and then use
writearound caching only for that. At least, in the fail-case, your
btrfs won't be destroyed. But your data chunks may have unreadable
files then. But it should be easy to select them and restore from
backup individually. Btrfs is very robust for that fail case: if
metadata is okay, data errors are properly detected and handled. If
you're not using btrfs, all of this doesn't apply ofc.

I'm not sure if write-back caching for drbd backing is a wise decision
anyways. drbd is slow for writes, that's part of the design (and no
writeback caching could fix that). I would not rely on
bcache-writeback to fix that for you because it is not prepared for
storage that may be temporarily not available, iow, it would freeze
and continue when drbd is available again. I think you should really
use writearound/writethrough so your FS can be sure data has been
written, replicated and persisted. In case of btrfs, you could still
split data and metadata as written above, and use writeback for data,
but reliable writes for metadata.

So concluding:

1. I'm now persisting metadata directly to disk with no intermediate
layers (no bcache, no md)

2. I'm using allocation-hinted data-only partitions with bcache
write-back, with bcache on mdraid1. If anything goes wrong, I have
file crc errors in btrfs files only, but the filesystem itself is
valid because no metadata is broken or lost. I have snapshots of
recently modified files. I have daily backups.

3. Your problem is that bcache can - by design - detect write errors
only when it's too late with no chance telling the filesystem. In that
case, writethrough/writearound is the correct choice.

4. Maybe bcache should know if backing is on storage that may be
temporarily unavailable and then freeze until the backing storage is
back online, similar to how iSCSI handles that. But otoh, maybe drbd
should freeze until the replicated storage is available again while
writing (from what I've read, it's designed to not do that but let
local storage get ahead of the replica, which is btw incompatible with
bcache-writeback assumptions). Or maybe using async mirroring can fix
this for you but then, the mirror will be compromised if a hardware
failure immediately follows a previous drbd network connection loss.
But, it may still be an issue with the local hardware (bit-flips etc)
because maybe just bcache internals broke - Coly may have a better
idea of that.

I think your main issue here is that bcache decouples writebarriers
from the underlying backing storage - and you should just not use
writeback, it is incompatible by design with how drbd works: your
replica will be broken when you need it.

> Here is our trace:
> [Sep 6 13:01] bcache: bch_cache_set_error() error on a3292185-39ff-4f67-bec7-0f738d3cc28a: spotted extent 829560:7447835265109722923 len 26330 -> [0:112365451 gen 48, 0:1163806048 gen 3: bad, length too big, disabling caching
> [  +0.001940] CPU: 12 PID: 2435752 Comm: kworker/12:0 Kdump: loaded Not tainted 5.15.0- #7
> [  +0.000548] block drbd8143: write: error=10 s=9205904s
> [  +0.000301] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.4 12/27/2021
> [  +0.000866] block drbd8143: Local IO failed in drbd_endio_write_sec_final.
> [  +0.000809] Workqueue: bcache bch_data_insert_keys
> [  +0.000833] block drbd8143: disk( UpToDate -> Inconsistent )
> [  +0.000826] Call Trace:
> [  +0.000875] block drbd8143: write: error=10 s=8394752s
> [  +0.000797]  <TASK>
> [  +0.000006]  dump_stack_lvl+0x57/0x7e
> [  +0.000791] block drbd8143: Local IO failed in drbd_endio_write_sec_final.
> [  +0.000755]  bch_extent_invalid.cold+0x9/0x10
> [  +0.000760] block drbd8143: write: error=10 s=8397840s
> [  +0.000759]  btree_mergesort+0x27e/0x36e
> [  +0.000005]  ? bch_cache_allocator_start+0x50/0x50
> [  +0.000009]  __btree_sort+0xa4/0x1e9
> [  +0.002085] block drbd8143: drbd_md_sync_page_io(,41943032s,WRITE) failed with error -5
> [  +0.000109]  bch_btree_sort_partial+0xbc/0x14d
> [  +0.000878] block drbd8143: meta data update failed!
> [  +0.000836]  bch_btree_init_next+0x39/0xb6
> [  +0.000004]  bch_btree_insert_node+0x26e/0x2d3
> [  +0.000877] block drbd8143: disk( Inconsistent -> Failed )
> [  +0.000863]  btree_insert_fn+0x20/0x48
> [  +0.000866] block drbd8143: Local IO failed in drbd_md_write. Detaching...
> [  +0.000864]  bch_btree_map_nodes_recurse+0x111/0x1a7
> [  +0.004270]  ? bch_btree_insert_check_key+0x1f0/0x1e1
> [  +0.000850]  __bch_btree_map_nodes+0x1e0/0x1fb
> [  +0.000858]  ? bch_btree_insert_check_key+0x1f0/0x1e1
> [  +0.000848]  bch_btree_insert+0x102/0x188
> [  +0.000844]  ? do_wait_intr_irq+0xb0/0xaf
> [  +0.000857]  bch_data_insert_keys+0x39/0xde
> [  +0.000845]  process_one_work+0x280/0x5cf
> [  +0.000858]  worker_thread+0x52/0x3bd
> [  +0.000851]  ? process_one_work.cold+0x52/0x51
> [  +0.000877]  kthread+0x13e/0x15b
> [  +0.000858]  ? set_kthread_struct+0x60/0x52
> [  +0.000855]  ret_from_fork+0x22/0x2d
> [  +0.000854]  </TASK>


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM Kernel]     [Linux Filesystem Development]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux