Re: bcacheX is missing after removing a backend and adding it again

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> On 2/25/21 11:12 AM, wubenqing@xxxxxxxxxxxxx wrote:
> > Hi guys,
> > I am testing a scenario where multiple backend attach one cache. When I
> removed one of the backend and added it back, I found that bcacheX was
> missing. I configured cache_mode to writeback.
> >
> > Before:
> > /dev/sdd
> > └─bcache0
> > /dev/sdc
> > └─bcache1
> >
> > After:
> > /dev/sdg
> > /dev/sdc
> > └─bcache1
> >
> >
> > The name of the block device /dev/sdd is changed to /dev/sdg, and
> bcache0 is missing when excuting lsblk. I found that
> /sys/bkock/bache0/bcache link to the old device which does not exist.
> > # ll /sys/block/bcache0/bcache
> > lrwxrwxrwx 1 root root 0 Feb 23 17:36 /sys/block/bcache0/bcache ->
> > ../../../pci0000:80/0000:80:01.0/0000:82:00.0/host1/port-1:3/end_devic
> > e-1:3/target1:0:3/1:0:3:0/block/sdd/bcache
> >
> >
> > The super block of /dev/sdg shows that there is still dirty data stored on
> the cache device.
> > # bcache-super-show /dev/sdg
> > sb.magicok
> > sb.first_sector8 [match]
> > sb.csum 80AE8CFCCC740075 [match]
> > sb.version1 [backing device]
> >
> > dev.label(empty)
> > dev.uuid22cb8e47-67d8-4f54-97b4-a8c86d986aac
> > dev.sectors_per_block 1
> > dev.sectors_per_bucket 1024
> > dev.data.first_sector 16
> > dev.data.cache_mode 1 [writeback]
> > dev.data.cache_state 2 [dirty]
> >
> >
> > When I checked the kernel log, I found that:
> > ...
> > [81701.447130] bcache: bch_count_backing_io_errors() sdd: IO error on
> > backing device, unrecoverable [81701.487543] bcache:
> > bch_count_backing_io_errors() sdd: IO error on backing device,
> > unrecoverable [81701.985562] bcache: bch_count_backing_io_errors()
> > sdd: IO error on backing device, unrecoverable [81702.590435] bcache:
> > backing_request_endio() Can't flush sdd: returned bi_status 10
> >
> > ...
> > [81849.890604] bcache: register_bdev() registered backing device sdg
> > [81849.890608] bcache: bch_cached_dev_attach() Tried to attach sdg but
> > duplicate UUID already attached
> >
> > ...
> >
> > "IO error on backing device, unrecoverable" appeared 63 times in total. It
> may be that the io_disable of the backend device is set to true due to
> io_error_limit is 64, but I did not find the log "too many IO errors on backing
> device" which bch_cached_dev_error will print.
> >
> > bch_writeback_thread is very high cpu usage and the SSD(cache) shows
> very high read traffic but no write traffic.
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s
> wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> > nvme0n1           0.00     0.00 33836.00    0.00  1619.07
> 0.00    98.00    76.55    2.68    2.68    0.00   0.03 100.00
> >
> >
> > The important problem is that there is no way to recover bcache0, even if I
> try to re-execute "echo /dev/sdg > /sys/fs/bcache/register". The kernel log
> shows that:
> > [91091.621773] bcache: register_bcache() error : device already
> > registered
> >
> > I suspect that /dev/sdd still remains in c->cached_devs, and it is set to
> io_disable, and /dev/sdd does not exist anymore, so writeback cannot flush
> dirty data. Since the name of the block device has become /dev/sdg,
> /dev/sdg cannot be reattached successfully.
> > Does bcache support backend for hot-swapping scenarios? If not, what
> command should I use to manually restore bcache0.
> 
> Which kernel version do you use ?
> 
> A reboot might solve the problem. But I feel it could be improved to avoid
> the extra reboot.
> 
> Let me add it into my todo list, if no one else posts patch before I work on
> it...
> 
> Thanks for the suggestion.
> 
> Coly Li

Thank you, your suggestion worked. When I restarted the machine, the problem disappeared. The kernel version is "5.4.90-1.el7.elrepo.x86_64". 
What is the reason why bch_writeback_thread occupies a high CPU, and always reads data from SSDs at a very high rate. This phenomenon continued for several hours until I restarted the machine. Please help to confirm whether it is a bug.

Another question is about the cache device. I also unplugged and plugged the cache device and found a problem. After the cache disk was back online, the backend did not automatically attach successfully. And manually using the command can only make one of the backend attach succeed, and the other one fails.

Before:
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdd                 8:48   0   9.1T  0 disk 
└─bcache0         252:0    0   9.1T  0 disk 
sdb                 8:16   0 931.5G  0 disk 
└─bcache1         252:128  0 931.5G  0 disk 
sdh                 8:112  0 447.1G  0 disk 
├─sdh1              8:113  0   200G  0 part 
│ ├─bcache0       252:0    0   9.1T  0 disk 
│ └─bcache1       252:128  0 931.5G  0 disk 
└─sdh2              8:114  0 247.1G  0 part
/dev/sdh is cache device.

After:
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdd                 8:48   0   9.1T  0 disk 
└─bcache0         252:0    0   9.1T  0 disk 
sdb                 8:16   0 931.5G  0 disk 
└─bcache1         252:128  0 931.5G  0 disk 
sdg                 8:96   0 447.1G  0 disk 
├─sdg1              8:97   0   200G  0 part 
└─sdg2              8:98   0 247.1G  0 part
/dev/sdg is cache device.

The kernel log is:
...
[82566.436157] blk_update_request: I/O error, dev sdh, sector 17868400 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[82566.436162] bcache: bch_count_io_errors() sdh1: IO error on reading from cache, recovering.
[82566.436171] blk_update_request: I/O error, dev sdh, sector 28192496 op 0x1:(WRITE) flags 0x0 phys_seg 4 prio class 0
[82566.436173] bcache: bch_count_io_errors() sdh1: IO error on writing data to cache.
[82566.436175] bcache: bch_count_io_errors() sdh1: IO error on writing data to cache.
[82566.436180] blk_update_request: I/O error, dev sdh, sector 29303936 op 0x1:(WRITE) flags 0x0 phys_seg 3 prio class 0
[82566.436182] bcache: bch_count_io_errors() sdh1: IO error on writing data to cache.
[82566.436186] blk_update_request: I/O error, dev sdh, sector 30462320 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[82566.436188] bcache: bch_count_io_errors() sdh1: IO error on writing data to cache.
[82566.436691] bcache: bch_count_io_errors() sdh1: IO error on writing data to cache.
[82566.438654] blk_update_request: I/O error, dev sdh, sector 28192528 op 0x1:(WRITE) flags 0x0 phys_seg 16 prio class 0
[82566.440356] bcache: bch_cache_set_error() bcache: error on f4e8b4d6-c354-45be-9510-725fe69c1b16: 
[82566.443227] bcache: bch_cache_set_error() CACHE_SET_IO_DISABLE already set
[82566.445156] sdh1: too many IO errors writing data to cache
[82566.445158] bcache: bch_cache_set_error() , disabling caching

[82566.445183] bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache0 is "auto" and cache is dirty, stop it to avoid potential data corruption.
[82566.448983] bcache: bch_cache_set_error() bcache: error on f4e8b4d6-c354-45be-9510-725fe69c1b16: 
[82566.450875] bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache1 is "auto" and cache is clean, keep it alive.
[82566.452653] sdh1: too many IO errors writing data to cache
[82566.452654] bcache: bch_cache_set_error() , disabling caching

[82566.454453] bcache: cached_dev_detach_finish() Caching disabled for sdb
[82566.459006] bcache: cached_dev_detach_finish() Caching disabled for sdd
[82566.499025] bcache: cache_set_free() Cache set f4e8b4d6-c354-45be-9510-725fe69c1b16 unregistered
...
[82866.099218] scsi 1:0:2:0: Direct-Access     ATA      INTEL SSDSC2KB48 0110 PQ: 0 ANSI: 6
[82866.099229] scsi 1:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221103000000), phy(3), device_name(0x55cd2e4152128da4)
[82866.099232] scsi 1:0:2:0: enclosure logical id (0x56c92bf0002b9105), slot(1) 
[82866.099234] scsi 1:0:2:0: enclosure level(0x0000), connector name(     )
[82866.099298] scsi 1:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[82866.100072] sd 1:0:2:0: Attached scsi generic sg3 type 0
[82866.101121] sd 1:0:2:0: [sdg] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[82866.101125] sd 1:0:2:0: [sdg] 4096-byte physical blocks
[82866.101366] sd 1:0:2:0: [sdg] Write Protect is off
[82866.101369] sd 1:0:2:0: [sdg] Mode Sense: 9b 00 10 08
[82866.101646] sd 1:0:2:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[82866.118234]  sdg: sdg1 sdg2
[82866.121101] sd 1:0:2:0: [sdg] Attached SCSI disk
[82866.432383] bcache: bch_journal_replay() journal replay done, 75408 keys in 376 entries, seq 4483
[82866.432655] bcache: register_cache() registered cache device sdg1
...

# bcache-super-show /dev/sdd
sb.magic		ok
sb.first_sector		8 [match]
sb.csum			59C6B401494EAB2F [match]
sb.version		1 [backing device]

dev.label		(empty)
dev.uuid		8307dc11-4dff-4286-a4c5-76a79f518b27
dev.sectors_per_block	1
dev.sectors_per_bucket	1024
dev.data.first_sector	16
dev.data.cache_mode	1 [writeback]
dev.data.cache_state	0 [detached]

cset.uuid		00000000-0000-0000-0000-000000000000
# bcache-super-show /dev/sdb
sb.magic		ok
sb.first_sector		8 [match]
sb.csum			4BDD5D546EA35D9E [match]
sb.version		1 [backing device]

dev.label		(empty)
dev.uuid		0996b86e-a467-4dda-9fa3-8f2f3da8567a
dev.sectors_per_block	1
dev.sectors_per_bucket	1024
dev.data.first_sector	16
dev.data.cache_mode	1 [writeback]
dev.data.cache_state	0 [detached]

cset.uuid		00000000-0000-0000-0000-000000000000

# bcache-super-show /dev/sdg1
sb.magic		ok
sb.first_sector		8 [match]
sb.csum			9D67FC82E9F8952A [match]
sb.version		3 [cache device]

dev.label		(empty)
dev.uuid		021c2064-ca55-4228-97e1-e95af4dd9113
dev.sectors_per_block	1
dev.sectors_per_bucket	1024
dev.cache.first_sector	1024
dev.cache.cache_sectors	419429376
dev.cache.total_sectors	419430400
dev.cache.ordered	yes
dev.cache.discard	no
dev.cache.pos		0
dev.cache.replacement	0 [lru]

cset.uuid		f4e8b4d6-c354-45be-9510-725fe69c1b16

bcache1 failed. I find that /sys/block/sdd/bcache disappeared, and I do not know why.
# echo f4e8b4d6-c354-45be-9510-725fe69c1b16 > /sys/block/bcache0/bcache/attach
-bash: /sys/block/bcache0/bcache/attach: No such file or directory

I tried to manually register the backend and it failed. How can I restore the backend's bcache directory, if I don't restart the machine.
# echo /dev/sdd > /sys/fs/bcache/register
-bash: echo: write error: Invalid argument
[132767.174638] bcache: register_bcache() error : device already registered

bcache0 Succeed.
# echo f4e8b4d6-c354-45be-9510-725fe69c1b16 > /sys/block/bcache1/bcache/attach




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM Kernel]     [Linux Filesystem Development]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux