make-bcache -B /dev/bcache0 fails, Can't stack bcache on top of bcache

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



­TLDR: I can't stack bcache on top of bcache, This command fails:
# make-bcache -B /dev/bcache0
Name			/dev/bcache0
Label			
Type			data
UUID:			758be840-f232-44dd-8e56-83fb55ba20ea
Set UUID:		ddbcc87b-d25d-483f-8874-49b80ba26472
version:		1
block_size_in_sectors:	1
data_offset_in_sectors:	16
[ 5187.054662] sysfs: cannot create duplicate filename '/devices/virtual/block/bcache0/bcache'
[ 5187.054667] CPU: 2 PID: 567 Comm: bcache-register Not tainted 5.9.14-arch1-1 #1
[ 5187.054669] Hardware name: Hewlett-Packard HP Compaq 8200 Elite CMT PC/1494, BIOS J01 v02.28 03/24/2015
[ 5187.054670] Call Trace:
[ 5187.054681]  dump_stack+0x6b/0x83
[ 5187.054686]  sysfs_warn_dup.cold+0x17/0x24
[ 5187.054694]  sysfs_create_dir_ns+0xc6/0xe0
[ 5187.054699]  kobject_add_internal+0xab/0x2f0
[ 5187.054703]  kobject_add+0x98/0xd0
[ 5187.054707]  ? blk_queue_write_cache+0x2f/0x60
[ 5187.054723]  register_bdev+0x337/0x360 [bcache]
[ 5187.054735]  register_bcache+0x43c/0x910 [bcache]
[ 5187.054740]  ? kernfs_fop_write+0xce/0x1b0
[ 5187.054750]  ? register_cache+0x1290/0x1290 [bcache]
[ 5187.054753]  kernfs_fop_write+0xce/0x1b0
[ 5187.054756]  vfs_write+0xc7/0x210
[ 5187.054759]  ksys_write+0x67/0xe0
[ 5187.054763]  do_syscall_64+0x33/0x40
[ 5187.054766]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5187.054769] RIP: 0033:0x7fbd1edb3f67
[ 5187.054772] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 5187.054774] RSP: 002b:00007fff335fa418 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 5187.054777] RAX: ffffffffffffffda RBX: 000000000000000d RCX: 00007fbd1edb3f67
[ 5187.054778] RDX: 000000000000000d RSI: 0000562bf3f902a0 RDI: 0000000000000003
[ 5187.054780] RBP: 0000562bf3f902a0 R08: 00007fbd1ee4a040 R09: 00007fbd1ee4a0c0
[ 5187.054781] R10: 00007fbd1ee49fc0 R11: 0000000000000246 R12: 000000000000000d
[ 5187.054782] R13: 00007fff335fa4a0 R14: 000000000000000d R15: 00007fbd1ee86720
[ 5187.054787] kobject_add_internal failed for bcache with -EEXIST, don't try to register things with the same name in the same directory.
[ 5187.054789] bcache: register_bdev() error bcache0: error creating kobject
[ 5187.054794] bcache: register_bcache() error : failed to register device
[ 5187.054963] bcache: bcache_device_free() bcache3 stopped

(very) long version:

I want to create a silent and fast NAS. I want to keep the hard drives idle for as long as possible. Therefore I need flash memory as write cache, so the hard drives stay idle when the user writes a file. The write caches rarely get flushed into the drives, so they are dirty for a long time. Therefore each drive needs it's own write cache device, so btrfs can recreate the lost data when a device fails. The read cache can be shared across all drives, as all the data in it also exists in the devices under it, so failure of the read cache does not result in data loss.

I'm testing this on a single drive with a lot of partitions. This kills performance (and fault tolerance). In production, each partition will be on a different drive.

+--------------------------------------------+
|         btrfs raid 1 (2 copies) /mnt       |
+--------------+--------------+--------------+
| /dev/bcache3 | /dev/bcache4 | /dev/bcache5 |
+--------------+--------------+--------------+
|              Read Cache (SSD)              |
|                /dev/sda4                   |
+--------------+--------------+--------------+
| /dev/bcache0 | /dev/bcache1 | /dev/bcache2 |
+--------------+--------------+--------------+
| Write Cache  | Write Cache  | Write Cache  |
|(Flash Drive) |(Flash Drive) |(Flash Drive) |
| /dev/sda5    | /dev/sda6    | /dev/sda7    |
+--------------+--------------+--------------+
| Data         | Data         | Data         |
| /dev/sda8    | /dev/sda9    | /dev/sda10   |
+--------------+--------------+--------------+
0. make the partitions on the drive:
# fdisk -l
/dev/sda1            2048   6293503   6291456     3G 83 Linux
/dev/sda2         6293504  14682111   8388608     4G 83 Linux
/dev/sda3        14682112  25167871  10485760     5G 83 Linux
/dev/sda4        25167872 976773167 951605296 453.8G  5 Extended
/dev/sda5        25169920  37752831  12582912     6G 83 Linux
/dev/sda6        37754880  52434943  14680064     7G 83 Linux
/dev/sda7        52436992  69214207  16777216     8G 83 Linux
/dev/sda8        69216256  88090623  18874368     9G 83 Linux
/dev/sda9        88092672 109064191  20971520    10G 83 Linux
/dev/sda10      109066240 132134911  23068672    11G 83 Linux

Wipe the partitions:
[root@bcache-test cedric]# wipefs -a /dev/sda5
[root@bcache-test cedric]# wipefs -a /dev/sda6
[root@bcache-test cedric]# wipefs -a /dev/sda7
[root@bcache-test cedric]# wipefs -a /dev/sda8
[root@bcache-test cedric]# wipefs -a /dev/sda9
[root@bcache-test cedric]# wipefs -a /dev/sda10

Check for old bcache devices:
# ls /dev/bca*
ls: cannot access '/dev/bca*': No such file or directory

1. Format the backing devices:
root@bcache-test cedric]# make-bcache -B /dev/sda8
Name			/dev/sda8
Label			
Type			data
UUID:			3bae90e9-d2c5-4d62-96a8-07dc1cb1440d
Set UUID:		271e657c-2a3a-417e-9a8d-430129095a47
version:		1
block_size_in_sectors:	1
data_offset_in_sectors:	16
[ 4196.603418] bcache: register_bdev() registered backing device sda8

# make-bcache -B /dev/sda9
Name			/dev/sda9
Label			
Type			data
UUID:			28ce8fa5-6918-4637-b612-374cf75635c2
Set UUID:		c12faa51-6042-4b46-8448-ee4a5acd0e8e
version:		1
block_size_in_sectors:	1
data_offset_in_sectors:	16
[ 4269.009004] bcache: register_bdev() registered backing device sda9

[root@bcache-test cedric]# make-bcache -B /dev/sda10
Name			/dev/sda10
Label			
Type			data
UUID:			4bca4225-4143-41af-b0e2-2ca2700e4931
Set UUID:		5442ccbe-6f12-40df-a309-30f66191773f
version:		1
block_size_in_sectors:	1
data_offset_in_sectors:	16
[ 4285.151623] bcache: register_bdev() registered backing device sda10

Now 3 bcache devices are created:
[root@bcache-test cedric]# ls -l /dev/bca*
brw-rw---- 1 root disk 254,   0 Jan  2 20:46 /dev/bcache0
brw-rw---- 1 root disk 254, 128 Jan  2 20:47 /dev/bcache1
brw-rw---- 1 root disk 254, 256 Jan  2 20:47 /dev/bcache2

/dev/bcache:
total 0
drwxr-xr-x 2 root root 100 Jan  2 20:47 by-uuid

2. Format the cache devices:
[root@bcache-test cedric]# make-bcache -C /dev/sda5
Name			/dev/sda5
Label			
Type			cache
UUID:			6e7610b2-5f28-4699-aed0-ffd78c1db77d
Set UUID:		924e51c2-9b49-4802-b87f-b379cd9a06a6
version:		0
nbuckets:		12288
block_size_in_sectors:	1
bucket_size_in_sectors:	1024
nr_in_set:		1
nr_this_dev:		0
first_bucket:		1
[ 4505.224494] bcache: run_cache_set() invalidating existing data
[ 4505.265255] bcache: register_cache() registered cache device sda5

[root@bcache-test cedric]# make-bcache -C /dev/sda6
Name			/dev/sda6
Label			
Type			cache
UUID:			4ff65c49-e225-4975-aac1-1be7da22d434
Set UUID:		ab10dafe-e8a4-4aa9-89f7-1a0575659f06
version:		0
nbuckets:		14336
block_size_in_sectors:	1
bucket_size_in_sectors:	1024
nr_in_set:		1
nr_this_dev:		0
first_bucket:		1
[ 4558.676081] bcache: run_cache_set() invalidating existing data
[ 4558.724051] bcache: register_cache() registered cache device sda6

[root@bcache-test cedric]# make-bcache -C /dev/sda7
Name			/dev/sda7
Label			
Type			cache
UUID:			02d4fe5e-5c01-4c94-9e89-3a97b01118b2
Set UUID:		31350209-3c13-4e78-b782-2c16e9d52e6a
version:		0
nbuckets:		16384
block_size_in_sectors:	1
bucket_size_in_sectors:	1024
nr_in_set:		1
nr_this_dev:		0
first_bucket:		1
[ 4583.036101] bcache: run_cache_set() invalidating existing data
[ 4583.077035] bcache: register_cache() registered cache device sda7

Register the cache devices against the backing devices.
# echo 924e51c2-9b49-4802-b87f-b379cd9a06a6 > /sys/block/bcache0/bcache/attach
[ 5008.251480] bcache: bch_cached_dev_run() cached dev sda8 is running already
[ 5008.251500] bcache: bch_cached_dev_attach() Caching sda8 as bcache0 on set 924e51c2-9b49-4802-b87f-b379cd9a06a6
# echo ab10dafe-e8a4-4aa9-89f7-1a0575659f06 > /sys/block/bcache1/bcache/attach
[ 5037.562635] bcache: bch_cached_dev_run() cached dev sda9 is running already
[ 5037.562648] bcache: bch_cached_dev_attach() Caching sda9 as bcache1 on set ab10dafe-e8a4-4aa9-89f7-1a0575659f06
# echo 31350209-3c13-4e78-b782-2c16e9d52e6a > /sys/block/bcache2/bcache/attach
[ 5061.280096] bcache: bch_cached_dev_run() cached dev sda10 is running already
[ 5061.280110] bcache: bch_cached_dev_attach() Caching sda10 as bcache2 on set 31350209-3c13-4e78-b782-2c16e9d52e6a
 
Format /dev/bcache0 as backing devices for the read cache fails:
make-bcache -B /dev/bcache0
# make-bcache -B /dev/bcache0
Name			/dev/bcache0
Label			
Type			data
UUID:			758be840-f232-44dd-8e56-83fb55ba20ea
Set UUID:		ddbcc87b-d25d-483f-8874-49b80ba26472
version:		1
block_size_in_sectors:	1
data_offset_in_sectors:	16
[ 5187.054662] sysfs: cannot create duplicate filename '/devices/virtual/block/bcache0/bcache'
[ 5187.054667] CPU: 2 PID: 567 Comm: bcache-register Not tainted 5.9.14-arch1-1 #1
[ 5187.054669] Hardware name: Hewlett-Packard HP Compaq 8200 Elite CMT PC/1494, BIOS J01 v02.28 03/24/2015
[ 5187.054670] Call Trace:
[ 5187.054681]  dump_stack+0x6b/0x83
[ 5187.054686]  sysfs_warn_dup.cold+0x17/0x24
[ 5187.054694]  sysfs_create_dir_ns+0xc6/0xe0
[ 5187.054699]  kobject_add_internal+0xab/0x2f0
[ 5187.054703]  kobject_add+0x98/0xd0
[ 5187.054707]  ? blk_queue_write_cache+0x2f/0x60
[ 5187.054723]  register_bdev+0x337/0x360 [bcache]
[ 5187.054735]  register_bcache+0x43c/0x910 [bcache]
[ 5187.054740]  ? kernfs_fop_write+0xce/0x1b0
[ 5187.054750]  ? register_cache+0x1290/0x1290 [bcache]
[ 5187.054753]  kernfs_fop_write+0xce/0x1b0
[ 5187.054756]  vfs_write+0xc7/0x210
[ 5187.054759]  ksys_write+0x67/0xe0
[ 5187.054763]  do_syscall_64+0x33/0x40
[ 5187.054766]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5187.054769] RIP: 0033:0x7fbd1edb3f67
[ 5187.054772] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 5187.054774] RSP: 002b:00007fff335fa418 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 5187.054777] RAX: ffffffffffffffda RBX: 000000000000000d RCX: 00007fbd1edb3f67
[ 5187.054778] RDX: 000000000000000d RSI: 0000562bf3f902a0 RDI: 0000000000000003
[ 5187.054780] RBP: 0000562bf3f902a0 R08: 00007fbd1ee4a040 R09: 00007fbd1ee4a0c0
[ 5187.054781] R10: 00007fbd1ee49fc0 R11: 0000000000000246 R12: 000000000000000d
[ 5187.054782] R13: 00007fff335fa4a0 R14: 000000000000000d R15: 00007fbd1ee86720
[ 5187.054787] kobject_add_internal failed for bcache with -EEXIST, don't try to register things with the same name in the same directory.
[ 5187.054789] bcache: register_bdev() error bcache0: error creating kobject
[ 5187.054794] bcache: register_bcache() error : failed to register device
[ 5187.054963] bcache: bcache_device_free() bcache3 stopped

I expected this command to create /dev/bcache3

My versions:
# uname -a
Linux bcache-test 5.9.14-arch1-1 #1 SMP PREEMPT Sat, 12 Dec 2020 14:37:12 +0000 x86_64 GNU/Linux
# pacman -Q bcache-tools
bcache-tools 1.1-1

I've used this guide for inspiration:
https://wiki.archlinux.org/index.php/Bcache



---

Take your mailboxes with you. Free, fast and secure Mail &amp; Cloud: https://www.eclipso.eu - Time to change!





[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM Kernel]     [Linux Filesystem Development]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux