Le Thu, 17 Oct 2019 18:21:24 +0300
Teodor Milkov <tm@xxxxxx> écrivait:
> Hello,
>
> I've tried using bcache with a large 6.4TB NVMe device, but found it
> takes long time to register after clean reboot -- around 10 minutes.
> That's even with idle machine reboot.
>
> Things look like this soon after reboot:
>
> root@node420:~# ps axuww |grep md12
> root 9768 88.1 0.0 2268 744 pts/0 D+ 16:20 0:25
> /lib/udev/bcache-register /dev/md12
>
>
> Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s
> %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
> nvme0n1 420.00 0.00 52.50 0.00 0.00 0.00
> 0.00 0.00 0.30 0.00 1.04 128.00 0.00 2.38 99.87
> nvme1n1 417.67 0.00 52.21 0.00 0.00 0.00
> 0.00 0.00 0.30 0.00 1.03 128.00 0.00 2.39 100.00
> md12 838.00 0.00 104.75 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 128.00 0.00 0.00 0.00
>
> As you can see nvme1n1, which is Micron 9200, is reading with the
> humble 52MB/s (417r/s), and that is very far bellow it's capabilities
> of 3500MB/s & 840K IOPS.
>
> At the same time it seems like the bcache-register process is
> saturating the CPU core it's running on, so maybe that's the
> bottleneck?
>
> Tested with kernels 4.9 and 4.19.
>
> 1. Is this current state of affairs -- i.e. this known/expected
> behaviour with such a large cache?
>
> 2. If this isn't expected -- any ideas how to debug or fix it?
>
> 3. What is max recommended cache size?
I can't say anything of much help, but a few months ago when I tried
adding a 800 GB NVMe drive as a cache, I had to drop it after a loooong
wait for initialisation, with the same problem as you had: very little
IO on the NVMe drive, one core at 100%, and initialisation actually (as
far as I can tell) never ends.
My bet is that there's some overflow/math error that prevents bcache to
work properly with large caches. That's a pity because new NVMe SSDs
are quite cheap.
--
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <eflorac@xxxxxxxxxxxxxx>
| +33 1 78 94 84 02
------------------------------------------------------------------------
Attachment:
pgpgvrJiuBaEu.pgp
Description: Signature digitale OpenPGP
