On 12.02.20 10:33, Matthias Ferdinand wrote: > Sorry for being late to reply. Your bcache is set to "writeback", and > your test is doing reads from a bcache with a still empty cache device; > while the "1GB" file resides initially only on the backing device. > > My understanding so far was that in writeback mode only writes will ever > add data to the caching device (first as dirty, and post-writeback as > clean for as long as as space restrictions / usage patterns allow). > Therefore I would not have expected any piece of your "1GB" file to ever > be added to your caching device, and reading it would have to resort to > the backing device, even when repeatedly reading it. I would have said, that in writeback mode bcache acts as a read/write cache caching both reads and writes. Furthermore I also did the benchmark with all three caching modes and all behaved the same. As you can see, with kernel 5.6 I've got back the behavior like with kernels >= 4.18, the read is cached in writeback mode. But perhaps I'm wrong and this shouldn't happen? > > I have seen discussion about caching of readahead requests for newer > kernels, and I thought (but never really checked) this would only > concern read-caching modes. Am I wrong in this assumption? > > Matthias > > > On Wed, Feb 12, 2020 at 07:02:28AM +0100, Postgarage Graz IT wrote: >> On 10.02.20 17:10, Ville Aakko wrote: >>> Hi, >>> >>> A fellow user responding here. >>> >>> I've noticed similar behavior and have asked on this same mailing list >>> previously. See: >>> https://www.spinics.net/lists/linux-bcache/msg07859.html >>> >>> Also seems there are other users with this issue on the Arch Forum, >>> where I have also started a discussion: >>> https://bbs.archlinux.org/viewtopic.php?id=250525 >>> There is yet to be a single user to reply there (or on this mailing >>> list) claiming they have a working setup (for caching reads). >>> >>> Judging from the Arch Linux thread, I have a hunch there were some >>> changes ~4.18, which broke read caching for many (all?) desktop users >>> (as anything which is flagged as readahed will not be cached, despite >>> setting sequential_cutoff). Also (again from the Arch thread) a >>> planned patch might enable expected read caching: "[PATCH 3/5] bcache: >>> add readahead cache policy options via sysfs interface" / see: >>> https://www.spinics.net/lists/linux-bcache/msg08074.html >> >> Indeed that patch works. >> Now I'm using the 5.6-rc1 kernel and the performance gain is huge. >> >> I only wonder, why the used cache number doesn't go up anymore like it >> did for pre-4.19 kernels. >> >> >> Linux kkb 5.6.0-050600rc1-generic #202002092032 SMP Mon Feb 10 01:36:50 >> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux >> >> --- bcache --- >> Device /dev/md0 (9:0) >> UUID d4f0e4cd-c2dc-4cec-bf5b-96f1f87ff0b8 >> Block Size 0.50KiB >> Bucket Size 512.00KiB >> Congested? False >> Read Congestion 0.0ms >> Write Congestion 0.0ms >> Total Cache Size 111.79GiB >> Total Cache Used 1.12GiB (0%) >> Total Cache Unused 110.67GiB (99%) >> Evictable Cache 111.79GiB (100%) >> Replacement Policy [lru] fifo random >> Cache Mode writethrough [writeback] writearound none >> Total Hits 0 >> Total Misses 0 >> Total Bypass Hits 0 >> Total Bypass Misses 0 >> Total Bypassed 0B >> >> 1st pass >> 0.00user 0.60system 0:08.66elapsed 6%CPU (0avgtext+0avgdata >> 2220maxresident)k >> 2097632inputs+0outputs (1major+115minor)pagefaults 0swaps >> >> 2nd pass >> 0.00user 0.47system 0:03.34elapsed 14%CPU (0avgtext+0avgdata >> 2216maxresident)k >> 2097128inputs+0outputs (1major+113minor)pagefaults 0swaps >> >> 3rd pass >> 0.00user 0.45system 0:02.58elapsed 17%CPU (0avgtext+0avgdata >> 2096maxresident)k >> 2097296inputs+0outputs (1major+110minor)pagefaults 0swaps >> --- bcache --- >> Device /dev/md0 (9:0) >> UUID d4f0e4cd-c2dc-4cec-bf5b-96f1f87ff0b8 >> Block Size 0.50KiB >> Bucket Size 512.00KiB >> Congested? False >> Read Congestion 0.0ms >> Write Congestion 0.0ms >> Total Cache Size 111.79GiB >> Total Cache Used 1.12GiB (0%) >> Total Cache Unused 110.67GiB (99%) >> Evictable Cache 110.67GiB (99%) >> Replacement Policy [lru] fifo random >> Cache Mode writethrough [writeback] writearound none >> Total Hits 6352 (51%) >> Total Misses 6075 >> Total Bypass Hits 0 >> Total Bypass Misses 0 >> >> >> As you can see, the reads must come from the SSD in the 2nd and 3rd >> pass, still "Total Cache Used" stays the same. >> >> >>> >>> However this is highly speculative from someone not understanding file >>> systems or insides of bcache or the code at all. >>> >>> Perhaps someone more involved can reply: is the current behavior >>> expected (reads are not getting cached practically at all). Also, is >>> the patch I've linked possibly going to fix the current issues? >>> >>> Kind Regards, >>> Ville Aakko >>> >>> >>> >>> su 9. helmik. 2020 klo 23.37 Postgarage Graz IT (it@xxxxxxxxxxxxx) kirjoitti: >>>> >>>> Hello! >>>> >>>> I noticed, that bcache is no longer caching reads on my system which >>>> makes it behave like if there were only hdds. >>>> >>>> I'm using two hdds in a raid 1 as the backing device and a single ssd as >>>> cache device: >>>> >>>> sda 8:0 0 111,8G 0 disk >>>> └─bcache0 252:0 0 921,9G 0 disk / >>>> sdb 8:16 0 931,5G 0 disk >>>> ├─sdb1 8:17 0 922G 0 part >>>> │ └─md0 9:0 0 921,9G 0 raid1 >>>> │ └─bcache0 252:0 0 921,9G 0 disk / >>>> ├─sdb2 8:18 0 1K 0 part >>>> ├─sdb5 8:21 0 1,9G 0 part >>>> │ └─md1 9:1 0 1,9G 0 raid1 /boot >>>> └─sdb6 8:22 0 7,6G 0 part [SWAP] >>>> sdc 8:32 0 931,5G 0 disk >>>> ├─sdc1 8:33 0 922G 0 part >>>> │ └─md0 9:0 0 921,9G 0 raid1 >>>> │ └─bcache0 252:0 0 921,9G 0 disk / >>>> ├─sdc2 8:34 0 1K 0 part >>>> ├─sdc5 8:37 0 1,9G 0 part >>>> │ └─md1 9:1 0 1,9G 0 raid1 /boot >>>> └─sdc6 8:38 0 7,6G 0 part [SWAP] >>>> >>>> >>>> For benchmarking every time I detach the cache device, stop the bcache >>>> device, do a wipefs on the cache device, then make-bcache -C /dev/sda >>>> and finally reattach the cache. >>>> After that, I'm using the following script to repeatedly read a 1gb file: >>>> >>>> #!/bin/sh >>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff >>>> echo 0 > /sys/block/bcache0/bcache/cache/congested_read_threshold_us >>>> echo 0 > /sys/block/bcache0/bcache/cache/congested_write_threshold_us >>>> uname -a >>>> echo >>>> bcache-status >>>> echo >>>> echo "1st pass" >>>> sync; echo 3 > /proc/sys/vm/drop_caches >>>> (time cat 1GB.bin > /dev/null) >>>> echo >>>> echo "2nd pass" >>>> sync; echo 3 > /proc/sys/vm/drop_caches >>>> (time cat 1GB.bin > /dev/null) >>>> echo >>>> echo "3rd pass" >>>> sync; echo 3 > /proc/sys/vm/drop_caches >>>> (time cat 1GB.bin > /dev/null) >>>> bcache-status >>>> >>>> >>>> >>>> As you can see from the results below, kernel 4.18.20 is the last >>>> kernel, where the cache grows and the performance goes up. >>>> >>>> I also compiled 4.19.0 with the bcache files from 4.18.20 and much to my >>>> suprise, that didn't change 4.19's behavior - still no caching. So some >>>> other changes must be the culprit or I did something wrong. >>>> I'm not that much into compiling the kernel, but I checked out the >>>> 4.19.0 and 4.18.20 commits and replaced the 4.19.0 drivers/md/bcache >>>> directory with the one from 4.18.20 - then recompiled and installed the >>>> new kernel. >>>> >>>> So i am at my wits end. Any help would be appreciated. >>>> Thanks >>>> Flo >>>> >>>> >>>> Linux kkb 4.18.20-041820-generic #201812030624 SMP Mon Dec 3 11:25:55 >>>> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 8275bf01-f0b3-423e-87fa-48336ce33068 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 1.12GiB (0%) >>>> Total Cache Unused 110.67GiB (99%) >>>> Evictable Cache 111.79GiB (100%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 0 (0%) >>>> Total Misses 6 >>>> Total Bypass Hits 0 >>>> Total Bypass Misses 0 >>>> Total Bypassed 0B >>>> >>>> 1st pass >>>> 0.00user 0.36system 0:08.58elapsed 4%CPU (0avgtext+0avgdata >>>> 2196maxresident)k >>>> 2097608inputs+0outputs (1major+113minor)pagefaults 0swaps >>>> >>>> 2nd pass >>>> 0.00user 0.32system 0:03.29elapsed 9%CPU (0avgtext+0avgdata >>>> 2100maxresident)k >>>> 2097184inputs+0outputs (1major+110minor)pagefaults 0swaps >>>> >>>> 3rd pass >>>> 0.00user 0.32system 0:02.64elapsed 12%CPU (0avgtext+0avgdata >>>> 2092maxresident)k >>>> 2097280inputs+0outputs (1major+111minor)pagefaults 0swaps >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 8275bf01-f0b3-423e-87fa-48336ce33068 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 2.24GiB (2%) >>>> Total Cache Unused 109.55GiB (98%) >>>> Evictable Cache 110.67GiB (99%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 5 (0%) >>>> Total Misses 4079 >>>> Total Bypass Hits 0 (0%) >>>> Total Bypass Misses 615 >>>> Total Bypassed 2.40MiB >>>> >>>> >>>> >>>> Linux kkb 4.19.0-041900-generic #201810221809 SMP Mon Oct 22 22:11:45 >>>> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 67269654-92e8-4c3b-a524-8e8910082146 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 1.12GiB (0%) >>>> Total Cache Unused 110.67GiB (99%) >>>> Evictable Cache 111.79GiB (100%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 0 (0%) >>>> Total Misses 1 >>>> Total Bypass Hits 0 >>>> Total Bypass Misses 0 >>>> Total Bypassed 0B >>>> >>>> 1st pass >>>> 0.00user 0.33system 0:09.29elapsed 3%CPU (0avgtext+0avgdata >>>> 2280maxresident)k >>>> 2097624inputs+0outputs (1major+113minor)pagefaults 0swaps >>>> >>>> 2nd pass >>>> 0.00user 0.33system 0:08.47elapsed 4%CPU (0avgtext+0avgdata >>>> 2248maxresident)k >>>> 2097280inputs+0outputs (1major+111minor)pagefaults 0swaps >>>> >>>> 3rd pass >>>> 0.00user 0.37system 0:10.46elapsed 3%CPU (0avgtext+0avgdata >>>> 2220maxresident)k >>>> 2097616inputs+0outputs (1major+114minor)pagefaults 0swaps >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 67269654-92e8-4c3b-a524-8e8910082146 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 1.12GiB (0%) >>>> Total Cache Unused 110.67GiB (99%) >>>> Evictable Cache 111.79GiB (100%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 132 (23%) >>>> Total Misses 436 >>>> Total Bypass Hits 51 (0%) >>>> Total Bypass Misses 17399 >>>> Total Bypassed 43.50MiB >>>> >>>> >>>> >>>> >>>> Linux kkb 5.5.2-050502-generic #202002041931 SMP Tue Feb 4 19:33:15 UTC >>>> 2020 x86_64 x86_64 x86_64 GNU/Linux >>>> >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 38a8b675-e332-4076-b0cf-44e4be72c300 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 1.12GiB (0%) >>>> Total Cache Unused 110.67GiB (99%) >>>> Evictable Cache 111.79GiB (100%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 0 (0%) >>>> Total Misses 1 >>>> Total Bypass Hits 0 (0%) >>>> Total Bypass Misses 3 >>>> Total Bypassed 52.00KiB >>>> >>>> 1st pass >>>> 0.00user 0.42system 0:09.21elapsed 4%CPU (0avgtext+0avgdata >>>> 2216maxresident)k >>>> 2097608inputs+0outputs (1major+112minor)pagefaults 0swaps >>>> >>>> 2nd pass >>>> 0.00user 0.42system 0:09.62elapsed 4%CPU (0avgtext+0avgdata >>>> 2248maxresident)k >>>> 2097280inputs+0outputs (1major+112minor)pagefaults 0swaps >>>> >>>> 3rd pass >>>> 0.00user 0.43system 0:08.75elapsed 5%CPU (0avgtext+0avgdata >>>> 2220maxresident)k >>>> 2097224inputs+0outputs (1major+114minor)pagefaults 0swaps >>>> --- bcache --- >>>> Device /dev/md0 (9:0) >>>> UUID 38a8b675-e332-4076-b0cf-44e4be72c300 >>>> Block Size 0.50KiB >>>> Bucket Size 512.00KiB >>>> Congested? False >>>> Read Congestion 0.0ms >>>> Write Congestion 0.0ms >>>> Total Cache Size 111.79GiB >>>> Total Cache Used 1.12GiB (0%) >>>> Total Cache Unused 110.67GiB (99%) >>>> Evictable Cache 111.79GiB (100%) >>>> Replacement Policy [lru] fifo random >>>> Cache Mode writethrough [writeback] writearound none >>>> Total Hits 121 (32%) >>>> Total Misses 246 >>>> Total Bypass Hits 15 (0%) >>>> Total Bypass Misses 12811 >>>> Total Bypassed 39.70MiB >>> >>> >>> >>> -- >>> -- >>> Ville Aakko - ville.aakko@xxxxxxxxx
