On 2018/4/20 12:29 AM, Eric Wheeler wrote: > On Thu, 19 Apr 2018, Coly Li wrote: > >> On 2018/4/19 8:12 AM, Eric Wheeler wrote: >>> On Tue, 17 Apr 2018, Coly Li wrote: >>> >>>> On 2018/4/17 6:45 AM, Eric Wheeler wrote: >>>>> On Sat, 14 Apr 2018, Coly Li wrote: >>>>> >>>>>> On 2018/4/14 6:46 AM, Eric Wheeler wrote: >>>>>>> Hello all, >>>>>>> >>>>>>> We are running bcache in 4.1.49 with both the cache and backing device >>>>>>> having 4k blocks. The disk stack is DRBD->dm-thin->bcache->[sdc->sdb] >>>>>>> Where sdc is the cache. >>>>>>> >>>>>>> Sometimes we get errors like the following: >>>>>>> >>>>>>> [432015.934869] block drbd8065: Began resync as SyncTarget (will sync 880 KB [220 bits set]). >>>>>>> [432015.949469] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15724561783, blk_rq=9 >>>>>>> [432015.950347] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=353041040, blk_rq=7 >>>>>>> [432015.951146] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering >>>>>>> [432015.952015] block drbd8065: read: error=-5 s=19281488s >>>>>>> [432015.952866] block drbd8065: Local IO failed in drbd_endio_read_sec_final. >>>>>>> [432015.953777] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=387084784, blk_rq=7 >>>>>>> [432015.954710] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering >>>>>>> [432015.959037] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15725385535, blk_rq=1 >>>>>>> [432015.959938] block drbd8065: read: error=-5 s=19391384s >>>>>>> [432015.960862] block drbd8065: Local IO failed in drbd_endio_read_sec_final. >>>>>>> >>>>>>> >>>>>>> Note that 15724561783 is not divisible by 8, thus it is unaligned to 4k >>>>>>> blocks. >>>>>>> >>>>>>> Does anyone know if the bcache code is enforcing correct alignment? >>>>>>> >>>>>>> Is there any way that bcache could introduce misalignment? >>>>>>> >>>>>>> We ran blockdev --getbsz and --getpbsz all the way down the stack and >>>>>>> everything reports 4k. >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> Do you use 4.1 stable tree, or with your extra patches ? It would be >>>>>> helpful if I may access your kernel tree. >>>>>> >>>>>> So far I cannot tell where the problem is, I just feel there might be >>>>>> some hidden issue triggered by 4KB sector size hard drive. Maybe adding >>>>>> a garden code to detect unaligned I/O request from bcache will be >>>>>> helpful to diagnose the root cause. >>>>> >>>>> Hi Coly, >>>>> >>>>> I just pushed the branch that we built for this system to bitbucket: >>>>> https://bitbucket.org/ewheelerinc/linux/branch/ewi-4.1.49-rpmbuild >>>>> >>>>> There are a few changes, but probably nothing that you haven't seen: >>>>> >>>>> 1. We use the BFQ scheduler (but the problem presents in CFQ also) >>>>> 2. dmthin fixes were backported from 4.2/4.3 to fix pool space issues on >>>>> rolling snapshots >>>>> 3. bcache and dmcrypt have my ioprio patches, but no dmcrypt in this bug >>>>> 4. ibrs patches from Oracle are included for Spectre mitigation >>>>> 5. We attempted to use Mauricio's patch to fix 4k alignment issues, which >>>>> shows in our tree as 4a595ccc, but it did not fix the issue. >>>>> 6. We updated the error strings in sd.c with our commits aa372d91 and >>>>> f603bf7e while troubleshooting this issue. >>>>> >>>>> Thank you for your help, let me know if there is anything else that you >>>>> need to help troubleshoot. I can produce this pretty reliably at the >>>>> moment. >>>> >>>> Eric, >>>> >>>> Is it possible to apply the attached patch ? This patch will dump call >>>> trace if the request to backing device is not 4KB aligned. Then we can >>>> see what happens indeed. >>> >>> >>> Thanks Coly. It produced this trace. Can you tell what it is doing? >>> >>> [ 81.426439] bcache: check_4k_alignment() PTR_OFFSET is not 4KB aligned: 4368280612101 >>> [ 81.426439] CPU: 4 PID: 37 Comm: kworker/4:0 Tainted: G W O 4.1.49-2.el7.x86_64 #1 >>> [ 81.426440] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.10 01/09/2014 >>> [ 81.426443] Workqueue: bcache bch_data_insert_keys [bcache] >>> [ 81.426444] 0000000000000286 c3f07f1af1e6f322 ffff88080ba276d8 ffffffff816ff534 >>> [ 81.426445] 0000000000000000 ffff8807f8d5d4e8 ffff88080ba27728 ffffffffa057bfa4 >>> [ 81.426445] ffff88080ba27718 00000000812a17ec 0000000000000086 ffff8807f8d5d4e8 >>> [ 81.426446] Call Trace: >>> [ 81.426447] [<ffffffff816ff534>] dump_stack+0x63/0x81 >>> [ 81.426450] [<ffffffffa057bfa4>] __bch_cut_front+0x1a4/0x1b0 [bcache] >>> [ 81.426454] [<ffffffffa0585cb3>] bch_extent_insert_fixup+0x593/0x680 [bcache] >>> [ 81.426458] [<ffffffffa057c347>] ? __bch_btree_iter_init+0x77/0xc0 [bcache] >>> [ 81.426461] [<ffffffffa057c474>] bch_btree_insert_key+0xc4/0x320 [bcache] >>> [ 81.426465] [<ffffffffa057ddae>] btree_insert_key+0x5e/0x120 [bcache] >>> [ 81.426468] [<ffffffffa057e6f2>] bch_btree_insert_keys+0x62/0x250 [bcache] >>> [ 81.426472] [<ffffffffa0581c4c>] bch_btree_insert_node+0x13c/0x3d0 [bcache] >>> [ 81.426476] [<ffffffffa0582e88>] btree_insert_fn+0x28/0x50 [bcache] >>> [ 81.426479] [<ffffffffa05804f4>] bch_btree_map_nodes_recurse+0x54/0x190 [bcache] >>> [ 81.426482] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] >>> [ 81.426483] [<ffffffff817051a6>] ? down_write+0x16/0x50 >>> [ 81.426487] [<ffffffffa0580278>] ? bch_btree_node_get+0x78/0x2a0 [bcache] >>> [ 81.426490] [<ffffffffa05805a5>] bch_btree_map_nodes_recurse+0x105/0x190 [bcache] >>> [ 81.426494] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] >>> [ 81.426497] [<ffffffffa05802d5>] ? bch_btree_node_get+0xd5/0x2a0 [bcache] >>> [ 81.426500] [<ffffffffa05805a5>] bch_btree_map_nodes_recurse+0x105/0x190 [bcache] >>> [ 81.426504] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] >>> [ 81.426507] [<ffffffffa058376c>] __bch_btree_map_nodes+0x11c/0x150 [bcache] >>> [ 81.426510] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] >>> [ 81.426514] [<ffffffffa0583894>] bch_btree_insert+0xf4/0x170 [bcache] >>> [ 81.426515] [<ffffffff810e7000>] ? prepare_to_wait_event+0xf0/0xf0 >>> [ 81.426519] [<ffffffffa058b2be>] bch_data_insert_keys+0x3e/0x160 [bcache] >>> [ 81.426520] [<ffffffff810bac60>] process_one_work+0x150/0x450 >>> [ 81.426521] [<ffffffff810bb8c2>] worker_thread+0x112/0x530 >>> [ 81.426522] [<ffffffff810bb7b0>] ? rescuer_thread+0x3e0/0x3e0 >>> [ 81.426523] [<ffffffff810c10d8>] kthread+0xd8/0xf0 >>> [ 81.426524] [<ffffffff810c1000>] ? kthread_create_on_node+0x1b0/0x1b0 >>> [ 81.426525] [<ffffffff817074d2>] ret_from_fork+0x42/0x70 >>> [ 81.426525] [<ffffffff810c1000>] ? kthread_create_on_node+0x1b0/0x1b0 >> >> Hi Eric, >> >> I will post another patch for your test. This patch only provides >> information of LBA offset on cache device, backing device LBA check is >> missing. For cache device, 4KB align is not hardware mandatory IMHO. >> >> Will post an update patch soon. > > Thank you Coly! > > Note that our cache and backing device both have 4k logical blocks. Aha, thanks for the hint. So it is good to check 4K alignment for both cache and cached devices :-) Coly Li -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
