On Thu, 19 Apr 2018, Coly Li wrote: > On 2018/4/19 8:12 AM, Eric Wheeler wrote: > > On Tue, 17 Apr 2018, Coly Li wrote: > > > >> On 2018/4/17 6:45 AM, Eric Wheeler wrote: > >>> On Sat, 14 Apr 2018, Coly Li wrote: > >>> > >>>> On 2018/4/14 6:46 AM, Eric Wheeler wrote: > >>>>> Hello all, > >>>>> > >>>>> We are running bcache in 4.1.49 with both the cache and backing device > >>>>> having 4k blocks. The disk stack is DRBD->dm-thin->bcache->[sdc->sdb] > >>>>> Where sdc is the cache. > >>>>> > >>>>> Sometimes we get errors like the following: > >>>>> > >>>>> [432015.934869] block drbd8065: Began resync as SyncTarget (will sync 880 KB [220 bits set]). > >>>>> [432015.949469] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15724561783, blk_rq=9 > >>>>> [432015.950347] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=353041040, blk_rq=7 > >>>>> [432015.951146] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering > >>>>> [432015.952015] block drbd8065: read: error=-5 s=19281488s > >>>>> [432015.952866] block drbd8065: Local IO failed in drbd_endio_read_sec_final. > >>>>> [432015.953777] sd 0:0:0:2: [sdc] Unaligned block number requested: sector_size=4096, block=387084784, blk_rq=7 > >>>>> [432015.954710] bcache: bch_count_io_errors() dm-6: IO error on reading from cache, recovering > >>>>> [432015.959037] sd 0:0:0:1: [sdb] Unaligned block number requested: sector_size=4096, block=15725385535, blk_rq=1 > >>>>> [432015.959938] block drbd8065: read: error=-5 s=19391384s > >>>>> [432015.960862] block drbd8065: Local IO failed in drbd_endio_read_sec_final. > >>>>> > >>>>> > >>>>> Note that 15724561783 is not divisible by 8, thus it is unaligned to 4k > >>>>> blocks. > >>>>> > >>>>> Does anyone know if the bcache code is enforcing correct alignment? > >>>>> > >>>>> Is there any way that bcache could introduce misalignment? > >>>>> > >>>>> We ran blockdev --getbsz and --getpbsz all the way down the stack and > >>>>> everything reports 4k. > >>>> > >>>> Hi Eric, > >>>> > >>>> Do you use 4.1 stable tree, or with your extra patches ? It would be > >>>> helpful if I may access your kernel tree. > >>>> > >>>> So far I cannot tell where the problem is, I just feel there might be > >>>> some hidden issue triggered by 4KB sector size hard drive. Maybe adding > >>>> a garden code to detect unaligned I/O request from bcache will be > >>>> helpful to diagnose the root cause. > >>> > >>> Hi Coly, > >>> > >>> I just pushed the branch that we built for this system to bitbucket: > >>> https://bitbucket.org/ewheelerinc/linux/branch/ewi-4.1.49-rpmbuild > >>> > >>> There are a few changes, but probably nothing that you haven't seen: > >>> > >>> 1. We use the BFQ scheduler (but the problem presents in CFQ also) > >>> 2. dmthin fixes were backported from 4.2/4.3 to fix pool space issues on > >>> rolling snapshots > >>> 3. bcache and dmcrypt have my ioprio patches, but no dmcrypt in this bug > >>> 4. ibrs patches from Oracle are included for Spectre mitigation > >>> 5. We attempted to use Mauricio's patch to fix 4k alignment issues, which > >>> shows in our tree as 4a595ccc, but it did not fix the issue. > >>> 6. We updated the error strings in sd.c with our commits aa372d91 and > >>> f603bf7e while troubleshooting this issue. > >>> > >>> Thank you for your help, let me know if there is anything else that you > >>> need to help troubleshoot. I can produce this pretty reliably at the > >>> moment. > >> > >> Eric, > >> > >> Is it possible to apply the attached patch ? This patch will dump call > >> trace if the request to backing device is not 4KB aligned. Then we can > >> see what happens indeed. > > > > > > Thanks Coly. It produced this trace. Can you tell what it is doing? > > > > [ 81.426439] bcache: check_4k_alignment() PTR_OFFSET is not 4KB aligned: 4368280612101 > > [ 81.426439] CPU: 4 PID: 37 Comm: kworker/4:0 Tainted: G W O 4.1.49-2.el7.x86_64 #1 > > [ 81.426440] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.10 01/09/2014 > > [ 81.426443] Workqueue: bcache bch_data_insert_keys [bcache] > > [ 81.426444] 0000000000000286 c3f07f1af1e6f322 ffff88080ba276d8 ffffffff816ff534 > > [ 81.426445] 0000000000000000 ffff8807f8d5d4e8 ffff88080ba27728 ffffffffa057bfa4 > > [ 81.426445] ffff88080ba27718 00000000812a17ec 0000000000000086 ffff8807f8d5d4e8 > > [ 81.426446] Call Trace: > > [ 81.426447] [<ffffffff816ff534>] dump_stack+0x63/0x81 > > [ 81.426450] [<ffffffffa057bfa4>] __bch_cut_front+0x1a4/0x1b0 [bcache] > > [ 81.426454] [<ffffffffa0585cb3>] bch_extent_insert_fixup+0x593/0x680 [bcache] > > [ 81.426458] [<ffffffffa057c347>] ? __bch_btree_iter_init+0x77/0xc0 [bcache] > > [ 81.426461] [<ffffffffa057c474>] bch_btree_insert_key+0xc4/0x320 [bcache] > > [ 81.426465] [<ffffffffa057ddae>] btree_insert_key+0x5e/0x120 [bcache] > > [ 81.426468] [<ffffffffa057e6f2>] bch_btree_insert_keys+0x62/0x250 [bcache] > > [ 81.426472] [<ffffffffa0581c4c>] bch_btree_insert_node+0x13c/0x3d0 [bcache] > > [ 81.426476] [<ffffffffa0582e88>] btree_insert_fn+0x28/0x50 [bcache] > > [ 81.426479] [<ffffffffa05804f4>] bch_btree_map_nodes_recurse+0x54/0x190 [bcache] > > [ 81.426482] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] > > [ 81.426483] [<ffffffff817051a6>] ? down_write+0x16/0x50 > > [ 81.426487] [<ffffffffa0580278>] ? bch_btree_node_get+0x78/0x2a0 [bcache] > > [ 81.426490] [<ffffffffa05805a5>] bch_btree_map_nodes_recurse+0x105/0x190 [bcache] > > [ 81.426494] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] > > [ 81.426497] [<ffffffffa05802d5>] ? bch_btree_node_get+0xd5/0x2a0 [bcache] > > [ 81.426500] [<ffffffffa05805a5>] bch_btree_map_nodes_recurse+0x105/0x190 [bcache] > > [ 81.426504] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] > > [ 81.426507] [<ffffffffa058376c>] __bch_btree_map_nodes+0x11c/0x150 [bcache] > > [ 81.426510] [<ffffffffa0582e60>] ? bch_btree_insert_check_key+0x1c0/0x1c0 [bcache] > > [ 81.426514] [<ffffffffa0583894>] bch_btree_insert+0xf4/0x170 [bcache] > > [ 81.426515] [<ffffffff810e7000>] ? prepare_to_wait_event+0xf0/0xf0 > > [ 81.426519] [<ffffffffa058b2be>] bch_data_insert_keys+0x3e/0x160 [bcache] > > [ 81.426520] [<ffffffff810bac60>] process_one_work+0x150/0x450 > > [ 81.426521] [<ffffffff810bb8c2>] worker_thread+0x112/0x530 > > [ 81.426522] [<ffffffff810bb7b0>] ? rescuer_thread+0x3e0/0x3e0 > > [ 81.426523] [<ffffffff810c10d8>] kthread+0xd8/0xf0 > > [ 81.426524] [<ffffffff810c1000>] ? kthread_create_on_node+0x1b0/0x1b0 > > [ 81.426525] [<ffffffff817074d2>] ret_from_fork+0x42/0x70 > > [ 81.426525] [<ffffffff810c1000>] ? kthread_create_on_node+0x1b0/0x1b0 > > Hi Eric, > > I will post another patch for your test. This patch only provides > information of LBA offset on cache device, backing device LBA check is > missing. For cache device, 4KB align is not hardware mandatory IMHO. > > Will post an update patch soon. Thank you Coly! Note that our cache and backing device both have 4k logical blocks. -- Eric Wheeler > > Thanks. > > Coly Li > -- > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
