On Tue, Nov 12, 2024 at 3:04 AM Xiao Ni <xni@xxxxxxxxxx> wrote: > > On Mon, Nov 11, 2024 at 9:56 PM Haris Iqbal <haris.iqbal@xxxxxxxxx> wrote: > > > > On Mon, Nov 11, 2024 at 2:39 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote: > > > > > > Hi, > > > > > > 在 2024/11/11 21:29, Haris Iqbal 写道: > > > > Hello, > > > > > > > > I gave both the patches a try, and here are my findings. > > > > > > > > > > Thanks for the test! > > > > > > > With the first patch by Yu, I did not see any hang or errors. I tried > > > > a number of bitmap chunk sizes, and ran fio for few hours, and there > > > > was no hang. > > > > > > This is good news! However, there is still a long road for my approch > > > to land, this requires a lot of other changes to work. > > > > > > > > With the second patch Xiao, I hit the following BUG_ON on the first > > > > minute of my fio run. > > Hi Haris > > What's the fio command do you use? I did some simple io and couldn't > trigger this. I use the following fio config, [global] description=Emulation of Storage Server Access Pattern bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4 fadvise_hint=0 rw=randrw:2 direct=1 random_distribution=zipf:1.2 time_based=1 runtime=600 ramp_time=1 ioengine=libaio iodepth=128 iodepth_batch_submit=128 iodepth_batch_complete_min=1 iodepth_batch_complete_max=128 numjobs=10 group_reporting [job1] filename=/dev/rnbd0 do_verify=1 > > Regards > Xiao > > > This is sad. :( > > > > > > > > [ 113.902982] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI > > > > [ 113.903315] CPU: 38 UID: 0 PID: 9767 Comm: kworker/38:3H Kdump: > > > > loaded Not tainted 6.11.5-storage > > > > #6.11.5-1+feature+v6.11+20241111.0643+cbe84cc3~deb12 > > > > [ 113.904120] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021 > > > > [ 113.904519] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core] > > > > [ 113.904888] RIP: 0010:__add_stripe_bio+0x23f/0x250 [raid456] > > > > > > Can you provide the addr2line of this? > > > > > > gdb raid456.ko > > > list *(__add_stripe_bio+0x23f) > > > > Sorry. I missed the first line while copying. > > > > [ 113.902680] kernel BUG at drivers/md/raid5.c:3525! > > [ 113.902982] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI > > [ 113.903315] CPU: 38 UID: 0 PID: 9767 Comm: kworker/38:3H Kdump: > > loaded Not tainted 6.11.5-storage > > #6.11.5-1+feature+v6.11+20241111.0643+cbe84cc3~deb12 > > [ 113.904120] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021 > > [ 113.904519] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core] > > [ 113.904888] RIP: 0010:__add_stripe_bio+0x23f/0x250 [raid456] > > [ 113.905232] Code: 29 ff ff ff 41 8b 84 24 80 01 00 00 83 c0 01 89 > > 45 54 f0 80 4d 49 02 e9 11 ff ff ff 45 85 c0 0f 84 4e fe ff ff e9 31 > > ff ff ff <0f > > [ 113.906352] RSP: 0018:ffffb5d30ed27aa0 EFLAGS: 00010006 > > [ 113.906661] RAX: ffff992cb9549818 RBX: 0000000000000000 RCX: 0000000000000001 > > [ 113.907086] RDX: ffff992c989c3158 RSI: ffff992c989c3a58 RDI: 0000000000000000 > > [ 113.907511] RBP: ffff991d19e923a0 R08: 0000000000000000 R09: 0000000000000160 > > [ 113.907936] R10: 0000000000000007 R11: ffffb5d30ed27b70 R12: ffff991d0854b800 > > [ 113.908361] R13: 0000000000000001 R14: ffff991d19e92718 R15: 0000000000000001 > > [ 113.908786] FS: 0000000000000000(0000) GS:ffff993c3fc80000(0000) > > knlGS:0000000000000000 > > [ 113.909267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 113.909609] CR2: 00007f21b85473d8 CR3: 000000145d82c001 CR4: 00000000001706f0 > > [ 113.910034] Call Trace: > > [ 113.910181] <TASK > > [ 113.910304] ? die+0x36/0x90 > > [ 113.910478] ? do_trap+0xdd/0x100 > > [ 113.910675] ? __add_stripe_bio+0x23f/0x250 [raid456] > > [ 113.910979] ? do_error_trap+0x65/0x80 > > [ 113.911200] ? __add_stripe_bio+0x23f/0x250 [raid456] > > [ 113.911503] ? exc_invalid_op+0x50/0x70 > > [ 113.911731] ? __add_stripe_bio+0x23f/0x250 [raid456] > > [ 113.912033] ? asm_exc_invalid_op+0x1a/0x20 > > [ 113.912283] ? __add_stripe_bio+0x23f/0x250 [raid456] > > [ 113.912586] raid5_make_request+0x35f/0x1210 [raid456] > > [ 113.912896] ? submit_bio_noacct+0x47/0x4c0 > > [ 113.913145] ? __pfx_woken_wake_function+0x10/0x10 > > [ 113.913430] ? bio_split_rw+0x143/0x290 > > [ 113.913659] md_handle_request+0x156/0x270 > > [ 113.913905] __submit_bio+0x15c/0x1f0 > > [ 113.914126] submit_bio_noacct_nocheck+0x19a/0x3c0 > > [ 113.914412] ? submit_bio_noacct+0x47/0x4c0 > > [ 113.914662] rnbd_srv_rdma_ev+0x501/0xf70 [rnbd_server] > > [ 113.914976] ? rtrs_post_recv_empty+0x5d/0x80 [rtrs_core] > > [ 113.930375] process_io_req+0x169/0x4e0 [rtrs_server] > > [ 113.945660] __ib_process_cq+0x7b/0x170 [ib_core] > > > > > > > > Thanks, > > > Kuai > > > > [ 113.905232] Code: 29 ff ff ff 41 8b 84 24 80 01 00 00 83 c0 01 89 > > > > 45 54 f0 80 4d 49 02 e9 11 ff ff ff 45 85 c0 0f 84 4e fe ff ff e9 31 > > > > ff ff ff <0f > > > > [ 113.906352] RSP: 0018:ffffb5d30ed27aa0 EFLAGS: 00010006 > > > > [ 113.906661] RAX: ffff992cb9549818 RBX: 0000000000000000 RCX: 0000000000000001 > > > > [ 113.907086] RDX: ffff992c989c3158 RSI: ffff992c989c3a58 RDI: 0000000000000000 > > > > [ 113.907511] RBP: ffff991d19e923a0 R08: 0000000000000000 R09: 0000000000000160 > > > > [ 113.907936] R10: 0000000000000007 R11: ffffb5d30ed27b70 R12: ffff991d0854b800 > > > > [ 113.908361] R13: 0000000000000001 R14: ffff991d19e92718 R15: 0000000000000001 > > > > [ 113.908786] FS: 0000000000000000(0000) GS:ffff993c3fc80000(0000) > > > > knlGS:0000000000000000 > > > > [ 113.909267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > [ 113.909609] CR2: 00007f21b85473d8 CR3: 000000145d82c001 CR4: 00000000001706f0 > > > > [ 113.910034] Call Trace: > > > > [ 113.910181] <TASK > > > > [ 113.910304] ? die+0x36/0x90 > > > > [ 113.910478] ? do_trap+0xdd/0x100 > > > > [ 113.910675] ? __add_stripe_bio+0x23f/0x250 [raid456] > > > > [ 113.910979] ? do_error_trap+0x65/0x80 > > > > [ 113.911200] ? __add_stripe_bio+0x23f/0x250 [raid456] > > > > [ 113.911503] ? exc_invalid_op+0x50/0x70 > > > > [ 113.911731] ? __add_stripe_bio+0x23f/0x250 [raid456] > > > > [ 113.912033] ? asm_exc_invalid_op+0x1a/0x20 > > > > [ 113.912283] ? __add_stripe_bio+0x23f/0x250 [raid456] > > > > [ 113.912586] raid5_make_request+0x35f/0x1210 [raid456] > > > > [ 113.912896] ? submit_bio_noacct+0x47/0x4c0 > > > > [ 113.913145] ? __pfx_woken_wake_function+0x10/0x10 > > > > [ 113.913430] ? bio_split_rw+0x143/0x290 > > > > [ 113.913659] md_handle_request+0x156/0x270 > > > > [ 113.913905] __submit_bio+0x15c/0x1f0 > > > > [ 113.914126] submit_bio_noacct_nocheck+0x19a/0x3c0 > > > > [ 113.914412] ? submit_bio_noacct+0x47/0x4c0 > > > > [ 113.914662] rnbd_srv_rdma_ev+0x501/0xf70 [rnbd_server] > > > > [ 113.914976] ? rtrs_post_recv_empty+0x5d/0x80 [rtrs_core] > > > > [ 113.930375] process_io_req+0x169/0x4e0 [rtrs_server] > > > > [ 113.945660] __ib_process_cq+0x7b/0x170 [ib_core] > > > > > >
