On Wed, Nov 13, 2024 at 7:01 AM Xiao Ni <xni@xxxxxxxxxx> wrote:
>
> On Tue, Nov 12, 2024 at 5:46 PM Haris Iqbal <haris.iqbal@xxxxxxxxx> wrote:
> >
> > On Tue, Nov 12, 2024 at 3:04 AM Xiao Ni <xni@xxxxxxxxxx> wrote:
> > >
> > > On Mon, Nov 11, 2024 at 9:56 PM Haris Iqbal <haris.iqbal@xxxxxxxxx> wrote:
> > > >
> > > > On Mon, Nov 11, 2024 at 2:39 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > 在 2024/11/11 21:29, Haris Iqbal 写道:
> > > > > > Hello,
> > > > > >
> > > > > > I gave both the patches a try, and here are my findings.
> > > > > >
> > > > >
> > > > > Thanks for the test!
> > > > >
> > > > > > With the first patch by Yu, I did not see any hang or errors. I tried
> > > > > > a number of bitmap chunk sizes, and ran fio for few hours, and there
> > > > > > was no hang.
> > > > >
> > > > > This is good news! However, there is still a long road for my approch
> > > > > to land, this requires a lot of other changes to work.
> > > > > >
> > > > > > With the second patch Xiao, I hit the following BUG_ON on the first
> > > > > > minute of my fio run.
> > >
> > > Hi Haris
> > >
> > > What's the fio command do you use? I did some simple io and couldn't
> > > trigger this.
> >
> > I use the following fio config,
> >
> > [global]
> > description=Emulation of Storage Server Access Pattern
> > bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4
> > fadvise_hint=0
> > rw=randrw:2
> > direct=1
> > random_distribution=zipf:1.2
> > time_based=1
> > runtime=600
> > ramp_time=1
> > ioengine=libaio
> > iodepth=128
> > iodepth_batch_submit=128
> > iodepth_batch_complete_min=1
> > iodepth_batch_complete_max=128
> > numjobs=10
> > group_reporting
> >
> > [job1]
> > filename=/dev/rnbd0
> > do_verify=1
>
> I tried this pattern against the raid5 array directly with 6.11.0 and
> it can run successfully.
In my setup, I am running this over an RDMA block device.
The RDMA mapping is through the RNBD/RTRS kernel module, in which RTRS
provides the RDMA transport, and RNBD creates the virtual block device
layer on the client side.
The md400 device is mapped through RNBD/RTRS and on the client side
the above fio profile is ran on the /dev/rnbd0
$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md400 : active raid0 md300[0] md302[2] md301[1]
19688371968 blocks super 1.2 128k chunks
md302 : active raid5 sds[0] sdz[7] sdy[6] sdx[5] sdw[4] sdv[3] sdu[2] sdt[1]
6562922800 blocks super 1.2 level 5, 16k chunk, algorithm 2
[8/8] [UUUUUUUU]
bitmap: 0/1 pages [0KB], 524288KB chunk
md301 : active raid5 sdk[0] sdr[7] sdq[6] sdp[5] sdo[4] sdn[3] sdm[2] sdl[1]
6562922800 blocks super 1.2 level 5, 16k chunk, algorithm 2
[8/8] [UUUUUUUU]
bitmap: 0/1 pages [0KB], 524288KB chunk
md300 : active raid5 sda[0] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
6562922800 blocks super 1.2 level 5, 16k chunk, algorithm 2
[8/8] [UUUUUUUU]
bitmap: 0/1 pages [0KB], 524288KB chunk
>
> Regards
> Xiao
>
>
>
> >
> >
> > >
> > > Regards
> > > Xiao
> > > > > This is sad. :(
> > > > > >
> > > > > > [ 113.902982] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
> > > > > > [ 113.903315] CPU: 38 UID: 0 PID: 9767 Comm: kworker/38:3H Kdump:
> > > > > > loaded Not tainted 6.11.5-storage
> > > > > > #6.11.5-1+feature+v6.11+20241111.0643+cbe84cc3~deb12
> > > > > > [ 113.904120] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021
> > > > > > [ 113.904519] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
> > > > > > [ 113.904888] RIP: 0010:__add_stripe_bio+0x23f/0x250 [raid456]
> > > > >
> > > > > Can you provide the addr2line of this?
> > > > >
> > > > > gdb raid456.ko
> > > > > list *(__add_stripe_bio+0x23f)
> > > >
> > > > Sorry. I missed the first line while copying.
> > > >
> > > > [ 113.902680] kernel BUG at drivers/md/raid5.c:3525!
> > > > [ 113.902982] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
> > > > [ 113.903315] CPU: 38 UID: 0 PID: 9767 Comm: kworker/38:3H Kdump:
> > > > loaded Not tainted 6.11.5-storage
> > > > #6.11.5-1+feature+v6.11+20241111.0643+cbe84cc3~deb12
> > > > [ 113.904120] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021
> > > > [ 113.904519] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
> > > > [ 113.904888] RIP: 0010:__add_stripe_bio+0x23f/0x250 [raid456]
> > > > [ 113.905232] Code: 29 ff ff ff 41 8b 84 24 80 01 00 00 83 c0 01 89
> > > > 45 54 f0 80 4d 49 02 e9 11 ff ff ff 45 85 c0 0f 84 4e fe ff ff e9 31
> > > > ff ff ff <0f
> > > > [ 113.906352] RSP: 0018:ffffb5d30ed27aa0 EFLAGS: 00010006
> > > > [ 113.906661] RAX: ffff992cb9549818 RBX: 0000000000000000 RCX: 0000000000000001
> > > > [ 113.907086] RDX: ffff992c989c3158 RSI: ffff992c989c3a58 RDI: 0000000000000000
> > > > [ 113.907511] RBP: ffff991d19e923a0 R08: 0000000000000000 R09: 0000000000000160
> > > > [ 113.907936] R10: 0000000000000007 R11: ffffb5d30ed27b70 R12: ffff991d0854b800
> > > > [ 113.908361] R13: 0000000000000001 R14: ffff991d19e92718 R15: 0000000000000001
> > > > [ 113.908786] FS: 0000000000000000(0000) GS:ffff993c3fc80000(0000)
> > > > knlGS:0000000000000000
> > > > [ 113.909267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 113.909609] CR2: 00007f21b85473d8 CR3: 000000145d82c001 CR4: 00000000001706f0
> > > > [ 113.910034] Call Trace:
> > > > [ 113.910181] <TASK
> > > > [ 113.910304] ? die+0x36/0x90
> > > > [ 113.910478] ? do_trap+0xdd/0x100
> > > > [ 113.910675] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > [ 113.910979] ? do_error_trap+0x65/0x80
> > > > [ 113.911200] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > [ 113.911503] ? exc_invalid_op+0x50/0x70
> > > > [ 113.911731] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > [ 113.912033] ? asm_exc_invalid_op+0x1a/0x20
> > > > [ 113.912283] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > [ 113.912586] raid5_make_request+0x35f/0x1210 [raid456]
> > > > [ 113.912896] ? submit_bio_noacct+0x47/0x4c0
> > > > [ 113.913145] ? __pfx_woken_wake_function+0x10/0x10
> > > > [ 113.913430] ? bio_split_rw+0x143/0x290
> > > > [ 113.913659] md_handle_request+0x156/0x270
> > > > [ 113.913905] __submit_bio+0x15c/0x1f0
> > > > [ 113.914126] submit_bio_noacct_nocheck+0x19a/0x3c0
> > > > [ 113.914412] ? submit_bio_noacct+0x47/0x4c0
> > > > [ 113.914662] rnbd_srv_rdma_ev+0x501/0xf70 [rnbd_server]
> > > > [ 113.914976] ? rtrs_post_recv_empty+0x5d/0x80 [rtrs_core]
> > > > [ 113.930375] process_io_req+0x169/0x4e0 [rtrs_server]
> > > > [ 113.945660] __ib_process_cq+0x7b/0x170 [ib_core]
> > > >
> > > > >
> > > > > Thanks,
> > > > > Kuai
> > > > > > [ 113.905232] Code: 29 ff ff ff 41 8b 84 24 80 01 00 00 83 c0 01 89
> > > > > > 45 54 f0 80 4d 49 02 e9 11 ff ff ff 45 85 c0 0f 84 4e fe ff ff e9 31
> > > > > > ff ff ff <0f
> > > > > > [ 113.906352] RSP: 0018:ffffb5d30ed27aa0 EFLAGS: 00010006
> > > > > > [ 113.906661] RAX: ffff992cb9549818 RBX: 0000000000000000 RCX: 0000000000000001
> > > > > > [ 113.907086] RDX: ffff992c989c3158 RSI: ffff992c989c3a58 RDI: 0000000000000000
> > > > > > [ 113.907511] RBP: ffff991d19e923a0 R08: 0000000000000000 R09: 0000000000000160
> > > > > > [ 113.907936] R10: 0000000000000007 R11: ffffb5d30ed27b70 R12: ffff991d0854b800
> > > > > > [ 113.908361] R13: 0000000000000001 R14: ffff991d19e92718 R15: 0000000000000001
> > > > > > [ 113.908786] FS: 0000000000000000(0000) GS:ffff993c3fc80000(0000)
> > > > > > knlGS:0000000000000000
> > > > > > [ 113.909267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > [ 113.909609] CR2: 00007f21b85473d8 CR3: 000000145d82c001 CR4: 00000000001706f0
> > > > > > [ 113.910034] Call Trace:
> > > > > > [ 113.910181] <TASK
> > > > > > [ 113.910304] ? die+0x36/0x90
> > > > > > [ 113.910478] ? do_trap+0xdd/0x100
> > > > > > [ 113.910675] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > > > [ 113.910979] ? do_error_trap+0x65/0x80
> > > > > > [ 113.911200] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > > > [ 113.911503] ? exc_invalid_op+0x50/0x70
> > > > > > [ 113.911731] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > > > [ 113.912033] ? asm_exc_invalid_op+0x1a/0x20
> > > > > > [ 113.912283] ? __add_stripe_bio+0x23f/0x250 [raid456]
> > > > > > [ 113.912586] raid5_make_request+0x35f/0x1210 [raid456]
> > > > > > [ 113.912896] ? submit_bio_noacct+0x47/0x4c0
> > > > > > [ 113.913145] ? __pfx_woken_wake_function+0x10/0x10
> > > > > > [ 113.913430] ? bio_split_rw+0x143/0x290
> > > > > > [ 113.913659] md_handle_request+0x156/0x270
> > > > > > [ 113.913905] __submit_bio+0x15c/0x1f0
> > > > > > [ 113.914126] submit_bio_noacct_nocheck+0x19a/0x3c0
> > > > > > [ 113.914412] ? submit_bio_noacct+0x47/0x4c0
> > > > > > [ 113.914662] rnbd_srv_rdma_ev+0x501/0xf70 [rnbd_server]
> > > > > > [ 113.914976] ? rtrs_post_recv_empty+0x5d/0x80 [rtrs_core]
> > > > > > [ 113.930375] process_io_req+0x169/0x4e0 [rtrs_server]
> > > > > > [ 113.945660] __ib_process_cq+0x7b/0x170 [ib_core]
> > > > >
> > > >
> > >
> >
>
