Hi,
在 2024/11/14 0:46, Haris Iqbal 写道:
On Wed, Nov 13, 2024 at 8:46 AM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
Hi,
在 2024/11/11 21:56, Haris Iqbal 写道:
On Mon, Nov 11, 2024 at 2:39 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
Hi,
在 2024/11/11 21:29, Haris Iqbal 写道:
Hello,
I gave both the patches a try, and here are my findings.
Thanks for the test!
With the first patch by Yu, I did not see any hang or errors. I tried
a number of bitmap chunk sizes, and ran fio for few hours, and there
was no hang.
This is good news! However, there is still a long road for my approch
to land, this requires a lot of other changes to work.
With the second patch Xiao, I hit the following BUG_ON on the first
minute of my fio run.
This is sad. :(
[ 113.902982] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 113.903315] CPU: 38 UID: 0 PID: 9767 Comm: kworker/38:3H Kdump:
loaded Not tainted 6.11.5-storage
#6.11.5-1+feature+v6.11+20241111.0643+cbe84cc3~deb12
[ 113.904120] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021
[ 113.904519] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[ 113.904888] RIP: 0010:__add_stripe_bio+0x23f/0x250 [raid456]
Can you provide the addr2line of this?
gdb raid456.ko
list *(__add_stripe_bio+0x23f)
Sorry. I missed the first line while copying.
[ 113.902680] kernel BUG at drivers/md/raid5.c:3525!
Can you give the following patch a test again, on the top of Xiao's
patch.
Thanks,
Kuai
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 6e318598a7b6..189f784aed00 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3516,7 +3516,7 @@ static void __add_stripe_bio(struct stripe_head
*sh, struct bio *bi,
bip = &sh->dev[dd_idx].toread;
}
- while (*bip && (*bip)->bi_iter.bi_sector < bi->bi_iter.bi_sector)
+ while (*bip && (*bip)->bi_iter.bi_sector <= bi->bi_iter.bi_sector)
bip = &(*bip)->bi_next;
if (!forwrite || previous)
Still hangs. Following is the stack trace.
[ 22.702034] netconsole-setup: Test log message to verify netconsole
configuration.
[ 134.949923] Oops: general protection fault, probably for
non-canonical address 0x761acac3b7d57b17: 0000 [#1] PREEMPT SMP PTI
[ 134.950621] CPU: 35 UID: 0 PID: 833 Comm: md300_raid5 Kdump: loaded
Not tainted 6.11.5-storage
#6.11.5-1+feature+v6.11+20241113.0858+ed8e31b5~deb12
[ 134.951414] Hardware name: Supermicro X10DRi/X10DRi, BIOS 3.3 03/03/2021
[ 134.951814] RIP: 0010:rnbd_dev_bi_end_io+0x1b/0x70 [rnbd_server]
[ 134.952185] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3
0f 1e fa 0f 1f 44 00 00 55 b8 ff ff ff ff 53 48 8b 6f 40 48 89 fb 48
8b 55 08 <f0
[ 134.953311] RSP: 0018:ffffb5b94818fb80 EFLAGS: 00010282
[ 134.953624] RAX: 00000000ffffffff RBX: ffff96e6a1d8aa80 RCX: 00000000802a0016
[ 134.954051] RDX: 761acac3b7d57aa7 RSI: 00000000802a0016 RDI: ffff96e6a1d8aa80
[ 134.954476] RBP: ffff96d705c7d8b0 R08: 0000000000000001 R09: 0000000000000001
[ 134.954901] R10: ffff96d730c59d40 R11: 0000000000000000 R12: ffff96d71b3e5000
[ 134.955326] R13: 0000000000000000 R14: ffff96d730c589d8 R15: ffff96d715882e20
[ 134.955752] FS: 0000000000000000(0000) GS:ffff96f63fbc0000(0000)
knlGS:0000000000000000
[ 134.956237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 134.956578] CR2: 00007fb5962bbe00 CR3: 000000060882c006 CR4: 00000000001706f0
[ 134.957003] Call Trace:
[ 134.957151] <TASK
[ 134.957274] ? die_addr+0x36/0x90
[ 134.957480] ? exc_general_protection+0x1bc/0x3c0
[ 134.957762] ? asm_exc_general_protection+0x26/0x30
[ 134.958054] ? rnbd_dev_bi_end_io+0x1b/0x70 [rnbd_server]
[ 134.958377] md_end_clone_io+0x42/0xa0
[ 134.958602] md_end_clone_io+0x42/0xa0
[ 134.958826] handle_stripe_clean_event+0x240/0x430 [raid456]
[ 134.959168] handle_stripe+0x783/0x1cb0 [raid456]
[ 134.959452] ? common_interrupt+0x13/0xa0
[ 134.959690] handle_active_stripes.constprop.0+0x353/0x540 [raid456]
[ 134.960073] raid5d+0x41a/0x600 [raid456]
Maybe the same BIO handled twice - and so the clone (for IO-acct) got
put again (somehow) into md_account_bio()?
I think the last change is reasonable, the BUG_ON() can be avoided and
bio chain won't be messed up.
The problem here looks like bio reference is not correct, I'll need some
time to sort that out, too complicated in raid5.
Meanwhile, can you try the following workround? I just revert the
changes that I think introduce this problem, noted that performace can
be degraded.
Thanks,
Kuai
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index f09e7677ee9f..07aa453bdb2f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5874,17 +5874,6 @@ static int add_all_stripe_bios(struct r5conf *conf,
wait_on_bit(&dev->flags, R5_Overlap,
TASK_UNINTERRUPTIBLE);
return 0;
}
- }
-
- for (dd_idx = 0; dd_idx < sh->disks; dd_idx++) {
- struct r5dev *dev = &sh->dev[dd_idx];
-
- if (dd_idx == sh->pd_idx || dd_idx == sh->qd_idx)
- continue;
-
- if (dev->sector < ctx->first_sector ||
- dev->sector >= ctx->last_sector)
- continue;
__add_stripe_bio(sh, bi, dd_idx, forwrite, previous);
clear_bit((dev->sector - ctx->first_sector) >>
