yuyufen> We recently report a BUG in raid5 on redhat 7.2. We want to yuyufen> know is anyone encountered the same problem? Are you talking RHEL 7.2 (current-ish) or Redhat 7.2, which is something like 15 years out of date? In any case, since this is a redhat supported kernel, you'll have to bring it up with them. But in general, I'd suggest you upgrade to RHEL 7.5 and see if the problem still happens. But you also would need to provide the kernel you're using, and if not something recent like 4.18 or newer, then the first thing to do is to upgrade your kernel and see if that fixed the issue. But in any case, please provide more details. John yuyufen> The stack as follow: yuyufen> [78008.110094] ------------[ cut here ]------------ yuyufen> [78008.110898] kernel BUG at drivers/md/raid5.c:4966! yuyufen> [78008.111718] invalid opcode: 0000 [#1] SMP yuyufen> [78008.131948] task: ffff882f2ab28b80 ti: ffff882f2ab30000 task.ti: yuyufen> ffff882f2ab30000 yuyufen> [78008.132734] RIP: 0010:[<ffffffffa01d584e>] [<ffffffffa01d584e>] yuyufen> handle_active_stripes.isra.41+0x4de/0x4f0 [raid456] yuyufen> [78008.133776] RSP: 0018:ffff882f2ab33cc8 EFLAGS: 00010086 yuyufen> [78008.134330] RAX: 00000000ffffffff RBX: ffff882f36ff5400 RCX: yuyufen> dead000000000200 yuyufen> [78008.135025] RDX: ffff882f36ff5488 RSI: 00000000ffffffff RDI: yuyufen> ffff882f2b7d8508 yuyufen> [78008.135804] RBP: ffff882f2ab33d68 R08: ffff882f2b7d8508 R09: yuyufen> ffff882f36ff5428 yuyufen> [78008.136723] R10: 0000000000000000 R11: 0000000000000000 R12: yuyufen> ffff882f2b7d84f8 yuyufen> [78008.137936] R13: 0000000000000000 R14: ffff882f36ff5498 R15: yuyufen> ffff882f2b7d8508 yuyufen> [78008.139151] FS: 0000000000000000(0000) GS:ffff882f7fc80000(0000) yuyufen> knlGS:0000000000000000 yuyufen> [78008.140496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 yuyufen> [78008.141441] CR2: 00007f888240d000 CR3: 0000002af30c0000 CR4: yuyufen> 00000000003407e0 yuyufen> [78008.142620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: yuyufen> 0000000000000000 yuyufen> [78008.143815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: yuyufen> 0000000000000400 yuyufen> [78008.145027] Stack: yuyufen> [78008.145383] 0000000000000046 0000000000011628 0000000000011628 yuyufen> ffff882f36ff5670 yuyufen> [78008.146700] ffffffffffffffd8 ffffffff2b5a31c0 ffff882f2b5a0ee8 yuyufen> ffff882f2b177740 yuyufen> [78008.148079] ffff882f2b7d89f0 ffff882f2ae8ca88 ffff882f2b170ee8 yuyufen> ffff882f2b5a7740 yuyufen> [78008.149385] ffff882f2a84acb8 ffff882f2b5a31b0 000000009f37ab80 yuyufen> 0000000000000000 yuyufen> [78008.150709] ffff882f2ab33dd0 ffff882f36ff5400 ffff882f36ff5670 yuyufen> ffff882f2ab28b80 yuyufen> [78008.152009] Call Trace: yuyufen> [78008.152450] [<ffffffffa01d5d68>] raid5d+0x508/0x760 [raid456] yuyufen> [78008.153343] [<ffffffff814c68d5>] md_thread+0x155/0x1a0 yuyufen> [78008.154214] [<ffffffff810a71d0>] ? wake_up_atomic_t+0x30/0x30 yuyufen> [78008.155181] [<ffffffff814c6780>] ? md_safemode_timeout+0x50/0x50 yuyufen> [78008.156194] [<ffffffff810a61af>] kthread+0xcf/0xe0 yuyufen> [78008.157015] [<ffffffff810a60e0>] ? kthread_create_on_node+0x140/0x140 yuyufen> [78008.158128] [<ffffffff816523d8>] ret_from_fork+0x58/0x90 yuyufen> [78008.161126] [<ffffffff810a60e0>] ? kthread_create_on_node+0x140/0x140 yuyufen> [78008.164297] Code: c6 f8 a4 1d a0 4c 0f 44 c2 4c 39 e0 48 0f 44 ca 48 yuyufen> c7 c2 50 84 1d a0 31 c0 e8 df 6d 14 e1 49 8b 04 24 44 8b 5d 88 e9 a8 fb yuyufen> ff ff <0f> 0b e8 cb 57 ea e0 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 yuyufen> [78008.172435] RIP [<ffffffffa01d584e>] yuyufen> handle_active_stripes.isra.41+0x4de/0x4f0 [raid456] yuyufen> [78008.174520] RSP <ffff882f2ab33cc8> yuyufen> [78008.178475] ---[ end trace 7f1857a07ac12adf ]--- yuyufen> [78008.181447] Kernel panic - not syncing: Fatal exception yuyufen> [78008.184775] die even has been record! yuyufen> BUG_ON code as follow: yuyufen> 4893 static struct stripe_head *__get_priority_stripe(struct r5conf yuyufen> *conf, int group) yuyufen> 4894 { yuyufen> 4895 struct stripe_head *sh = NULL, *tmp; yuyufen> 4896 struct list_head *handle_list = NULL; yuyufen> 4897 struct r5worker_group *wg = NULL; yuyufen> ...... yuyufen> 4957 yuyufen> 4958 if (!sh) yuyufen> 4959 return NULL; yuyufen> 4960 yuyufen> 4961 if (wg) { yuyufen> 4962 wg->stripes_cnt--; yuyufen> 4963 sh->group = NULL; yuyufen> 4964 } yuyufen> 4965 list_del_init(&sh->lru); yuyufen> 4966 BUG_ON(atomic_inc_return(&sh->count) != 1); //BUG_ON here yuyufen> 4967 return sh; yuyufen> 4968 } yuyufen> # mdadm -D /dev/md6 yuyufen> /dev/md6: yuyufen> Version : 1.2 yuyufen> Creation Time : Mon Aug 27 12:45:44 2018 yuyufen> Raid Level : raid5 yuyufen> Array Size : 3516065792 (3353.18 GiB 3600.45 GB) yuyufen> Used Dev Size : 1758032896 (1676.59 GiB 1800.23 GB) yuyufen> Raid Devices : 3 yuyufen> Total Devices : 3 yuyufen> Persistence : Superblock is persistent yuyufen> Intent Bitmap : Internal yuyufen> Update Time : Thu Sep 13 03:57:33 2018 yuyufen> State : clean yuyufen> Active Devices : 3 yuyufen> Working Devices : 3 yuyufen> Failed Devices : 0 yuyufen> Spare Devices : 0 yuyufen> Layout : left-symmetric yuyufen> Chunk Size : 512K yuyufen> Name : host-172-16-39-106:6 (local to host host-172-16-39-106) yuyufen> UUID : 80e6ca07:7218a374:32c672c5:f516eb4e yuyufen> Events : 4241 yuyufen> Number Major Minor RaidDevice State yuyufen> 0 65 32 0 active sync /dev/sds yuyufen> 1 65 48 1 active sync /dev/sdt yuyufen> 3 65 64 2 active sync /dev/sdu yuyufen> # mdadm -E /dev/sds yuyufen> /dev/sds: yuyufen> Magic : a92b4efc yuyufen> Version : 1.2 yuyufen> Feature Map : 0x1 yuyufen> Array UUID : 80e6ca07:7218a374:32c672c5:f516eb4e yuyufen> Name : host-172-16-39-106:6 (local to host host-172-16-39-106) yuyufen> Creation Time : Mon Aug 27 12:45:44 2018 yuyufen> Raid Level : raid5 yuyufen> Raid Devices : 3 yuyufen> Avail Dev Size : 3516066224 (1676.59 GiB 1800.23 GB) yuyufen> Array Size : 3516065792 (3353.18 GiB 3600.45 GB) yuyufen> Used Dev Size : 3516065792 (1676.59 GiB 1800.23 GB) yuyufen> Data Offset : 262144 sectors yuyufen> Super Offset : 8 sectors yuyufen> Unused Space : before=262056 sectors, after=432 sectors yuyufen> State : clean yuyufen> Device UUID : 0eacd8fd:5d427c77:8000c2b4:035928de yuyufen> Internal Bitmap : 8 sectors from superblock yuyufen> Update Time : Thu Sep 13 03:59:33 2018 yuyufen> Bad Block Log : 512 entries available at offset 72 sectors yuyufen> Checksum : d7946e32 - correct yuyufen> Events : 4241 yuyufen> Layout : left-symmetric yuyufen> Chunk Size : 512K yuyufen> Device Role : Active device 0 yuyufen> Array State : AAA ('A' == active, '.' == missing, 'R' == replacing) yuyufen> Any suggestions are welcome. yuyufen> Thanks a lot yuyufen> Yufen