> On Aug 17, 2017, at 3:58 PM, Shaohua Li <shli@xxxxxxxxxx> wrote:
>
> On Tue, Aug 15, 2017 at 07:27:44PM +0000, Stephane Thiell wrote:
>>
>>> On Aug 15, 2017, at 11:39 AM, Shaohua Li <shli@xxxxxxxxxx> wrote:
>>>
>>> On Tue, Aug 15, 2017 at 05:59:38PM +0000, Stephane Thiell wrote:
>>>>
>>>>> On Aug 15, 2017, at 2:24 AM, Guoqing Jiang <gqjiang@xxxxxxxx> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 08/14/2017 01:29 AM, Stephane Thiell wrote:
>>>>>> Also known as "kernel BUG at drivers/md/raid5.c:527!"
>>>>>>
>>>>>> I’m still struggling with this issue in raid5.c. I currently have one crash per week in average on a system using Lustre and 24 x 10-disk raid6 per server. It did happen with or without check running. I now think there is a race at some point where a released stripe head (batched) is added to conf->inactive_list without sh->batch_head being set back to NULL.
>>>>>>
>>>>>> Because sh->batch_head is only being set back to NULL in a single function, break_stripe_batch_list(), my guess is that a call to raid5_release_stripe() on the head stripe is performed elsewhere. I didn’t find where this could come from yet. So perhaps sh->batch_head should be just set to NULL in do_release_stripe(), before being added to the inactive list?
>>>>>>
>>>>>> Alternatively, I wonder if it would make more sense to set batch_head to NULL where sh->overwrite_disks is also set back to 0 (like in ops_run_biodrain() today).
>>>>>>
>>>>>> Because sh->overwrite_disks is also set to 0 in init_stripe(), my current plan is to try this change, at least as a workaround: remove the BUG_ON(sh->batch_head) from init_stripe() and set sh->batch_head = NULL (I don’t think I need to lock on sh->stripe_lock there?). I will get new hardware soon so I will give it a try on a test system. Any suggestions welcomed…
>>>>>>
>>>>>> Quick analysis of the last vmcore:
>>>>>>
>>>>>> crash> ps | grep ">"
>>>>>> ...
>>>>>>> 16119 16111 22 ffff881f93bace70 RU 0.0 150772 10748 e2label
>>>>>>> 151448 2 34 ffff881ff753bec0 RU 0.0 0 0 [kiblnd_sd_00_00]
>>>>>>> 151449 2 2 ffff881ff753edd0 RU 0.0 0 0 [kiblnd_sd_00_01]
>>>>>>> 154637 2 29 ffff881f776b1f60 RU 0.0 0 0 [md32_raid6]
>>>>>>> 163643 2 11 ffff8820d200bec0 UN 0.0 0 0 [ll_ost_io01_008]
>>>>>>> 164083 2 13 ffff8820d2753ec0 UN 0.0 0 0 [ll_ost_io01_034]
>>>>>>> 171389 2 5 ffff883fd072de20 UN 0.0 0 0 [ll_ost_io01_058]
>>>>>>> 171391 2 17 ffff883fd072ce70 UN 0.0 0 0 [ll_ost_io01_060]
>>>>>>> 171399 2 33 ffff883fd9e9ce70 UN 0.0 0 0 [ll_ost_io01_068]
>>>>>>> 237019 2 15 ffff882715628fb0 UN 0.0 0 0 [ll_ost_io01_084]
>>>>>>> 237033 2 3 ffff881fb0250fb0 UN 0.0 0 0 [ll_ost_io01_098]
>>>>>> ...
>>>>>>
>>>>>> crash> bt 237033
>>>>>> PID: 237033 TASK: ffff881fb0250fb0 CPU: 3 COMMAND: "ll_ost_io01_098"
>>>>>> #0 [ffff8826fa3e7240] machine_kexec at ffffffff81059bdb
>>>>>> #1 [ffff8826fa3e72a0] __crash_kexec at ffffffff811057c2
>>>>>> #2 [ffff8826fa3e7370] crash_kexec at ffffffff811058b0
>>>>>> #3 [ffff8826fa3e7388] oops_end at ffffffff8168fac8
>>>>>> #4 [ffff8826fa3e73b0] die at ffffffff8102e93b
>>>>>> #5 [ffff8826fa3e73e0] do_trap at ffffffff8168f180
>>>>>> #6 [ffff8826fa3e7430] do_invalid_op at ffffffff8102b144
>>>>>> #7 [ffff8826fa3e74e0] invalid_op at ffffffff81698b5e
>>>>>> [exception RIP: raid5_get_active_stripe+1809]
>>>>>> RIP: ffffffffa0fefd71 RSP: ffff8826fa3e7590 RFLAGS: 00010086
>>>>>> RAX: 0000000000000000 RBX: ffff883df051641c RCX: dead000000000200
>>>>>> RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff881e08d54d30
>>>>>> RBP: ffff8826fa3e7638 R8: ffff881e08d54d40 R9: 0000000000000080
>>>>>> R10: 0000000000000000 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R13: ffff881e08d54d30 R14: 0000000000000000 R15: 0000000000000000
>>>>>> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
>>>>>> #8 [ffff8826fa3e7640] raid5_make_request at ffffffffa0feff55 [raid456]
>>>>>> #9 [ffff8826fa3e7730] md_make_request at ffffffff814f838c
>>>>>> #10 [ffff8826fa3e7790] generic_make_request at ffffffff812eedc9
>>>>>> #11 [ffff8826fa3e77d8] submit_bio at ffffffff812eef11
>>>>>> #12 [ffff8826fa3e7830] osd_submit_bio at ffffffffa10cdbcc [osd_ldiskfs]
>>>>>> #13 [ffff8826fa3e7840] osd_do_bio at ffffffffa10d03c1 [osd_ldiskfs]
>>>>>> #14 [ffff8826fa3e7960] osd_write_commit at ffffffffa10d0b94 [osd_ldiskfs]
>>>>>> #15 [ffff8826fa3e7a00] ofd_commitrw_write at ffffffffa120c734 [ofd]
>>>>>> #16 [ffff8826fa3e7a78] ofd_commitrw at ffffffffa120ff07 [ofd]
>>>>>> #17 [ffff8826fa3e7b08] obd_commitrw at ffffffffa0c7dc22 [ptlrpc]
>>>>>> #18 [ffff8826fa3e7b70] tgt_brw_write at ffffffffa0c55fc1 [ptlrpc]
>>>>>> #19 [ffff8826fa3e7cd8] tgt_request_handle at ffffffffa0c52275 [ptlrpc]
>>>>>> #20 [ffff8826fa3e7d20] ptlrpc_server_handle_request at ffffffffa0bfe1fb [ptlrpc]
>>>>>> #21 [ffff8826fa3e7de8] ptlrpc_main at ffffffffa0c022b0 [ptlrpc]
>>>>>> #22 [ffff8826fa3e7ec8] kthread at ffffffff810b0a4f
>>>>>> #23 [ffff8826fa3e7f50] ret_from_fork at ffffffff81697218
>>>>>>
>>>>>>
>>>>>> After some disassembly work, I found out that all active tasks are working on the same md (all are raid6):
>>>>>>
>>>>>> md32’s (struct r5conf *) is 0xffff883df0516400
>>>>>>
>>>>>> Our crashed task:
>>>>>>
>>>>>> crash> printf "%s", ((struct r5conf)*0xffff883df0516400).cache_name[0]
>>>>>> raid6-md32
>>>>>>
>>>>>> Example for md32_raid6 task in release_inactive_stripe_list:
>>>>>>
>>>>>> crash> bt 154637
>>>>>> PID: 154637 TASK: ffff881f776b1f60 CPU: 29 COMMAND: "md32_raid6"
>>>>>> #0 [ffff883ffc985e48] crash_nmi_callback at ffffffff8104d2e2
>>>>>> #1 [ffff883ffc985e58] nmi_handle at ffffffff8168fc77
>>>>>> #2 [ffff883ffc985eb0] do_nmi at ffffffff8168fe83
>>>>>> #3 [ffff883ffc985ef0] end_repeat_nmi at ffffffff8168f093
>>>>>> [exception RIP: _raw_spin_lock_irqsave+64]
>>>>>> RIP: ffffffff8168e5f0 RSP: ffff881e8d0cbc60 RFLAGS: 00000006
>>>>>> RAX: 00000000000031bb RBX: ffff883df0516740 RCX: 000000000000009a
>>>>>> RDX: 0000000000000098 RSI: 000000000000009a RDI: ffff883df051641c
>>>>>> RBP: ffff881e8d0cbc60 R8: 0000000000000206 R9: 0000000000000000
>>>>>> R10: 0000000000000000 R11: 0000000000000007 R12: ffff883df0516400
>>>>>> R13: ffff883df051641c R14: 0000000000000006 R15: 0000000000000005
>>>>>> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
>>>>>> --- <NMI exception stack> ---
>>>>>> #4 [ffff881e8d0cbc60] _raw_spin_lock_irqsave at ffffffff8168e5f0
>>>>>> #5 [ffff881e8d0cbc68] release_inactive_stripe_list at ffffffffa0fe5500 [raid456]
>>>>>> #6 [ffff881e8d0cbcb0] handle_active_stripes at ffffffffa0ff34c2 [raid456]
>>>>>> #7 [ffff881e8d0cbd60] raid5d at ffffffffa0ff3c28 [raid456]
>>>>>> #8 [ffff881e8d0cbe50] md_thread at ffffffff814fb745
>>>>>> #9 [ffff881e8d0cbec8] kthread at ffffffff810b0a4f
>>>>>> #10 [ffff881e8d0cbf50] ret_from_fork at ffffffff81697218
>>>>>>
>>>>>>
>>>>>> crash> bt 163643 164083 171389 171391 171399 237019 237033 |grep "exception RIP:"
>>>>>> [exception RIP: _raw_spin_lock_irq+63]
>>>>>> [exception RIP: _raw_spin_lock_irq+63]
>>>>>> [exception RIP: _raw_spin_lock_irq+58]
>>>>>> [exception RIP: _raw_spin_lock_irq+63]
>>>>>> [exception RIP: _raw_spin_lock_irq+58]
>>>>>> [exception RIP: _raw_spin_lock_irq+63]
>>>>>> [exception RIP: raid5_get_active_stripe+1809]
>>>>>>
>>>>>> crash> bt 163643 164083 171389 171391 171399 237019 237033 |grep "raid5_get_active_stripe"
>>>>>> #5 [ffff883bcee93588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> #5 [ffff883ba4af7588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> #5 [ffff88390877b588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> #5 [ffff88390803b588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> #5 [ffff88390831f588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> #5 [ffff8826fa7f7588] raid5_get_active_stripe at ffffffffa0fef6cb [raid456]
>>>>>> [exception RIP: raid5_get_active_stripe+1809]
>>>>>>
>>>>>> crash> bt 163643 164083 171389 171391 171399 237019 237033 |grep R12
>>>>>> R10: 0000000000000003 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000001 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000005 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000007 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000009 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000009 R11: 0000000000000000 R12: ffff883df0516400
>>>>>> R10: 0000000000000000 R11: 0000000000000000 R12: ffff883df0516400
>>>>>>
>>>>>>
>>>>>> The bad stripe_head (sh) is available at (usually in %r13):
>>>>>>
>>>>>> crash> p ((struct stripe_head)*0xffff881e08d54d30).raid_conf
>>>>>> $98 = (struct r5conf *) 0xffff883df0516400
>>>>>>
>>>>>> and batch_head is not NULL... that's the issue:
>>>>>>
>>>>>> crash> p ((struct stripe_head)*0xffff881e08d54d30).batch_head
>>>>>> $99 = (struct stripe_head *) 0xffff881e08d54d30
>>>>>>
>>>>>> Full struct dump below. We can see than overwrite_disks is set to 0, state is 0… but this batch_head not set to NULL makes it crash in init_stripe():
>>>>>>
>>>>>> crash> p ((struct stripe_head)*0xffff881e08d54d30)
>>>>>> $6 = {
>>>>>> hash = {
>>>>>> next = 0x0,
>>>>>> pprev = 0x0
>>>>>> },
>>>>>> lru = {
>>>>>> next = 0xffff881e08d54d40,
>>>>>> prev = 0xffff881e08d54d40
>>>>>> },
>>>>>> release_list = {
>>>>>> next = 0xffff881dfe405cc0
>>>>>> },
>>>>>> raid_conf = 0xffff883df0516400,
>>>>>> generation = 0,
>>>>>> sector = 3701224808,
>>>>>> pd_idx = 1,
>>>>>> qd_idx = 2,
>>>>>> ddf_layout = 0,
>>>>>> hash_lock_index = 5,
>>>>>> state = 0,
>>>>>> count = {
>>>>>> counter = 0
>>>>>> },
>>>>>> bm_seq = 22708787,
>>>>>> disks = 10,
>>>>>> overwrite_disks = 0,
>>>>>> check_state = check_state_idle,
>>>>>> reconstruct_state = reconstruct_state_idle,
>>>>>> stripe_lock = {
>>>>>> {
>>>>>> rlock = {
>>>>>> raw_lock = {
>>>>>> {
>>>>>> head_tail = 3709263126,
>>>>>> tickets = {
>>>>>> head = 56598,
>>>>>> tail = 56598
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> },
>>>>>> cpu = 15,
>>>>>> group = 0x0,
>>>>>> batch_head = 0xffff881e08d54d30,
>>>>>> batch_lock = {
>>>>>> {
>>>>>> rlock = {
>>>>>> raw_lock = {
>>>>>> {
>>>>>> head_tail = 9961624,
>>>>>> tickets = {
>>>>>> head = 152,
>>>>>> tail = 152
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> },
>>>>>> batch_list = {
>>>>>> next = 0xffff881dfe405d28,
>>>>>> prev = 0xffff881e0e2dec98
>>>>>> },
>>>>>> log_io = 0x0,
>>>>>> log_list = {
>>>>>> next = 0x0,
>>>>>> prev = 0x0
>>>>>> },
>>>>>> ops = {
>>>>>> target = 0,
>>>>>> target2 = 0,
>>>>>> zero_sum_result = (unknown: 0)
>>>>>> },
>>>>>> dev = {{
>>>>>> req = {
>>>>>> bi_sector = 0,
>>>>>> bi_next = 0x0,
>>>>>> bi_bdev = 0x0,
>>>>>> bi_flags = 1,
>>>>>> bi_rw = 0,
>>>>>> bi_vcnt = 0,
>>>>>> bi_idx = 0,
>>>>>> bi_phys_segments = 0,
>>>>>> bi_size = 0,
>>>>>> bi_seg_front_size = 0,
>>>>>> bi_seg_back_size = 0,
>>>>>> bi_end_io = 0x0,
>>>>>> bi_private = 0xffff881e08d54d30,
>>>>>> bi_ioc = 0x0,
>>>>>> bi_css = 0x0,
>>>>>> bi_integrity = 0x0,
>>>>>> bi_max_vecs = 1,
>>>>>> bi_cnt = {
>>>>>> counter = 1
>>>>>> },
>>>>>> bi_io_vec = 0xffff881e08d54f00,
>>>>>> bi_pool = 0x0,
>>>>>> {
>>>>>> bio_aux = 0x0,
>>>>>> __UNIQUE_ID_rh_kabi_hide15 = {
>>>>>> rh_reserved1 = 0x0
>>>>>> },
>>>>>> {<No data fields>}
>>>>>> },
>>>>>> bi_inline_vecs = 0xffff881e08d54e78
>>>>>> },
>>>>>> rreq = {
>>>>>> bi_sector = 0,
>>>>>> bi_next = 0x0,
>>>>>> bi_bdev = 0x0,
>>>>>> bi_flags = 1,
>>>>>> bi_rw = 0,
>>>>>> bi_vcnt = 0,
>>>>>> bi_idx = 0,
>>>>>> bi_phys_segments = 0,
>>>>>> bi_size = 0,
>>>>>> bi_seg_front_size = 0,
>>>>>> bi_seg_back_size = 0,
>>>>>> bi_end_io = 0x0,
>>>>>> bi_private = 0xffff881e08d54d30,
>>>>>> bi_ioc = 0x0,
>>>>>> bi_css = 0x0,
>>>>>> bi_integrity = 0x0,
>>>>>> bi_max_vecs = 1,
>>>>>> bi_cnt = {
>>>>>> counter = 1
>>>>>> },
>>>>>> bi_io_vec = 0xffff881e08d54f10,
>>>>>> bi_pool = 0x0,
>>>>>> {
>>>>>> bio_aux = 0x0,
>>>>>> __UNIQUE_ID_rh_kabi_hide15 = {
>>>>>> rh_reserved1 = 0x0
>>>>>> },
>>>>>> {<No data fields>}
>>>>>> },
>>>>>> bi_inline_vecs = 0xffff881e08d54f00
>>>>>> },
>>>>>> vec = {
>>>>>> bv_page = 0xffffea0078236080,
>>>>>> bv_len = 4096,
>>>>>> bv_offset = 0
>>>>>> },
>>>>>> rvec = {
>>>>>> bv_page = 0x0,
>>>>>> bv_len = 0,
>>>>>> bv_offset = 0
>>>>>> },
>>>>>> page = 0xffffea0078236080,
>>>>>> orig_page = 0xffffea0078236080,
>>>>>> toread = 0x0,
>>>>>> read = 0x0,
>>>>>> towrite = 0x0,
>>>>>> written = 0xffff8824832a7c00,
>>>>>> sector = 29609798632,
>>>>>> flags = 24,
>>>>>
>>>>> Seems the flags is R5_OVERWRITE | R5_Insync, handle_stripe_clean_event
>>>>> only calls break_stripe_batch_list if do_endio is set to true.
>>>>>
>>>>> if (head_sh->batch_head && do_endio)
>>>>> break_stripe_batch_list(head_sh, STRIPE_EXPAND_SYNC_FLAGS);
>>>>>
>>>>> And do_endio is set to true if we meet the below condition.
>>>>>
>>>>> if (!test_bit(R5_LOCKED, &dev->flags) &&
>>>>> (test_bit(R5_UPTODATE, &dev->flags) ||
>>>>> test_bit(R5_Discard, &dev->flags) ||
>>>>> test_bit(R5_SkipCopy, &dev->flags))) {
>>>>>
>>>>> I guess it is why the batch_head is not null, just my 2 cents.
>>>>
>>>>
>>>> That’s a very good point, thanks Guoqing!
>>>>
>>>> I checked another similar crash dump, and found out that this same flags was 16 (R5_Insync alone), which doesn’t meet the condition for having do_endio = true neither.
>>>>
>>>> The code below in raid5_end_write_request does directly release a sh->batch_head, couldn’t that be an issue?
>>>>
>>>> static void raid5_end_write_request(struct bio *bi)
>>>> {
>>>>
>>>> ...
>>>>
>>>> raid5_release_stripe(sh);
>>>>
>>>> if (sh->batch_head && sh != sh->batch_head)
>>>> raid5_release_stripe(sh->batch_head);
>>>> }
>>>
>>> Can you dump the full stripe_head info for this crash dump?
>>
>> Yes, of course!
>>
>> R13: ffff881e53f5dca0
>>
>> crash> p (struct stripe_head)*0xffff881e53f5dca0
>> $1 = {
>> hash = {
>> next = 0x0,
>> pprev = 0x0
>> },
>> lru = {
>> next = 0xffff881e53f5dcb0,
>> prev = 0xffff881e53f5dcb0
>> },
>> release_list = {
>> next = 0xffff881e63faec30
>> },
>> raid_conf = 0xffff881f9fb81400,
>> generation = 0,
>> sector = 2278766080,
>> pd_idx = 9,
>> qd_idx = 0,
>> ddf_layout = 0,
>> hash_lock_index = 0,
>> state = 0,
>> count = {
>> counter = 0
>> },
>> bm_seq = 21153188,
>> disks = 10,
>> overwrite_disks = 0,
>> check_state = check_state_idle,
>> reconstruct_state = reconstruct_state_idle,
>> stripe_lock = {
>> {
>> rlock = {
>> raw_lock = {
>> {
>> head_tail = 3483684772,
>> tickets = {
>> head = 53156,
>> tail = 53156
>> }
>> }
>> }
>> }
>> }
>> },
>> cpu = 12,
>> group = 0x0,
>> batch_head = 0xffff881e53f5dca0,
>> batch_lock = {
>> {
>> rlock = {
>> raw_lock = {
>> {
>> head_tail = 1983149620,
>> tickets = {
>> head = 30260,
>> tail = 30260
>> }
>> }
>> }
>> }
>> }
>> },
>> batch_list = {
>> next = 0xffff881e63faec98,
>> prev = 0xffff881e5c390088
>> },
>> log_io = 0x0,
>> log_list = {
>> next = 0x0,
>> prev = 0x0
>> },
>> ops = {
>> target = 0,
>> target2 = 0,
>> zero_sum_result = (unknown: 0)
>> },
>> dev = {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5de70,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5dde8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5de80,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5de70
>> },
>> vec = {
>> bv_page = 0xffffea00794fe000,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe000,
>> orig_page = 0xffffea00794fe000,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0x0,
>> sector = 0,
>> flags = 16,
>> log_checksum = 0
>> }}
>> }
>>
>>
>>
>>> Also stripe_head
>>> should include 10 dev for your 10-disk raid, but crash dump only show one, is
>>> it possible you can dump the info for the devs in the stripe?
>>
>> Oh right, crash didn’t display it probably because it is an array allocated dynamically.
>>
>> Let’s try this to print the 10 devs:
>>
>> crash> p ((struct stripe_head)*0xffff881e53f5dca0).dev@10
>> $10 =
>> {{{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5de70,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5dde8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5de80,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5de70
>> },
>> vec = {
>> bv_page = 0xffffea00794fe000,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe000,
>> orig_page = 0xffffea00794fe000,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0x0,
>> sector = 0,
>> flags = 16,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5dfe8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5df60
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5dff8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5dfe8
>> },
>> vec = {
>> bv_page = 0xffffea00794fe040,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe040,
>> orig_page = 0xffffea00794fe040,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230128640,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e160,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e0d8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e170,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e160
>> },
>> vec = {
>> bv_page = 0xffffea00794fe080,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe080,
>> orig_page = 0xffffea00794fe080,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230128768,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e2d8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e250
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e2e8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e2d8
>> },
>> vec = {
>> bv_page = 0xffffea00794fe0c0,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe0c0,
>> orig_page = 0xffffea00794fe0c0,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230128896,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e450,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e3c8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e460,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e450
>> },
>> vec = {
>> bv_page = 0xffffea00794fe100,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe100,
>> orig_page = 0xffffea00794fe100,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230129024,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e5c8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e540
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e5d8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e5c8
>> },
>> vec = {
>> bv_page = 0xffffea00794fe140,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe140,
>> orig_page = 0xffffea00794fe140,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230129152,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e740,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e6b8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e750,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e740
>> },
>> vec = {
>> bv_page = 0xffffea00794fe180,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe180,
>> orig_page = 0xffffea00794fe180,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230129280,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e8b8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e830
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5e8c8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e8b8
>> },
>> vec = {
>> bv_page = 0xffffea00794fe1c0,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe1c0,
>> orig_page = 0xffffea00794fe1c0,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230129408,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5ea30,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5e9a8
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5ea40,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5ea30
>> },
>> vec = {
>> bv_page = 0xffffea00794fe200,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe200,
>> orig_page = 0xffffea00794fe200,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff881fea351800,
>> sector = 18230129536,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5eba8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5eb20
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e53f5dca0,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e53f5ebb8,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e53f5eba8
>> },
>> vec = {
>> bv_page = 0xffffea00794fe240,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00794fe240,
>> orig_page = 0xffffea00794fe240,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0x0,
>> sector = 0,
>> flags = 16,
>> log_checksum = 0
>> }}}
>>
>>
>> ---------------------------------------------------
>>
>> And this is the stripe_head devs from the crash dump I sent yesterday:
>>
>> R13: ffff881e08d54d30
>>
>> crash> p ((struct stripe_head)*0xffff881e08d54d30).dev@10
>> $1 =
>> {{{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d54f00,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d54e78
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d54f10,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d54f00
>> },
>> vec = {
>> bv_page = 0xffffea0078236080,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236080,
>> orig_page = 0xffffea0078236080,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609798632,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55078,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d54ff0
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55088,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55078
>> },
>> vec = {
>> bv_page = 0xffffea00782360c0,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00782360c0,
>> orig_page = 0xffffea00782360c0,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0x0,
>> sector = 0,
>> flags = 16,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d551f0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55168
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55200,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d551f0
>> },
>> vec = {
>> bv_page = 0xffffea0078236100,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236100,
>> orig_page = 0xffffea0078236100,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0x0,
>> sector = 0,
>> flags = 16,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55368,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d552e0
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55378,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55368
>> },
>> vec = {
>> bv_page = 0xffffea0078236140,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236140,
>> orig_page = 0xffffea0078236140,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609797736,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d554e0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55458
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d554f0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d554e0
>> },
>> vec = {
>> bv_page = 0xffffea0078236180,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236180,
>> orig_page = 0xffffea0078236180,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609797864,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55658,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d555d0
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55668,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55658
>> },
>> vec = {
>> bv_page = 0xffffea00782361c0,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00782361c0,
>> orig_page = 0xffffea00782361c0,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609797992,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d557d0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55748
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d557e0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d557d0
>> },
>> vec = {
>> bv_page = 0xffffea0078236200,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236200,
>> orig_page = 0xffffea0078236200,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609798120,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55948,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d558c0
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55958,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55948
>> },
>> vec = {
>> bv_page = 0xffffea0078236240,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236240,
>> orig_page = 0xffffea0078236240,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609798248,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55ac0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55a38
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55ad0,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55ac0
>> },
>> vec = {
>> bv_page = 0xffffea0078236280,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea0078236280,
>> orig_page = 0xffffea0078236280,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609798376,
>> flags = 24,
>> log_checksum = 0
>> }}, {{
>> req = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55c38,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55bb0
>> },
>> rreq = {
>> bi_sector = 0,
>> bi_next = 0x0,
>> bi_bdev = 0x0,
>> bi_flags = 1,
>> bi_rw = 0,
>> bi_vcnt = 0,
>> bi_idx = 0,
>> bi_phys_segments = 0,
>> bi_size = 0,
>> bi_seg_front_size = 0,
>> bi_seg_back_size = 0,
>> bi_end_io = 0x0,
>> bi_private = 0xffff881e08d54d30,
>> bi_ioc = 0x0,
>> bi_css = 0x0,
>> bi_integrity = 0x0,
>> bi_max_vecs = 1,
>> bi_cnt = {
>> counter = 1
>> },
>> bi_io_vec = 0xffff881e08d55c48,
>> bi_pool = 0x0,
>> {
>> bio_aux = 0x0,
>> __UNIQUE_ID_rh_kabi_hide15 = {
>> rh_reserved1 = 0x0
>> },
>> {<No data fields>}
>> },
>> bi_inline_vecs = 0xffff881e08d55c38
>> },
>> vec = {
>> bv_page = 0xffffea00782362c0,
>> bv_len = 4096,
>> bv_offset = 0
>> },
>> rvec = {
>> bv_page = 0x0,
>> bv_len = 0,
>> bv_offset = 0
>> },
>> page = 0xffffea00782362c0,
>> orig_page = 0xffffea00782362c0,
>> toread = 0x0,
>> read = 0x0,
>> towrite = 0x0,
>> written = 0xffff8824832a7c00,
>> sector = 29609798504,
>> flags = 24,
>> log_checksum = 0
>> }}}
>
> Didn't find anything interesting yet, sorry. There might be a race condition
> which makes the stripe batch head and also adds it to another batch list,
> that's my current theory, but can't find such race condition.
Ok. In any case, thanks much for taking the time to look at this.
>
> So in the crashdump, we have something like this:
>
> batch_list = {
> next = 0xffff881dfe405d28,
> prev = 0xffff881e0e2dec98
> },
>
> Can you convert the next/prev to stripes and dump their state? That probably is
> helpful for debugging.
Results below. They do look partially zeroed, and disks=0 so I’m not sure that makes sense to dump their devs.
Note: next and prev in the other crash look similar.
Here is ‘next':
crash> p ((struct stripe_head)*0xffff881dfe405d28)
$5 = {
hash = {
next = 0xffff881e0e2dec98,
pprev = 0xffff881e08d54db8
},
lru = {
next = 0x0,
prev = 0x0
},
release_list = {
next = 0x0
},
raid_conf = 0x0,
generation = 0,
sector = 0,
pd_idx = 0,
qd_idx = 0,
ddf_layout = 0,
hash_lock_index = 0,
state = 0,
count = {
counter = 1
},
bm_seq = 0,
disks = 0,
overwrite_disks = 0,
check_state = check_state_idle,
reconstruct_state = reconstruct_state_idle,
stripe_lock = {
{
rlock = {
raw_lock = {
{
head_tail = 0,
tickets = {
head = 0,
tail = 0
}
}
}
}
}
},
cpu = 0,
group = 0x0,
batch_head = 0x0,
batch_lock = {
{
rlock = {
raw_lock = {
{
head_tail = 4265630880,
tickets = {
head = 23712,
tail = 65088
}
}
}
}
}
},
batch_list = {
next = 0x0,
prev = 0x0
},
log_io = 0x0,
log_list = {
next = 0x100000001,
prev = 0xffff881dfe405e70
},
ops = {
target = 0,
target2 = 0,
zero_sum_result = (unknown: 0)
},
dev = {{
req = {
bi_sector = 0,
bi_next = 0x0,
bi_bdev = 0x0,
bi_flags = 1,
bi_rw = 0,
bi_vcnt = 0,
bi_idx = 0,
bi_phys_segments = 0,
bi_size = 0,
bi_seg_front_size = 0,
bi_seg_back_size = 0,
bi_end_io = 0x0,
bi_private = 0xffff881dfe405ca0,
bi_ioc = 0x0,
bi_css = 0x0,
bi_integrity = 0x0,
bi_max_vecs = 1,
bi_cnt = {
counter = 1
},
bi_io_vec = 0xffff881dfe405e80,
bi_pool = 0x0,
{
bio_aux = 0x0,
__UNIQUE_ID_rh_kabi_hide15 = {
rh_reserved1 = 0x0
},
{<No data fields>}
},
bi_inline_vecs = 0xffff881dfe405e70
},
rreq = {
bi_sector = 18446719886466550720,
bi_next = 0x1000,
bi_bdev = 0x0,
bi_flags = 0,
bi_rw = 18446719886466550720,
bi_vcnt = 3008,
bi_idx = 30713,
bi_phys_segments = 4294961664,
bi_size = 0,
bi_seg_front_size = 0,
bi_seg_back_size = 0,
bi_end_io = 0x0,
bi_private = 0xffff8824832a7c00,
bi_ioc = 0x6e4e1abf0,
bi_css = 0x8,
bi_integrity = 0x0,
bi_max_vecs = 0,
bi_cnt = {
counter = 0
},
bi_io_vec = 0x0,
bi_pool = 0x0,
{
bio_aux = 0x1,
__UNIQUE_ID_rh_kabi_hide15 = {
rh_reserved1 = 0x1
},
{<No data fields>}
},
bi_inline_vecs = 0xffff881dfe405ef8
},
vec = {
bv_page = 0x0,
bv_len = 0,
bv_offset = 0
},
rvec = {
bv_page = 0x0,
bv_len = 0,
bv_offset = 0
},
page = 0x0,
orig_page = 0xffff881dfe405ca0,
toread = 0x0,
read = 0x0,
towrite = 0x0,
written = 0x100000001,
sector = 18446612261133901800,
flags = 0,
log_checksum = 0
}}
}
And this is ‘prev’:
crash> p ((struct stripe_head)*0xffff881e0e2dec98)
$6 = {
hash = {
next = 0xffff881e08d54db8,
pprev = 0xffff881dfe405d28
},
lru = {
next = 0x0,
prev = 0x0
},
release_list = {
next = 0x0
},
raid_conf = 0x0,
generation = 0,
sector = 0,
pd_idx = 0,
qd_idx = 0,
ddf_layout = 0,
hash_lock_index = 0,
state = 0,
count = {
counter = 1
},
bm_seq = 0,
disks = 0,
overwrite_disks = 0,
check_state = check_state_idle,
reconstruct_state = reconstruct_state_idle,
stripe_lock = {
{
rlock = {
raw_lock = {
{
head_tail = 0,
tickets = {
head = 0,
tail = 0
}
}
}
}
}
},
cpu = 0,
group = 0x0,
batch_head = 0x0,
batch_lock = {
{
rlock = {
raw_lock = {
{
head_tail = 237890576,
tickets = {
head = 60432,
tail = 3629
}
}
}
}
}
},
batch_list = {
next = 0x0,
prev = 0x0
},
log_io = 0x0,
log_list = {
next = 0x100000001,
prev = 0xffff881e0e2dede0
},
ops = {
target = 0,
target2 = 0,
zero_sum_result = (unknown: 0)
},
dev = {{
req = {
bi_sector = 0,
bi_next = 0x0,
bi_bdev = 0x0,
bi_flags = 1,
bi_rw = 0,
bi_vcnt = 0,
bi_idx = 0,
bi_phys_segments = 0,
bi_size = 0,
bi_seg_front_size = 0,
bi_seg_back_size = 0,
bi_end_io = 0x0,
bi_private = 0xffff881e0e2dec10,
bi_ioc = 0x0,
bi_css = 0x0,
bi_integrity = 0x0,
bi_max_vecs = 1,
bi_cnt = {
counter = 1
},
bi_io_vec = 0xffff881e0e2dedf0,
bi_pool = 0x0,
{
bio_aux = 0x0,
__UNIQUE_ID_rh_kabi_hide15 = {
rh_reserved1 = 0x0
},
{<No data fields>}
},
bi_inline_vecs = 0xffff881e0e2dede0
},
rreq = {
bi_sector = 18446719886470726848,
bi_next = 0x1000,
bi_bdev = 0x0,
bi_flags = 0,
bi_rw = 18446719886470726848,
bi_vcnt = 50368,
bi_idx = 30776,
bi_phys_segments = 4294961664,
bi_size = 0,
bi_seg_front_size = 0,
bi_seg_back_size = 0,
bi_end_io = 0x0,
bi_private = 0xffff8824832a7c00,
bi_ioc = 0x6e4e1abf8,
bi_css = 0x1b,
bi_integrity = 0x0,
bi_max_vecs = 0,
bi_cnt = {
counter = 0
},
bi_io_vec = 0x0,
bi_pool = 0x0,
{
bio_aux = 0x1,
__UNIQUE_ID_rh_kabi_hide15 = {
rh_reserved1 = 0x1
},
{<No data fields>}
},
bi_inline_vecs = 0xffff881e0e2dee68
},
vec = {
bv_page = 0x0,
bv_len = 0,
bv_offset = 0
},
rvec = {
bv_page = 0x0,
bv_len = 0,
bv_offset = 0
},
page = 0x0,
orig_page = 0xffff881e0e2dec10,
toread = 0x0,
read = 0x0,
towrite = 0x0,
written = 0x100000001,
sector = 18446612261401128792,
flags = 0,
log_checksum = 0
}}
}
Thanks again,
Stephane��.n��������+%������w��{.n�����{����w��ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
