Hi，
在 2023/08/07 6:48, Corey Hickey 写道:
Hello,
I have encountered a reproducible NULL pointer dereference when using
the write-back journal feature for RAID-5. This _seems_ to happen
only when the journal device is itself a RAID-1.
https://docs.kernel.org/driver-api/md/raid5-cache.html
This report supersedes a report I sent to Debian earlier:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043078
Steps to reproduce, including example commands:
1. Create a RAID-1 for the journal device.
$ sudo mdadm --create /dev/md101 -n 2 -l 1
/dev/disk/by-id/ata-Samsung_SSD_850_PRO_256GB_S251NX0H60631*
2. Create a RAID-5 with the journal included. I'm using '-z 10G' for
testing in order to reduce the initial sync time.
$ sudo mdadm --create /dev/md10 -n 3 -l 5 -z 10G --write-journal
/dev/md101 /dev/disk/by-id/ata-ST32000645NS_Z1K0*
3. Enable write-back (completes once re-sync is finished).
$ until echo write-back | sudo tee /sys/block/md10/md/journal_mode ; do
sleep 5 ; done
4. Write to the disk (may take a few attempts).
$ sudo dd if=/dev/zero of=/dev/md10 iflag=fullblock bs=1M count=10240
Notes:
* The bug does not always manifest immediately but for me, it nearly
always manifests on the first or second 'dd' run.
* The bug is not limited to buffered I/O: writes via 'oflag=direct'
can cause the bug as well.
* I was not able to reproduce the bug on 10 attempts when I used a
single non-RAID SSD as the journal.
* The bug can manifest while the journal RAID-1 is resyncing or not;
the resync does not seem relevant.
My SSDs are attached to an onboard SATA controller:
$ lspci | grep 06:00
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0
x2 4-port SATA 6 Gb/s Controller (rev 11)
My hard disks are attached to an external SATA-->USB enclosure,
but I this is not relevant--I had the same problem with hard disks
attached to internal SATA controllers in earlier tests.
I'm using Debian Sid on Linux 6.4.8. The kernel is compiled locally
and installed via:
--------------------------------------------------------------------
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.4.8.tar.xz
tar xf linux-6.4.8.tar.xz
cd linux-6.4.8
cp -p "/boot/config-$(uname -r)" .config
make oldconfig # and accept all defaults
make -j 12 bindeb-pkg
sudo dpkg -i linux-image-6.4.8_6.4.8-3_amd64.deb
--------------------------------------------------------------------
Here are the errors reported by the kernel:
--------------------------------------------------------------------
[ 2566.222104] BUG: kernel NULL pointer dereference, address:
0000000000000157
[ 2566.222111] #PF: supervisor read access in kernel mode
[ 2566.222114] #PF: error_code(0x0000) - not-present page
[ 2566.222117] PGD 0 P4D 0
[ 2566.222121] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 2566.222125] CPU: 1 PID: 5415 Comm: md10_raid5 Not tainted 6.4.8 #3
[ 2566.222129] Hardware name: ASUS System Product Name/ROG CROSSHAIR VII
HERO (WI-FI), BIOS 4603 09/13/2021
[ 2566.222132] RIP: 0010:submit_bio_noacct+0x182/0x5c0
Can you provide addr2line result? This will be helpful to locate the
problem.
Thanks,
Kuai
[ 2566.222139] Code: ff ff ff 4c 8b 63 48 4d 85 e4 74 0f 48 63 05 e5 ef
41 01 4d 8b a4 c4 d0 00 00 00 41 89 ed 41 83 e5 01 0f 1f 44 00 00 49 63
c5 <41> 80 bc 04 56 01 00 00 00 0f 85 fc 00 00 00 41 80 bc 04 54 01 00
[ 2566.222142] RSP: 0018:ffffa41d46e5bd00 EFLAGS: 00010202
[ 2566.222146] RAX: 0000000000000001 RBX: ffff93275b6668b8 RCX:
0000000000000000
[ 2566.222148] RDX: ffff932741380640 RSI: ffffffffb323f686 RDI:
00000000ffffffff
[ 2566.222151] RBP: 0000000000040001 R08: 0000000000000000 R09:
0000000000000000
[ 2566.222153] R10: 0000000000000001 R11: 0000000000000000 R12:
0000000000000000
[ 2566.222155] R13: 0000000000000001 R14: 000000001dcb2a80 R15:
0000000000000000
[ 2566.222157] FS: 0000000000000000(0000) GS:ffff93363ea40000(0000)
knlGS:0000000000000000
[ 2566.222160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2566.222162] CR2: 0000000000000157 CR3: 0000000140b8e000 CR4:
00000000003506e0
[ 2566.222165] Call Trace:
[ 2566.222167] <TASK>
[ 2566.222171] ? __die+0x23/0x70
[ 2566.222176] ? page_fault_oops+0x17d/0x4c0
[ 2566.222180] ? update_load_avg+0x7e/0x780
[ 2566.222185] ? exc_page_fault+0x7f/0x180
[ 2566.222190] ? asm_exc_page_fault+0x26/0x30
[ 2566.222196] ? submit_bio_noacct+0x182/0x5c0
[ 2566.222201] handle_active_stripes.isra.0+0x377/0x550 [raid456]
[ 2566.222220] raid5d+0x487/0x750 [raid456]
[ 2566.222234] ? __schedule+0x3e7/0xb80
[ 2566.222240] ? _raw_spin_lock_irqsave+0x27/0x60
[ 2566.222245] ? preempt_count_add+0x6e/0xa0
[ 2566.222248] ? _raw_spin_lock_irqsave+0x27/0x60
[ 2566.222254] ? __pfx_md_thread+0x10/0x10 [md_mod]
[ 2566.222273] md_thread+0xae/0x190 [md_mod]
[ 2566.222293] ? __pfx_autoremove_wake_function+0x10/0x10
[ 2566.222299] kthread+0xf7/0x130
[ 2566.222304] ? __pfx_kthread+0x10/0x10
[ 2566.222309] ret_from_fork+0x2c/0x50
[ 2566.222316] </TASK>
[ 2566.222318] Modules linked in: twofish_generic twofish_avx_x86_64
twofish_x86_64_3way twofish_x86_64 twofish_common essiv authenc dm_crypt
cpufreq_conservative cpufreq_userspace cpufreq_powersave rpcsec_gss_krb5
nfsv4 dns_resolver nfs fscache netfs bridge stp llc binfmt_misc amdgpu
eeepc_wmi intel_rapl_msr asus_wmi intel_rapl_common battery edac_mce_amd
hid_pl sparse_keymap platform_profile hid_dr snd_hda_codec_realtek
sp5100_tco drm_buddy rfkill ff_memless gpu_sched drm_suballoc_helper
kvm_amd snd_hda_codec_generic drm_display_helper ledtrig_audio
snd_hda_codec_hdmi cec rc_core drm_ttm_helper kvm snd_hda_intel
snd_intel_dspcfg ttm snd_intel_sdw_acpi asus_wmi_sensors irqbypass
drm_kms_helper snd_hda_codec rapl video acpi_cpufreq snd_hda_core
mxm_wmi pcspkr wmi_bmof k10temp watchdog ccp snd_hwdep rng_core button
sg cpufreq_ondemand lm90 snd_intel8x0 snd_ac97_codec ac97_bus
snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore evdev nfsd
psmouse i2c_dev sidewinder gameport joydev auth_rpcgss parport_pc
nfs_acl ppdev
[ 2566.222390] lockd lp grace parport drm fuse loop efi_pstore dm_mod
configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs
efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c crc32c_generic multipath
linear hid_generic raid0 bcache raid1 md_mod uas usb_storage sd_mod
usbhid crc32_pclmul crc32c_intel t10_pi hid crc64_rocksoft_generic
crc64_rocksoft crc_t10dif crct10dif_generic crct10dif_pclmul crc64
ghash_clmulni_intel crct10dif_common sha512_ssse3 sha512_generic ahci
xhci_pci libahci xhci_hcd aesni_intel crypto_simd libata cryptd usbcore
igb e1000e i2c_piix4 scsi_mod i2c_algo_bit dca usb_common scsi_common
gpio_amdpt wmi gpio_generic
[ 2566.222451] CR2: 0000000000000157
[ 2566.222454] ---[ end trace 0000000000000000 ]---
[ 2566.436029] RIP: 0010:submit_bio_noacct+0x182/0x5c0
[ 2566.436038] Code: ff ff ff 4c 8b 63 48 4d 85 e4 74 0f 48 63 05 e5 ef
41 01 4d 8b a4 c4 d0 00 00 00 41 89 ed 41 83 e5 01 0f 1f 44 00 00 49 63
c5 <41> 80 bc 04 56 01 00 00 00 0f 85 fc 00 00 00 41 80 bc 04 54 01 00
[ 2566.436041] RSP: 0018:ffffa41d46e5bd00 EFLAGS: 00010202
[ 2566.436044] RAX: 0000000000000001 RBX: ffff93275b6668b8 RCX:
0000000000000000
[ 2566.436047] RDX: ffff932741380640 RSI: ffffffffb323f686 RDI:
00000000ffffffff
[ 2566.436049] RBP: 0000000000040001 R08: 0000000000000000 R09:
0000000000000000
[ 2566.436051] R10: 0000000000000001 R11: 0000000000000000 R12:
0000000000000000
[ 2566.436053] R13: 0000000000000001 R14: 000000001dcb2a80 R15:
0000000000000000
[ 2566.436055] FS: 0000000000000000(0000) GS:ffff93363ea40000(0000)
knlGS:0000000000000000
[ 2566.436058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2566.436060] CR2: 0000000000000157 CR3: 0000000140b8e000 CR4:
00000000003506e0
[ 2566.436063] note: md10_raid5[5415] exited with irqs disabled
[ 2566.436109] ------------[ cut here ]------------
[ 2566.436112] WARNING: CPU: 1 PID: 5415 at kernel/exit.c:818
do_exit+0x8ef/0xb20
[ 2566.436119] Modules linked in: twofish_generic twofish_avx_x86_64
twofish_x86_64_3way twofish_x86_64 twofish_common essiv authenc dm_crypt
cpufreq_conservative cpufreq_userspace cpufreq_powersave rpcsec_gss_krb5
nfsv4 dns_resolver nfs fscache netfs bridge stp llc binfmt_misc amdgpu
eeepc_wmi intel_rapl_msr asus_wmi intel_rapl_common battery edac_mce_amd
hid_pl sparse_keymap platform_profile hid_dr snd_hda_codec_realtek
sp5100_tco drm_buddy rfkill ff_memless gpu_sched drm_suballoc_helper
kvm_amd snd_hda_codec_generic drm_display_helper ledtrig_audio
snd_hda_codec_hdmi cec rc_core drm_ttm_helper kvm snd_hda_intel
snd_intel_dspcfg ttm snd_intel_sdw_acpi asus_wmi_sensors irqbypass
drm_kms_helper snd_hda_codec rapl video acpi_cpufreq snd_hda_core
mxm_wmi pcspkr wmi_bmof k10temp watchdog ccp snd_hwdep rng_core button
sg cpufreq_ondemand lm90 snd_intel8x0 snd_ac97_codec ac97_bus
snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore evdev nfsd
psmouse i2c_dev sidewinder gameport joydev auth_rpcgss parport_pc
nfs_acl ppdev
[ 2566.436188] lockd lp grace parport drm fuse loop efi_pstore dm_mod
configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs
efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c crc32c_generic multipath
linear hid_generic raid0 bcache raid1 md_mod uas usb_storage sd_mod
usbhid crc32_pclmul crc32c_intel t10_pi hid crc64_rocksoft_generic
crc64_rocksoft crc_t10dif crct10dif_generic crct10dif_pclmul crc64
ghash_clmulni_intel crct10dif_common sha512_ssse3 sha512_generic ahci
xhci_pci libahci xhci_hcd aesni_intel crypto_simd libata cryptd usbcore
igb e1000e i2c_piix4 scsi_mod i2c_algo_bit dca usb_common scsi_common
gpio_amdpt wmi gpio_generic
[ 2566.436250] CPU: 1 PID: 5415 Comm: md10_raid5 Tainted: G
D 6.4.8 #3
[ 2566.436254] Hardware name: ASUS System Product Name/ROG CROSSHAIR VII
HERO (WI-FI), BIOS 4603 09/13/2021
[ 2566.436256] RIP: 0010:do_exit+0x8ef/0xb20
[ 2566.436260] Code: e9 12 ff ff ff 48 8b bb 98 09 00 00 31 f6 e8 88 d9
ff ff e9 a0 fd ff ff 4c 89 e6 bf 05 06 00 00 e8 f6 0b 01 00 e9 59 f8 ff
ff <0f> 0b e9 88 f7 ff ff 0f 0b e9 45 f7 ff ff 48 89 df e8 fb e0 11 00
[ 2566.436263] RSP: 0018:ffffa41d46e5bed8 EFLAGS: 00010286
[ 2566.436266] RAX: 0000000000000000 RBX: ffff9327df5a6600 RCX:
0000000000000000
[ 2566.436269] RDX: 0000000000000001 RSI: 0000000000002710 RDI:
00000000ffffffff
[ 2566.436271] RBP: ffff9327c0afb600 R08: 0000000000000000 R09:
ffffa41d46e5bde0
[ 2566.436273] R10: 0000000000000003 R11: ffff93363f2f7fe8 R12:
0000000000000009
[ 2566.436275] R13: ffff9327df4deb40 R14: 0000000000000000 R15:
0000000000000000
[ 2566.436277] FS: 0000000000000000(0000) GS:ffff93363ea40000(0000)
knlGS:0000000000000000
[ 2566.436280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2566.436282] CR2: 0000000000000157 CR3: 0000000140b8e000 CR4:
00000000003506e0
[ 2566.436284] Call Trace:
[ 2566.436287] <TASK>
[ 2566.436288] ? do_exit+0x8ef/0xb20
[ 2566.436292] ? __warn+0x81/0x130
[ 2566.436298] ? do_exit+0x8ef/0xb20
[ 2566.436301] ? report_bug+0x191/0x1c0
[ 2566.436308] ? handle_bug+0x3c/0x80
[ 2566.436312] ? exc_invalid_op+0x17/0x70
[ 2566.436316] ? asm_exc_invalid_op+0x1a/0x20
[ 2566.436321] ? do_exit+0x8ef/0xb20
[ 2566.436325] ? do_exit+0x70/0xb20
[ 2566.436329] make_task_dead+0x81/0x170
[ 2566.436333] rewind_stack_and_make_dead+0x17/0x20
[ 2566.436338] RIP: 0000:0x0
[ 2566.436344] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
[ 2566.436346] RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX:
0000000000000000
[ 2566.436349] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[ 2566.436350] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 2566.436352] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 2566.436354] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 2566.436355] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 2566.436359] </TASK>
[ 2566.436361] ---[ end trace 0000000000000000 ]---
--------------------------------------------------------------------
Thank you,
Corey
