Hi All,
CCed: Christoph and Robin, although this is not strictly DMA-mapping
related issue...
On 22.09.2022 14:16, Marek Szyprowski wrote:
> On 21.09.2022 06:39, Guenter Roeck wrote:
>> On Tue, Sep 06, 2022 at 03:13:08PM +0900, Masahiro Yamada wrote:
>>> Use the ordinary obj-y to list subdirectories.
>>>
>>> Note1:
>>> GNU Make seems to transform './.modules.order' to '.modules.order'
>>> before matching it against the target pattern. Split ./.modules.order
>>> to a dedicated rule to avoid "doesn't match the target pattern"
>>> warning. [1]
>>>
>>> Note2:
>>> Previously, the link order of lib-y depended on CONFIG_MODULES; lib-y
>>> was linked before drivers-y when CONFIG_MODULES=y, otherwise after
>>> drivers-y. This was a bug of commit 7273ad2b08f8 ("kbuild: link lib-y
>>> objects to vmlinux forcibly when CONFIG_MODULES=y"), but it was not a
>>> big deal after all. Now, libs-y (all objects that come from lib/ and
>>> arch/*/lib/) is linked last, irrespective of CONFIG_MODULES.
>>>
>>> Note3:
>>> Now, the single target build in arch/*/lib/ works correctly. There was
>>> a bug report about this. [2]
>>>
>>> $ make ARCH=arm arch/arm/lib/findbit.o
>>> CALL scripts/checksyscalls.sh
>>> AS arch/arm/lib/findbit.o
>>>
>>> [1]: https://lists.gnu.org/archive/html/bug-make/2022-08/msg00059.html
>>> [2]:
>>> https://lore.kernel.org/linux-kbuild/YvUQOwL6lD4%2F5%2FU6@xxxxxxxxxxxxxxxxxxxxx/
>>>
>>> Signed-off-by: Masahiro Yamada <masahiroy@xxxxxxxxxx>
>> With this patch in place, all parisc images crash during boot with the
>> crash dump below. Bisect on next-20220920 points to this patch.
>> Crash and bisect logs are attached.
>>
>> Loking through boot logs, the same problem (same backtrace) is seen
>> with various boot tests on alpha. There may be more, but -next
>> crashes all over the place right now so it is difficult to determine
>> the platforms affected by a specicfic problem.
>
> I confirm this. I've observed similar issue recently (from time to
> time) on the ARM 32bit based Exynos5250-based Arndale board. 'git
> bisect' lead me also to this change. A short investigation revealed
> that the issue is caused by the NULL sgp->pool in
> lib/sg_pool.c:sg_pool_alloc() function. Sometimes it works fine,
> sometimes not. It looks that there is a race in the sg_pools and scsi
> initialization somehow after this patch.
>
> To confirm that this change is really responsible for the issue I've
> tried to revert it on top of linux-next. This is however hard due to
> the dependencies. Instead I've only did:
>
> # git revert -m1 6fb5a32d95a9b1a9d4a6e135404468a8d74cfde6
>
> on top of the linux next-20220921 and resolved the conflict (trivial
> to resolve). After that the kernel boots fine and I was not able to
> reproduce the issue.
>
> The issue I've observed was:
>
> ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> ata1.00: HPA detected: current 117229295, native 117231408
> ata1.00: ATA-8: Corsair CSSD-F60GB2, 1.1, max UDMA/133
> ata1.00: 117229295 sectors, multi 1: LBA48 NCQ (depth 32)
> ata1.00: configured for UDMA/133
> scsi 0:0:0:0: Direct-Access ATA Corsair CSSD-F60 1.1 PQ: 0
> ANSI: 5
> sd 0:0:0:0: [sda] 117229295 512-byte logical blocks: (60.0 GB/55.9 GiB)
> sd 0:0:0:0: Attached scsi generic sg0 type 0
> sd 0:0:0:0: [sda] Write Protect is off
> sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
> 8<--- cut here ---
> Unable to handle kernel NULL pointer dereference at virtual address
> 00000034
> usb3503 usb-hub: switched to HUB mode
> [00000034] *pgd=00000000
> usb3503 usb-hub: usb3503_probe: probed in hub mode
> UDC core: g_ether: couldn't find an available UDC
>
> Internal error: Oops: 5 [#1] PREEMPT SMP ARM
> Modules linked in:
> CPU: 0 PID: 7 Comm: kworker/0:0H Not tainted
> 6.0.0-rc5-00016-g10d1d4b75525 #12835
> Hardware name: Samsung Exynos (Flattened Device Tree)
> Workqueue: kblockd blk_mq_run_work_fn
> PC is at mempool_alloc+0x48/0x14c
> LR is at mempool_alloc+0x2c/0x14c
> pc : [<c0267a2c>] lr : [<c0267a10>] psr: 40000013
> ...
> Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> Control: 10c5387d Table: 4000406a DAC: 00000051
> ...
> Process kworker/0:0H (pid: 7, stack limit = 0x(ptrval))
> Stack: (0xf0839cc8 to 0xf083a000)
> ...
> mempool_alloc from __sg_alloc_table+0x120/0x160
> __sg_alloc_table from sg_alloc_table_chained+0x5c/0xc0
> sg_alloc_table_chained from scsi_alloc_sgtables+0x5c/0x294
> scsi_alloc_sgtables from sd_init_command+0x118/0x908
> sd_init_command from scsi_queue_rq+0x360/0xbe0
> scsi_queue_rq from blk_mq_dispatch_rq_list+0x1d8/0x8c8
> blk_mq_dispatch_rq_list from blk_mq_do_dispatch_sched+0x2e0/0x338
> blk_mq_do_dispatch_sched from
> __blk_mq_sched_dispatch_requests+0xa8/0x150
> __blk_mq_sched_dispatch_requests from
> blk_mq_sched_dispatch_requests+0x34/0x5c
> blk_mq_sched_dispatch_requests from __blk_mq_run_hw_queue+0x88/0x22c
> __blk_mq_run_hw_queue from process_one_work+0x288/0x778
> process_one_work from worker_thread+0x44/0x504
> worker_thread from kthread+0xf0/0x124
> kthread from ret_from_fork+0x14/0x2c
> Exception stack(0xf0839fb0 to 0xf0839ff8)
> 9fa0: 00000000 00000000 00000000
> 00000000
> 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000
> 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> Code: e30ea764 e3c44d11 e34ca017 e3844a92 (e5953034)
> ---[ end trace 0000000000000000 ]---
>
I've analyzed this issue a bit more and it looks that the $subject patch
just revealed a bug in the lib/sg_pool.c. This looks a bit suspicious
for me:
module_init(sg_pool_init);
After changing the above to:
core_initcall(sg_pool_init);
the issue is gone.
Please confirm that this is a proper way of fixing this issue, so I will
send a final patch.
>>
>> Guenter
>>
>> ---
>> # bad: [ef08d387bbbc20df740ced8caee0ffac835869ac] Add linux-next
>> specific files for 20220920
>> # good: [521a547ced6477c54b4b0cc206000406c221b4d6] Linux 6.0-rc6
>> git bisect start 'HEAD' 'v6.0-rc6'
>> # bad: [df970c033333b10c728198606fe787535e08ab8a] Merge branch
>> 'drm-next' of git://git.freedesktop.org/git/drm/drm.git
>> git bisect bad df970c033333b10c728198606fe787535e08ab8a
>> # bad: [0120f1228dc162c1e00ac24b788a67cc669ff56f] Merge branch
>> 'docs-next' of git://git.lwn.net/linux.git
>> git bisect bad 0120f1228dc162c1e00ac24b788a67cc669ff56f
>> # bad: [ed1b38e88a53a328673e1fffdd6bb69e02c34af1] Merge branch
>> 'for-next' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap.git
>> git bisect bad ed1b38e88a53a328673e1fffdd6bb69e02c34af1
>> # bad: [4dd0700db9ab87d345d989c7589077d858a1f387] Merge branch
>> 'for-next/core' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
>> git bisect bad 4dd0700db9ab87d345d989c7589077d858a1f387
>> # good: [61aa10c4f088adac94f88ba967db44ffab627aef] Merge branch
>> 'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/uml/linux.git
>> git bisect good 61aa10c4f088adac94f88ba967db44ffab627aef
>> # bad: [94c23f8f94b27223a557c88075ea063c70afc0ae] Merge branch
>> 'perf/core' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git
>> git bisect bad 94c23f8f94b27223a557c88075ea063c70afc0ae
>> # good: [6357b3bb80d81a4ad5fe1d15f50ebe5de6c9cab2] perf intel-pt:
>> Support itrace option flag d+e to log on error
>> git bisect good 6357b3bb80d81a4ad5fe1d15f50ebe5de6c9cab2
>> # bad: [d2ed1be371b80f97d2141cfcb0c4ecd02e550180] scripts: remove
>> unused argument 'type'
>> git bisect bad d2ed1be371b80f97d2141cfcb0c4ecd02e550180
>> # good: [a521c97e2a63490c238865763fc86942dce8d6bb] kbuild: rewrite
>> check-local-export in sh/awk
>> git bisect good a521c97e2a63490c238865763fc86942dce8d6bb
>> # bad: [165b718fdd8c5a9165b4485019729c0cd8728120] kbuild: unify two
>> modpost invocations
>> git bisect bad 165b718fdd8c5a9165b4485019729c0cd8728120
>> # bad: [10d1d4b75525f3172c6930fb20445f669762ea95] kbuild: move core-y
>> and drivers-y to ./Kbuild
>> git bisect bad 10d1d4b75525f3172c6930fb20445f669762ea95
>> # good: [fd5f5437264c05b9fc0fff4349f0564f474bdf5e] kbuild: rename
>> modules.order in sub-directories to .modules.order
>> git bisect good fd5f5437264c05b9fc0fff4349f0564f474bdf5e
>> # first bad commit: [10d1d4b75525f3172c6930fb20445f669762ea95]
>> kbuild: move core-y and drivers-y to ./Kbuild
>>
>> ---
>> [ 1.812993] sym53c8xx 0000:00:00.0: enabling SERR and PARITY (0107
>> -> 0147)
>> [ 1.816056] sym0: <895a> rev 0x0 at pci 0000:00:00.0 irq 19
>> [ 1.823635] sym0: PA-RISC Firmware, ID 7, Fast-40, LVD, parity
>> checking
>> [ 1.827770] sym0: SCSI BUS has been reset.
>> [ 1.833713] scsi host0: sym-2.2.3
>> [ 1.845265] Backtrace:
>> [ 1.845750] [<10b83400>] sg_pool_alloc+0xac/0xc0
>> [ 1.846512] [<10ac8ba8>] __sg_alloc_table+0x16c/0x1c0
>> [ 1.846728] [<10b83540>] sg_alloc_table_chained+0x6c/0xec
>> [ 1.846911] [<106ff470>] scsi_alloc_sgtables+0x8c/0x268
>> [ 1.847129] [<10703d5c>] scsi_queue_rq+0xae4/0xb3c
>> [ 1.847296] [<1060f62c>] blk_mq_dispatch_rq_list+0x1e4/0xa2c
>> [ 1.847538] [<10617184>] __blk_mq_sched_dispatch_requests+0xb4/0x188
>> [ 1.847758] [<10617334>] blk_mq_sched_dispatch_requests+0x58/0x84
>> [ 1.847787] [<1060b6bc>] __blk_mq_run_hw_queue+0x78/0x108
>> [ 1.847787] [<1060d968>] __blk_mq_delay_run_hw_queue+0x114/0x1e0
>> [ 1.847787] [<1060dc64>] blk_mq_run_hw_queue+0xb0/0x12c
>> [ 1.847787] [<10617600>] blk_mq_sched_insert_request+0x12c/0x158
>> [ 1.847787] [<1060ad9c>] blk_execute_rq+0x9c/0x194
>> [ 1.847787] [<10700a1c>] __scsi_execute+0x130/0x1f4
>> [ 1.847787] [<10704d70>] scsi_probe_and_add_lun+0x214/0xc60
>> [ 1.847787] [<10706144>] __scsi_scan_target+0x164/0x5b4
>> [ 1.847787] [<10706804>] scsi_scan_host_selected+0x138/0x23c
>> [ 1.847787] [<107069d8>] do_scsi_scan_host+0xd0/0xf0
>> [ 1.847787] [<10706b98>] scsi_scan_host+0x1a0/0x1f0
>> [ 1.847787] [<10716ba0>] sym2_probe+0x824/0x850
>> [ 1.847787] [<10656110>] pci_device_probe+0x9c/0x148
>> [ 1.847787] [<106deee0>] really_probe+0xc4/0x34c
>> [ 1.847787] [<106df1c4>] __driver_probe_device+0x5c/0xbc
>> [ 1.847787] [<106df2c0>] driver_probe_device+0x9c/0x188
>> [ 1.847787] [<106dfb50>] __driver_attach+0xac/0x194
>> [ 1.847787] [<106dc784>] bus_for_each_dev+0x78/0xb8
>> [ 1.847787] [<106de698>] driver_attach+0x28/0x38
>> [ 1.847787] [<106de144>] bus_add_driver+0x198/0x224
>> [ 1.847787] [<106e05e0>] driver_register+0x98/0x168
>> [ 1.847787] [<106555d0>] __pci_register_driver+0x80/0x94
>> [ 1.847787]
>> [ 1.847787]
>> [ 1.847787] Kernel Fault: Code=26 (Data memory access rights trap)
>> at addr 0000002c
>> [ 1.847787] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
>> 6.0.0-rc6-next-20220920-32bit #1
>> [ 1.847787] Hardware name: 9000/778/B160L
>> [ 1.847787]
>> [ 1.847787] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
>> [ 1.847787] PSW: 00000000000001001011111100001111 Not tainted
>> [ 1.847787] r00-03 0004bf0f 1183d400 10b83400 1183d440
>> [ 1.847787] r04-07 00000000 00092820 00092820 00000080
>> [ 1.847787] r08-11 00000000 00000034 00000002 11830000
>> [ 1.847787] r12-15 000001f4 10db0000 1204f128 00000001
>> [ 1.847787] r16-19 11beb200 1204f038 1183d0c8 00000080
>> [ 1.847787] r20-23 00000820 00000001 10b83354 00000000
>> [ 1.847787] r24-27 00000080 00000820 00000000 10e42e18
>> [ 1.847787] r28-31 00092000 0000000f 1183d4c0 10ac8ba8
>> [ 1.847787] sr00-03 00000000 00000000 00000000 00000000
>> [ 1.847787] sr04-07 00000000 00000000 00000000 00000000
>> [ 1.847787]
>> [ 1.847787] IASQ: 00000000 00000000 IAOQ: 102bce5c 102bce60
>> [ 1.847787] IIR: 48960058 ISR: 00000000 IOR: 0000002c
>> [ 1.847787] CPU: 0 CR30: 11830000 CR31: 00000000
>> [ 1.847787] ORIG_R28: 86b51896
>> [ 1.847787] IAOQ[0]: mempool_alloc+0x78/0x1a8
>> [ 1.847787] IAOQ[1]: mempool_alloc+0x7c/0x1a8
>> [ 1.847787] RP(r2): sg_pool_alloc+0xac/0xc0
>> [ 1.847787] Backtrace:
>> [ 1.847787] [<10b83400>] sg_pool_alloc+0xac/0xc0
>> [ 1.847787] [<10ac8ba8>] __sg_alloc_table+0x16c/0x1c0
>> [ 1.847787] [<10b83540>] sg_alloc_table_chained+0x6c/0xec
>> [ 1.847787] [<106ff470>] scsi_alloc_sgtables+0x8c/0x268
>> [ 1.847787] [<10703d5c>] scsi_queue_rq+0xae4/0xb3c
>> [ 1.847787] [<1060f62c>] blk_mq_dispatch_rq_list+0x1e4/0xa2c
>> [ 1.847787] [<10617184>] __blk_mq_sched_dispatch_requests+0xb4/0x188
>> [ 1.847787] [<10617334>] blk_mq_sched_dispatch_requests+0x58/0x84
>> [ 1.847787] [<1060b6bc>] __blk_mq_run_hw_queue+0x78/0x108
>> [ 1.847787] [<1060d968>] __blk_mq_delay_run_hw_queue+0x114/0x1e0
>> [ 1.847787] [<1060dc64>] blk_mq_run_hw_queue+0xb0/0x12c
>> [ 1.847787] [<10617600>] blk_mq_sched_insert_request+0x12c/0x158
>> [ 1.847787] [<1060ad9c>] blk_execute_rq+0x9c/0x194
>> [ 1.847787] [<10700a1c>] __scsi_execute+0x130/0x1f4
>> [ 1.847787] [<10704d70>] scsi_probe_and_add_lun+0x214/0xc60
>> [ 1.847787] [<10706144>] __scsi_scan_target+0x164/0x5b4
>> [ 1.847787] [<10706804>] scsi_scan_host_selected+0x138/0x23c
>> [ 1.847787] [<107069d8>] do_scsi_scan_host+0xd0/0xf0
>> [ 1.847787] [<10706b98>] scsi_scan_host+0x1a0/0x1f0
>> [ 1.847787] [<10716ba0>] sym2_probe+0x824/0x850
>> [ 1.847787] [<10656110>] pci_device_probe+0x9c/0x148
>> [ 1.847787] [<106deee0>] really_probe+0xc4/0x34c
>> [ 1.847787] [<106df1c4>] __driver_probe_device+0x5c/0xbc
>> [ 1.847787] [<106df2c0>] driver_probe_device+0x9c/0x188
>> [ 1.847787] [<106dfb50>] __driver_attach+0xac/0x194
>> [ 1.847787] [<106dc784>] bus_for_each_dev+0x78/0xb8
>> [ 1.847787] [<106de698>] driver_attach+0x28/0x38
>> [ 1.847787] [<106de144>] bus_add_driver+0x198/0x224
>> [ 1.847787] [<106e05e0>] driver_register+0x98/0x168
>> [ 1.847787] [<106555d0>] __pci_register_driver+0x80/0x94
>> [ 1.847787]
>> [ 1.847787] Kernel panic - not syncing: Kernel Fault
>>
>
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
