On Fri, Sep 23, 2022 at 8:39 PM Marek Szyprowski
<m.szyprowski@xxxxxxxxxxx> wrote:
>
> Hi All,
>
> CCed: Christoph and Robin, although this is not strictly DMA-mapping
> related issue...
>
> On 22.09.2022 14:16, Marek Szyprowski wrote:
> > On 21.09.2022 06:39, Guenter Roeck wrote:
> >> On Tue, Sep 06, 2022 at 03:13:08PM +0900, Masahiro Yamada wrote:
> >>> Use the ordinary obj-y to list subdirectories.
> >>>
> >>> Note1:
> >>> GNU Make seems to transform './.modules.order' to '.modules.order'
> >>> before matching it against the target pattern. Split ./.modules.order
> >>> to a dedicated rule to avoid "doesn't match the target pattern"
> >>> warning. [1]
> >>>
> >>> Note2:
> >>> Previously, the link order of lib-y depended on CONFIG_MODULES; lib-y
> >>> was linked before drivers-y when CONFIG_MODULES=y, otherwise after
> >>> drivers-y. This was a bug of commit 7273ad2b08f8 ("kbuild: link lib-y
> >>> objects to vmlinux forcibly when CONFIG_MODULES=y"), but it was not a
> >>> big deal after all. Now, libs-y (all objects that come from lib/ and
> >>> arch/*/lib/) is linked last, irrespective of CONFIG_MODULES.
> >>>
> >>> Note3:
> >>> Now, the single target build in arch/*/lib/ works correctly. There was
> >>> a bug report about this. [2]
> >>>
> >>> $ make ARCH=arm arch/arm/lib/findbit.o
> >>> CALL scripts/checksyscalls.sh
> >>> AS arch/arm/lib/findbit.o
> >>>
> >>> [1]: https://lists.gnu.org/archive/html/bug-make/2022-08/msg00059.html
> >>> [2]:
> >>> https://lore.kernel.org/linux-kbuild/YvUQOwL6lD4%2F5%2FU6@xxxxxxxxxxxxxxxxxxxxx/
> >>>
> >>> Signed-off-by: Masahiro Yamada <masahiroy@xxxxxxxxxx>
> >> With this patch in place, all parisc images crash during boot with the
> >> crash dump below. Bisect on next-20220920 points to this patch.
> >> Crash and bisect logs are attached.
> >>
> >> Loking through boot logs, the same problem (same backtrace) is seen
> >> with various boot tests on alpha. There may be more, but -next
> >> crashes all over the place right now so it is difficult to determine
> >> the platforms affected by a specicfic problem.
> >
> > I confirm this. I've observed similar issue recently (from time to
> > time) on the ARM 32bit based Exynos5250-based Arndale board. 'git
> > bisect' lead me also to this change. A short investigation revealed
> > that the issue is caused by the NULL sgp->pool in
> > lib/sg_pool.c:sg_pool_alloc() function. Sometimes it works fine,
> > sometimes not. It looks that there is a race in the sg_pools and scsi
> > initialization somehow after this patch.
> >
> > To confirm that this change is really responsible for the issue I've
> > tried to revert it on top of linux-next. This is however hard due to
> > the dependencies. Instead I've only did:
> >
> > # git revert -m1 6fb5a32d95a9b1a9d4a6e135404468a8d74cfde6
> >
> > on top of the linux next-20220921 and resolved the conflict (trivial
> > to resolve). After that the kernel boots fine and I was not able to
> > reproduce the issue.
> >
> > The issue I've observed was:
> >
> > ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> > ata1.00: HPA detected: current 117229295, native 117231408
> > ata1.00: ATA-8: Corsair CSSD-F60GB2, 1.1, max UDMA/133
> > ata1.00: 117229295 sectors, multi 1: LBA48 NCQ (depth 32)
> > ata1.00: configured for UDMA/133
> > scsi 0:0:0:0: Direct-Access ATA Corsair CSSD-F60 1.1 PQ: 0
> > ANSI: 5
> > sd 0:0:0:0: [sda] 117229295 512-byte logical blocks: (60.0 GB/55.9 GiB)
> > sd 0:0:0:0: Attached scsi generic sg0 type 0
> > sd 0:0:0:0: [sda] Write Protect is off
> > sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
> > 8<--- cut here ---
> > Unable to handle kernel NULL pointer dereference at virtual address
> > 00000034
> > usb3503 usb-hub: switched to HUB mode
> > [00000034] *pgd=00000000
> > usb3503 usb-hub: usb3503_probe: probed in hub mode
> > UDC core: g_ether: couldn't find an available UDC
> >
> > Internal error: Oops: 5 [#1] PREEMPT SMP ARM
> > Modules linked in:
> > CPU: 0 PID: 7 Comm: kworker/0:0H Not tainted
> > 6.0.0-rc5-00016-g10d1d4b75525 #12835
> > Hardware name: Samsung Exynos (Flattened Device Tree)
> > Workqueue: kblockd blk_mq_run_work_fn
> > PC is at mempool_alloc+0x48/0x14c
> > LR is at mempool_alloc+0x2c/0x14c
> > pc : [<c0267a2c>] lr : [<c0267a10>] psr: 40000013
> > ...
> > Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> > Control: 10c5387d Table: 4000406a DAC: 00000051
> > ...
> > Process kworker/0:0H (pid: 7, stack limit = 0x(ptrval))
> > Stack: (0xf0839cc8 to 0xf083a000)
> > ...
> > mempool_alloc from __sg_alloc_table+0x120/0x160
> > __sg_alloc_table from sg_alloc_table_chained+0x5c/0xc0
> > sg_alloc_table_chained from scsi_alloc_sgtables+0x5c/0x294
> > scsi_alloc_sgtables from sd_init_command+0x118/0x908
> > sd_init_command from scsi_queue_rq+0x360/0xbe0
> > scsi_queue_rq from blk_mq_dispatch_rq_list+0x1d8/0x8c8
> > blk_mq_dispatch_rq_list from blk_mq_do_dispatch_sched+0x2e0/0x338
> > blk_mq_do_dispatch_sched from
> > __blk_mq_sched_dispatch_requests+0xa8/0x150
> > __blk_mq_sched_dispatch_requests from
> > blk_mq_sched_dispatch_requests+0x34/0x5c
> > blk_mq_sched_dispatch_requests from __blk_mq_run_hw_queue+0x88/0x22c
> > __blk_mq_run_hw_queue from process_one_work+0x288/0x778
> > process_one_work from worker_thread+0x44/0x504
> > worker_thread from kthread+0xf0/0x124
> > kthread from ret_from_fork+0x14/0x2c
> > Exception stack(0xf0839fb0 to 0xf0839ff8)
> > 9fa0: 00000000 00000000 00000000
> > 00000000
> > 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > 00000000
> > 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > Code: e30ea764 e3c44d11 e34ca017 e3844a92 (e5953034)
> > ---[ end trace 0000000000000000 ]---
> >
> I've analyzed this issue a bit more and it looks that the $subject patch
> just revealed a bug in the lib/sg_pool.c. This looks a bit suspicious
> for me:
>
> module_init(sg_pool_init);
>
> After changing the above to:
>
> core_initcall(sg_pool_init);
>
> the issue is gone.
>
>
> Please confirm that this is a proper way of fixing this issue, so I will
> send a final patch.
What a coincidence, I just submitted a patch...
>
> >>
> >> Guenter
> >>
> >> ---
> >> # bad: [ef08d387bbbc20df740ced8caee0ffac835869ac] Add linux-next
> >> specific files for 20220920
> >> # good: [521a547ced6477c54b4b0cc206000406c221b4d6] Linux 6.0-rc6
> >> git bisect start 'HEAD' 'v6.0-rc6'
> >> # bad: [df970c033333b10c728198606fe787535e08ab8a] Merge branch
> >> 'drm-next' of git://git.freedesktop.org/git/drm/drm.git
> >> git bisect bad df970c033333b10c728198606fe787535e08ab8a
> >> # bad: [0120f1228dc162c1e00ac24b788a67cc669ff56f] Merge branch
> >> 'docs-next' of git://git.lwn.net/linux.git
> >> git bisect bad 0120f1228dc162c1e00ac24b788a67cc669ff56f
> >> # bad: [ed1b38e88a53a328673e1fffdd6bb69e02c34af1] Merge branch
> >> 'for-next' of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap.git
> >> git bisect bad ed1b38e88a53a328673e1fffdd6bb69e02c34af1
> >> # bad: [4dd0700db9ab87d345d989c7589077d858a1f387] Merge branch
> >> 'for-next/core' of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> >> git bisect bad 4dd0700db9ab87d345d989c7589077d858a1f387
> >> # good: [61aa10c4f088adac94f88ba967db44ffab627aef] Merge branch
> >> 'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/uml/linux.git
> >> git bisect good 61aa10c4f088adac94f88ba967db44ffab627aef
> >> # bad: [94c23f8f94b27223a557c88075ea063c70afc0ae] Merge branch
> >> 'perf/core' of
> >> git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git
> >> git bisect bad 94c23f8f94b27223a557c88075ea063c70afc0ae
> >> # good: [6357b3bb80d81a4ad5fe1d15f50ebe5de6c9cab2] perf intel-pt:
> >> Support itrace option flag d+e to log on error
> >> git bisect good 6357b3bb80d81a4ad5fe1d15f50ebe5de6c9cab2
> >> # bad: [d2ed1be371b80f97d2141cfcb0c4ecd02e550180] scripts: remove
> >> unused argument 'type'
> >> git bisect bad d2ed1be371b80f97d2141cfcb0c4ecd02e550180
> >> # good: [a521c97e2a63490c238865763fc86942dce8d6bb] kbuild: rewrite
> >> check-local-export in sh/awk
> >> git bisect good a521c97e2a63490c238865763fc86942dce8d6bb
> >> # bad: [165b718fdd8c5a9165b4485019729c0cd8728120] kbuild: unify two
> >> modpost invocations
> >> git bisect bad 165b718fdd8c5a9165b4485019729c0cd8728120
> >> # bad: [10d1d4b75525f3172c6930fb20445f669762ea95] kbuild: move core-y
> >> and drivers-y to ./Kbuild
> >> git bisect bad 10d1d4b75525f3172c6930fb20445f669762ea95
> >> # good: [fd5f5437264c05b9fc0fff4349f0564f474bdf5e] kbuild: rename
> >> modules.order in sub-directories to .modules.order
> >> git bisect good fd5f5437264c05b9fc0fff4349f0564f474bdf5e
> >> # first bad commit: [10d1d4b75525f3172c6930fb20445f669762ea95]
> >> kbuild: move core-y and drivers-y to ./Kbuild
> >>
> >> ---
> >> [ 1.812993] sym53c8xx 0000:00:00.0: enabling SERR and PARITY (0107
> >> -> 0147)
> >> [ 1.816056] sym0: <895a> rev 0x0 at pci 0000:00:00.0 irq 19
> >> [ 1.823635] sym0: PA-RISC Firmware, ID 7, Fast-40, LVD, parity
> >> checking
> >> [ 1.827770] sym0: SCSI BUS has been reset.
> >> [ 1.833713] scsi host0: sym-2.2.3
> >> [ 1.845265] Backtrace:
> >> [ 1.845750] [<10b83400>] sg_pool_alloc+0xac/0xc0
> >> [ 1.846512] [<10ac8ba8>] __sg_alloc_table+0x16c/0x1c0
> >> [ 1.846728] [<10b83540>] sg_alloc_table_chained+0x6c/0xec
> >> [ 1.846911] [<106ff470>] scsi_alloc_sgtables+0x8c/0x268
> >> [ 1.847129] [<10703d5c>] scsi_queue_rq+0xae4/0xb3c
> >> [ 1.847296] [<1060f62c>] blk_mq_dispatch_rq_list+0x1e4/0xa2c
> >> [ 1.847538] [<10617184>] __blk_mq_sched_dispatch_requests+0xb4/0x188
> >> [ 1.847758] [<10617334>] blk_mq_sched_dispatch_requests+0x58/0x84
> >> [ 1.847787] [<1060b6bc>] __blk_mq_run_hw_queue+0x78/0x108
> >> [ 1.847787] [<1060d968>] __blk_mq_delay_run_hw_queue+0x114/0x1e0
> >> [ 1.847787] [<1060dc64>] blk_mq_run_hw_queue+0xb0/0x12c
> >> [ 1.847787] [<10617600>] blk_mq_sched_insert_request+0x12c/0x158
> >> [ 1.847787] [<1060ad9c>] blk_execute_rq+0x9c/0x194
> >> [ 1.847787] [<10700a1c>] __scsi_execute+0x130/0x1f4
> >> [ 1.847787] [<10704d70>] scsi_probe_and_add_lun+0x214/0xc60
> >> [ 1.847787] [<10706144>] __scsi_scan_target+0x164/0x5b4
> >> [ 1.847787] [<10706804>] scsi_scan_host_selected+0x138/0x23c
> >> [ 1.847787] [<107069d8>] do_scsi_scan_host+0xd0/0xf0
> >> [ 1.847787] [<10706b98>] scsi_scan_host+0x1a0/0x1f0
> >> [ 1.847787] [<10716ba0>] sym2_probe+0x824/0x850
> >> [ 1.847787] [<10656110>] pci_device_probe+0x9c/0x148
> >> [ 1.847787] [<106deee0>] really_probe+0xc4/0x34c
> >> [ 1.847787] [<106df1c4>] __driver_probe_device+0x5c/0xbc
> >> [ 1.847787] [<106df2c0>] driver_probe_device+0x9c/0x188
> >> [ 1.847787] [<106dfb50>] __driver_attach+0xac/0x194
> >> [ 1.847787] [<106dc784>] bus_for_each_dev+0x78/0xb8
> >> [ 1.847787] [<106de698>] driver_attach+0x28/0x38
> >> [ 1.847787] [<106de144>] bus_add_driver+0x198/0x224
> >> [ 1.847787] [<106e05e0>] driver_register+0x98/0x168
> >> [ 1.847787] [<106555d0>] __pci_register_driver+0x80/0x94
> >> [ 1.847787]
> >> [ 1.847787]
> >> [ 1.847787] Kernel Fault: Code=26 (Data memory access rights trap)
> >> at addr 0000002c
> >> [ 1.847787] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> >> 6.0.0-rc6-next-20220920-32bit #1
> >> [ 1.847787] Hardware name: 9000/778/B160L
> >> [ 1.847787]
> >> [ 1.847787] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
> >> [ 1.847787] PSW: 00000000000001001011111100001111 Not tainted
> >> [ 1.847787] r00-03 0004bf0f 1183d400 10b83400 1183d440
> >> [ 1.847787] r04-07 00000000 00092820 00092820 00000080
> >> [ 1.847787] r08-11 00000000 00000034 00000002 11830000
> >> [ 1.847787] r12-15 000001f4 10db0000 1204f128 00000001
> >> [ 1.847787] r16-19 11beb200 1204f038 1183d0c8 00000080
> >> [ 1.847787] r20-23 00000820 00000001 10b83354 00000000
> >> [ 1.847787] r24-27 00000080 00000820 00000000 10e42e18
> >> [ 1.847787] r28-31 00092000 0000000f 1183d4c0 10ac8ba8
> >> [ 1.847787] sr00-03 00000000 00000000 00000000 00000000
> >> [ 1.847787] sr04-07 00000000 00000000 00000000 00000000
> >> [ 1.847787]
> >> [ 1.847787] IASQ: 00000000 00000000 IAOQ: 102bce5c 102bce60
> >> [ 1.847787] IIR: 48960058 ISR: 00000000 IOR: 0000002c
> >> [ 1.847787] CPU: 0 CR30: 11830000 CR31: 00000000
> >> [ 1.847787] ORIG_R28: 86b51896
> >> [ 1.847787] IAOQ[0]: mempool_alloc+0x78/0x1a8
> >> [ 1.847787] IAOQ[1]: mempool_alloc+0x7c/0x1a8
> >> [ 1.847787] RP(r2): sg_pool_alloc+0xac/0xc0
> >> [ 1.847787] Backtrace:
> >> [ 1.847787] [<10b83400>] sg_pool_alloc+0xac/0xc0
> >> [ 1.847787] [<10ac8ba8>] __sg_alloc_table+0x16c/0x1c0
> >> [ 1.847787] [<10b83540>] sg_alloc_table_chained+0x6c/0xec
> >> [ 1.847787] [<106ff470>] scsi_alloc_sgtables+0x8c/0x268
> >> [ 1.847787] [<10703d5c>] scsi_queue_rq+0xae4/0xb3c
> >> [ 1.847787] [<1060f62c>] blk_mq_dispatch_rq_list+0x1e4/0xa2c
> >> [ 1.847787] [<10617184>] __blk_mq_sched_dispatch_requests+0xb4/0x188
> >> [ 1.847787] [<10617334>] blk_mq_sched_dispatch_requests+0x58/0x84
> >> [ 1.847787] [<1060b6bc>] __blk_mq_run_hw_queue+0x78/0x108
> >> [ 1.847787] [<1060d968>] __blk_mq_delay_run_hw_queue+0x114/0x1e0
> >> [ 1.847787] [<1060dc64>] blk_mq_run_hw_queue+0xb0/0x12c
> >> [ 1.847787] [<10617600>] blk_mq_sched_insert_request+0x12c/0x158
> >> [ 1.847787] [<1060ad9c>] blk_execute_rq+0x9c/0x194
> >> [ 1.847787] [<10700a1c>] __scsi_execute+0x130/0x1f4
> >> [ 1.847787] [<10704d70>] scsi_probe_and_add_lun+0x214/0xc60
> >> [ 1.847787] [<10706144>] __scsi_scan_target+0x164/0x5b4
> >> [ 1.847787] [<10706804>] scsi_scan_host_selected+0x138/0x23c
> >> [ 1.847787] [<107069d8>] do_scsi_scan_host+0xd0/0xf0
> >> [ 1.847787] [<10706b98>] scsi_scan_host+0x1a0/0x1f0
> >> [ 1.847787] [<10716ba0>] sym2_probe+0x824/0x850
> >> [ 1.847787] [<10656110>] pci_device_probe+0x9c/0x148
> >> [ 1.847787] [<106deee0>] really_probe+0xc4/0x34c
> >> [ 1.847787] [<106df1c4>] __driver_probe_device+0x5c/0xbc
> >> [ 1.847787] [<106df2c0>] driver_probe_device+0x9c/0x188
> >> [ 1.847787] [<106dfb50>] __driver_attach+0xac/0x194
> >> [ 1.847787] [<106dc784>] bus_for_each_dev+0x78/0xb8
> >> [ 1.847787] [<106de698>] driver_attach+0x28/0x38
> >> [ 1.847787] [<106de144>] bus_add_driver+0x198/0x224
> >> [ 1.847787] [<106e05e0>] driver_register+0x98/0x168
> >> [ 1.847787] [<106555d0>] __pci_register_driver+0x80/0x94
> >> [ 1.847787]
> >> [ 1.847787] Kernel panic - not syncing: Kernel Fault
> >>
> >
> Best regards
> --
> Marek Szyprowski, PhD
> Samsung R&D Institute Poland
>
--
Best Regards
Masahiro Yamada
