On Wed, 2017-08-16 at 18:18 -0500, Brian King wrote: > On 08/16/2017 12:21 PM, Bart Van Assche wrote: > > On Wed, 2017-08-16 at 22:30 +0530, Abdul Haleem wrote: > > > As of next-20170809, linux-next on powerpc boot hung with below trace > > > message. > > > > > > [ ... ] > > > > > > A bisection resulted in first bad commit (270065e92 - scsi: scsi-mq: > > > Always unprepare ...) in the merge branch 'scsi/for-next' > > > > > > System booted fine when the below commit is reverted: > > > > > > commit 270065e92c317845d69095ec8e3d18616b5b39d5 > > > Author: Bart Van Assche <bart.vanassche@xxxxxxx> > > > Date: Thu Aug 3 14:40:14 2017 -0700 > > > > > > scsi: scsi-mq: Always unprepare before requeuing a request > > > > Hello Brian and Michael, > > > > Do you agree that this probably indicates a bug in the PowerPC block driver > > that is used to access the boot disk? Anyway, since a solution is not yet > > available, I will submit a revert for this patch. > > I've been looking at this a bit, and can recreate the issue, but haven't > got to root cause of the issue as of yet. If I do a sysrq-w while the system is hung > during boot I see this: > > [ 25.561523] Workqueue: events_unbound async_run_entry_fn > [ 25.561527] Call Trace: > [ 25.561529] [c0000001697873f0] [c000000169701600] 0xc000000169701600 (unreliable) > [ 25.561534] [c0000001697875c0] [c00000000001ab78] __switch_to+0x2e8/0x430 > [ 25.561539] [c000000169787620] [c00000000091ccb0] __schedule+0x310/0xa00 > [ 25.561543] [c0000001697876f0] [c00000000091d3e0] schedule+0x40/0xb0 > [ 25.561548] [c000000169787720] [c000000000921e40] schedule_timeout+0x200/0x430 > [ 25.561553] [c000000169787810] [c00000000091db10] io_schedule_timeout+0x30/0x70 > [ 25.561558] [c000000169787840] [c00000000091e978] wait_for_common_io.constprop.3+0x178/0x280 > [ 25.561563] [c0000001697878c0] [c00000000047f7ec] blk_execute_rq+0x7c/0xd0 > [ 25.561567] [c000000169787910] [c000000000614cd0] scsi_execute+0x100/0x230 > [ 25.561572] [c000000169787990] [c00000000060d29c] scsi_report_opcode+0xbc/0x170 > [ 25.561577] [c000000169787a50] [d000000004fe6404] sd_revalidate_disk+0xe04/0x1620 [sd_mod] > [ 25.561583] [c000000169787b80] [d000000004fe6d84] sd_probe_async+0xb4/0x230 [sd_mod] > [ 25.561588] [c000000169787c00] [c00000000010fc44] async_run_entry_fn+0x74/0x210 > [ 25.561593] [c000000169787c90] [c000000000102f48] process_one_work+0x198/0x480 > [ 25.561598] [c000000169787d30] [c0000000001032b8] worker_thread+0x88/0x510 > [ 25.561603] [c000000169787dc0] [c00000000010b030] kthread+0x160/0x1a0 > [ 25.561608] [c000000169787e30] [c00000000000b3a4] ret_from_kernel_thread+0x5c/0xb8 > > I was noticing that we are commonly in scsi_report_opcode. Since ipr RAID arrays don't support > the MAINTENANCE_IN / MI_REPORT_SUPPORTED_OPERATION_CODES, I tried setting sdev->no_report_opcodes = 1 > in ipr's slave configure. This seems to eliminate the boot hang for me, but is only working around > the issue. Since this command is not supported by ipr, it should return with an illegal request. > When I'm hung at this point, there is nothing outstanding to the adapter / driver. I'll continue > debugging... (+linux-scsi) Hello Brian, Is kernel debugging enabled on your test system? Is lockdep enabled? Anyway, stack traces like the above usually mean that a request got stuck in a block or scsi driver (ipr in this case). Information about pending requests, including the SCSI CDB, is available under /sys/kernel/debug/block (see also commit 0eebd005dd07 ("scsi: Implement blk_mq_ops.show_rq()")). Bart.��.n��������+%������w��{.n�����{��w����ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f