Hi Ming Lei, On 24.02.2016 08:59, Ming Lei wrote: > On Wed, Feb 24, 2016 at 10:28 AM, John David Anglin > <dave.anglin@xxxxxxxx> wrote: >> The following block change breaks boot on parisc-linux: >> >> commit 54efd50bfd873e2dbf784e0b21a8027ba4299a3e >> Author: Kent Overstreet <kent.overstreet@xxxxxxxxx> >> Date: Thu Apr 23 22:37:18 2015 -0700 >> >> block: make generic_make_request handle arbitrarily sized bios >> >> The way the block layer is currently written, it goes to great lengths >> to avoid having to split bios; upper layer code (such as bio_add_page()) >> checks what the underlying device can handle and tries to always create >> bios that don't need to be split. >> >> But this approach becomes unwieldy and eventually breaks down with >> stacked devices and devices with dynamic limits, and it adds a lot of >> complexity. If the block layer could split bios as needed, we could >> eliminate a lot of complexity elsewhere - particularly in stacked >> drivers. Code that creates bios can then create whatever size bios are >> convenient, and more importantly stacked drivers don't have to deal with >> both their own bio size limitations and the limitations of the >> (potentially multiple) devices underneath them. In the future this will >> let us delete merge_bvec_fn and a bunch of other code. >> >> We do this by adding calls to blk_queue_split() to the various >> make_request functions that need it - a few can already handle arbitrary >> size bios. Note that we add the call _after_ any call to >> blk_queue_bounce(); this means that blk_queue_split() and >> blk_recalc_rq_segments() don't need to be concerned with bouncing >> affecting segment merging. >> >> Some make_request_fn() callbacks were simple enough to audit and verify >> they don't need blk_queue_split() calls. The skipped ones are: >> >> * nfhd_make_request (arch/m68k/emu/nfblock.c) >> * axon_ram_make_request (arch/powerpc/sysdev/axonram.c) >> * simdisk_make_request (arch/xtensa/platforms/iss/simdisk.c) >> * brd_make_request (ramdisk - drivers/block/brd.c) >> * mtip_submit_request (drivers/block/mtip32xx/mtip32xx.c) >> * loop_make_request >> * null_queue_bio >> * bcache's make_request fns >> >> Some others are almost certainly safe to remove now, but will be left >> for future patches. >> >> Cc: Jens Axboe <axboe@xxxxxxxxx> >> Cc: Christoph Hellwig <hch@xxxxxxxxxxxxx> >> Cc: Al Viro <viro@xxxxxxxxxxxxxxxxxx> >> Cc: Ming Lei <ming.lei@xxxxxxxxxxxxx> >> Cc: Neil Brown <neilb@xxxxxxx> >> Cc: Alasdair Kergon <agk@xxxxxxxxxx> >> Cc: Mike Snitzer <snitzer@xxxxxxxxxx> >> Cc: dm-devel@xxxxxxxxxx >> Cc: Lars Ellenberg <drbd-dev@xxxxxxxxxxxxxxxx> >> Cc: drbd-user@xxxxxxxxxxxxxxxx >> Cc: Jiri Kosina <jkosina@xxxxxxx> >> Cc: Geoff Levand <geoff@xxxxxxxxxxxxx> >> Cc: Jim Paris <jim@xxxxxxxx> >> Cc: Philip Kelleher <pjk1939@xxxxxxxxxxxxxxxxxx> >> Cc: Minchan Kim <minchan@xxxxxxxxxx> >> Cc: Nitin Gupta <ngupta@xxxxxxxxxx> >> Cc: Oleg Drokin <oleg.drokin@xxxxxxxxx> >> Cc: Andreas Dilger <andreas.dilger@xxxxxxxxx> >> Acked-by: NeilBrown <neilb@xxxxxxx> (for the 'md/md.c' bits) >> Acked-by: Mike Snitzer <snitzer@xxxxxxxxxx> >> Reviewed-by: Martin K. Petersen <martin.petersen@xxxxxxxxxx> >> Signed-off-by: Kent Overstreet <kent.overstreet@xxxxxxxxx> >> [dpark: skip more mq-based drivers, resolve merge conflicts, etc.] >> Signed-off-by: Dongsu Park <dpark@xxxxxxxxxx> >> Signed-off-by: Ming Lin <ming.l@xxxxxxxxxxxxxxx> >> Signed-off-by: Jens Axboe <axboe@xxxxxx> >> >> This thread on the linux-parisc has most of the discussion and analysis: >> http://www.spinics.net/lists/linux-parisc/msg06710.html >> >> Essentially, the SCSI layer underestimates the number of sg segments needed and we run off the end of the sg list and crash. >> This happens because the protect bit is ignored. As a result 4.3 and later kernels fail to boot. This includes the current Debian >> kernel for hppa. >> >> Hopefully, the block group can help resolve this issue. We can help with testing if needed. >> > > We fixed several similar bugs, but maybe there is another one, :-( Thanks for your help! > Could you apply the attached debug patch and post the log after the issue is > triggered? Sadly I was not yet able to produce the requested output for you. It seems we have - probably triggered due to the block splitting itself - some kind of memory/stack corruption in here as well. Note, the stack grows upwards(!) on parisc, so maybe local variables get overwritten somehow...? First I applied your patch as is. Here is the system log so far: [ 24.940000] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 25.000000] sd 3:0:6:0: [sdb] 71132960 512-byte logical blocks: (36.4 GB/33.9 GiB) [ 25.092000] sd 3:0:5:0: [sda] Write Protect is off [ 25.152000] sd 3:0:5:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 25.268000] sd 3:0:6:0: [sdb] Write Protect is off [ 25.328000] sd 3:0:6:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 25.452000] sda: sda1 sda2 sda3 < sda5 sda6 > [ 25.508000] sdb: sdb1 sdb2 sdb3 < sdb5 sdb6 > [ 25.584000] scsi_id(112): unaligned access to 0x00000000facac009 at ip=0x000000004100390b [ 25.688000] sd 3:0:6:0: [sdb] Attached SCSI disk [ 25.752000] sd 3:0:5:0: [sda] Attached SCSI disk [ 25.840000] scsi_id(113): unaligned access to 0x00000000fa799009 at ip=0x000000004100390b [ 25.972000] random: nonblocking pool is initialized [ 26.064000] ------------[ cut here ]------------ [ 26.120000] WARNING: at /build/linux-4.4/linux-4.4.2/block/blk-merge.c:484 [ 26.200000] Modules linked in: sd_mod sr_mod cdrom ata_generic ohci_pci ehci_pci ohci_hcd ehci_hcd pata_ns87415 libata sym53c8xx scsi_transport_spi usbcore usb_common scsi_mod tulip [ 26.396000] CPU: 0 PID: 67 Comm: systemd-udevd Not tainted 4.4.0-1-parisc64-smp #7 Debian 4.4.2-2 [ 26.504000] task: 00000000bbe96b18 ti: 00000000bbf00000 task.ti: 00000000bbf00000 [ 26.592000] [ 26.608000] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [ 26.664000] PSW: 00001000000001001111111100001110 Not tainted [ 26.736000] r00-03 000000ff0804ff0e 00000000409e9280 00000000404f22a4 00000000bbf011e0 Here it stops. Usually you would see a stack backtrace now, but as I mentioned above maybe the stack got corrupted and as such your tracing code is not being executed. Then I moved the WARN_ON behind you tracing code. With that I got: [ 25.352000] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 25.444000] sda: sda1 sda2 sda3 < sda5 sda6 > [ 25.496000] sdb: sdb1 sdb2 sdb3 < sdb5 sdb6 > [ 25.568000] sd 3:0:5:0: [sda] Attached SCSI disk [ 25.644000] scsi_id(112): unaligned access to 0x00000000fae80009 at ip=0x000000004100390b [ 25.752000] sd 3:0:6:0: [sdb] Attached SCSI disk [ 25.836000] scsi_id(113): unaligned access to 0x00000000fac76009 at ip=0x000000004100390b [ 25.988000] random: nonblocking pool is initialized [ 26.048000] ------------[ cut here ]------------ [ 26.104000] kernel BUG at /build/linux-4.4/linux-4.4.2/include/linux/scatterlist.h:92! [ 26.196000] CPU: 0 PID: 68 Comm: systemd-udevd Not tainted 4.4.0-1-parisc64-smp #8 Debian 4.4.2-2 [ 26.304000] task: 00000000bbeb40f8 ti: 00000000bbf08000 task.ti: 00000000bbf08000 [ 26.392000] [ 26.412000] YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI [ 26.468000] PSW: 00001000000001001111111100001110 Not tainted [ 26.536000] r00-03 000000ff0804ff0e 00000000bbf091e0 00000000404f228c 00000000bbf091e0 [ 26.632000] r04-07 00000000409b3a80 0000000000005000 0000000000000000 0000000000001000 [ 26.728000] r08-11 000000000000001b 00000000000001b0 00000000bfd6e0f0 00000000bbe15a60 [ 26.824000] r12-15 0000000000000000 0000000000000000 0000000000000003 000000007fa95178 Again, no stack backtrace. Maybe Dave has more luck, otherwise I'll continue to try to get some info. Helge -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
