Re: mvsas still has problems with 2.6.34

[Konstantinos Skarlatos <k.skarlatos@xxxxxxxxx>]

 I am another user of mvsas,  attached you can find two recent emails 
with kernel logs that i have sent to the linux-scsi list regarding my 
problems with that driver

Kind regards

On 16/7/2010 12:26 μμ, Thomas Fjellstrom wrote:
> On July 16, 2010, Caspar Smit wrote:
> > > On July 16, 2010, Caspar Smit wrote:
> > > > Thomas,
> > > >
> > > > The patches you are using are the ones from november '09 i presume?
> > > > Those
> > > > patches still had a lot of SATA issues so I think they didn't make the
> > > > kernel. The patches seemed to handle SAS disks just fine though. SATA
> > > > disks was a whole different story.
> > > I'm actually using some that Andy Yan sent me privately, I'm not sure
> > > if they are the same exact ones he sent to linux-scsi. Probably are
> > > though.
> > The november patches were a set of 7 patches where only the first 6
> > needed to be applied.
> Yeah, I was given a zip of the driver a little while before he posted the
> patches to the list.
>
> > > > Srinivas Naga Venkatasatya Pasagadugula created a patch instead of
> > > > Andy Yan's patches which seemed to handle SATA disks a lot better but
> > > > still after some tests it had alot of problems. Srinivas Naga
> > > > Venkatasatya Pasagadugula is now in the process of creating a new
> > > > patch to fix the remaining issues. He told me it would take a long
> > > > time to create those and
> > > > it is now a few months ago since. I and others submitted extensive
> > > > logging
> > > > for him to check.
> > > >
> > > > As for production I could only advise this:
> > > >
> > > > Using SAS disks: Use stock 2.6.34 kernel + Andy Yan's patches
> > > > Using SATA disks: DO NOT GO INTO PRODCUTION.
> > > I've been using the code Andy Yan sent me for 7 months now with 5 SATA
> > > disks
> > > on a md raid5 array. I haven't noticed anything serious in that time.
> > > Prior
> > > to tonight I had been using 2.6.32 for quite some time.
> > >
> > > Maybe the issues only show up with serious load? My raid array doesn't
> > > get hammered, at least not often.
> > The main problem was hotplugging a SATA disk. This results in a kernel
> > panic almost all of the time. There were more issues like the
> > HDIO_GET_IDENTITY failed messages during boot for SATA disks and VERY
> > SLOW xfs creation times.
> I don't recall xfs taking /that/ long for a 4TB fs. With 2.6.34 I don't see
> any HDIO_GET_IDENTITY messages in dmesg. But I'll bet it freaks out if I try
> and hot remove one of the drives. I remember seeing the card lockup, and/or
> the kernel oopsing the last time I tried (2.6.30-2.6.32 time frame).
> Thankfully its not something I often do. While I Can, since I have a hot
> swap unit, its just not something I've had to do yet.
>
> This array has been pretty solid for the past 6 months. Not sure it helps
> but I've been very careful with this machine, it gets shut down
> automatically and safely when the UPS battery gets low, so there hasn't been
> any abrupt shutdowns, except today when a forkbomb hit, and I had to
> SYSRQ+S+U+B the box.
>
> At any rate I can help test whatever new patches might come along.
>
> > Kind regards,
> > Caspar Smit
> >
> > > Thanks
> > >
> > > > Kind regards,
> > > > Caspar Smit
> > > >
> > > > > On July 16, 2010, Thomas Fjellstrom wrote:
> > > > > > On July 16, 2010, Thomas Fjellstrom wrote:
> > > > > > > I've recently updated my server, and the mvsas driver included in
> > > > 2.6.34.1 still causes my AOC-SASLP-MV8 card to completely lock up
> > > >
> > > > > > after
> > > > > >
> > > > > > > mdraid starts up on the devices. The machine is essentially in
> > > > "production" so I can't do a heck of a lot of testing on it anymore.
> > > > The mvsas driver I got from Andy Yan seems to be a little outdated,
> > > > it
> > > >
> > > > > > > fails to compile due to a missing argument to
> > > > sas_change_queue_depth,
> > > > which I managed to fix, and I will try testing. I hope it works.
> > > >
> > > > > > It seems to work with the change I made.
> > > > > Sorry for the noise, I forgot to post the following in my last
> > > > > couple
> > > > messages:
> > > > > It works, but I do get a kernel warning:
> > > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104295] ------------[ cut here
> > > > ]------------
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104315] WARNING: at
> > > > > drivers/ata/libata-core.c:5216 ata_qc_issue+0x31b/0x330 [libata]()
> > > > > Jul
> > > > 16 00:38:05 boris kernel: [   20.104323] Hardware name:
> > > > > GA-MA790FXT-UD5P
> > > > > Jul 16 00:38:05 boris kernel: [   20.104327] Modules linked in:
> > > > > snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep
> > > > snd_pcm_oss
> > > > snd_mixer_oss nouveau ttm snd_pcm drm_kms_helper snd_seq_midi k10temp
> > > > drm
> > > >
> > > > > agpgart i2c_algo_bit snd_rawmidi snd_seq_midi_event i2c_piix4
> > > > > i2c_core
> > > > evdev edac_core edac_mce_amd tpm_tis snd_seq pcspkr tpm button
> > > > tpm_bios wmi snd_timer snd_seq_device processor snd soundcore
> > > > snd_page_alloc ext3 jbd mbcache dm_mod raid1 md_mod sg sr_mod sd_mod
> > > > crc_t10dif cdrom ata_generic ohci_hcd ide_pci_generic ahci mvsas
> > > > libsas libata atiixp scsi_transport_sas firewire_ohci firewire_core
> > > > crc_itu_t thermal skge thermal_sys ide_core ehci_hcd r8169 mii
> > > > usbcore scsi_mod nls_base [last unloaded: scsi_wait_scan]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104448] Pid: 6091, comm: ata_id
> > > > Not
> > > > tainted 2.6.34.1 #2
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104453] Call Trace:
> > > > > Jul 16 00:38:05 boris kernel: [   20.104462]  [<ffffffff81049bb3>] ?
> > > > warn_slowpath_common+0x73/0xb0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104472]  [<ffffffffa011686b>] ?
> > > > ata_qc_issue+0x31b/0x330 [libata]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104482]  [<ffffffffa000ef7f>] ?
> > > > scsi_init_io+0x2f/0x190 [scsi_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104492]  [<ffffffffa011e020>] ?
> > > > ata_scsi_pass_thru+0x0/0x2e0 [libata]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104500]  [<ffffffffa0007990>] ?
> > > > scsi_done+0x0/0x20 [scsi_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104509]  [<ffffffffa011bfae>] ?
> > > > ata_scsi_translate+0x9e/0x180 [libata]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104517]  [<ffffffffa0007990>] ?
> > > > scsi_done+0x0/0x20 [scsi_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104525]  [<ffffffffa015522b>] ?
> > > > sas_queuecommand+0x9b/0x330 [libsas]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104533]  [<ffffffffa0007c7e>] ?
> > > > scsi_dispatch_cmd+0x17e/0x2b0 [scsi_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104542]  [<ffffffffa000e830>] ?
> > > > scsi_request_fn+0x3e0/0x570 [scsi_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104549]  [<ffffffff81058161>] ?
> > > > del_timer+0x71/0xd0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104556]  [<ffffffff811baed3>] ?
> > > > __blk_run_queue+0x63/0x130
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104563]  [<ffffffff811b43a2>] ?
> > > > elv_insert+0x132/0x1f0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104570]  [<ffffffff811bf1c9>] ?
> > > > blk_execute_rq_nowait+0x59/0xb0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104576]  [<ffffffff811bf292>] ?
> > > > blk_execute_rq+0x72/0xe0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104582]  [<ffffffff811bf05b>] ?
> > > > blk_rq_map_user+0x1ab/0x290
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104588]  [<ffffffff811c32f1>] ?
> > > > sg_io+0x241/0x3f0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104594]  [<ffffffff811c38fc>] ?
> > > > scsi_cmd_ioctl+0x45c/0x4b0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104601]  [<ffffffff8110e02f>] ?
> > > > __dentry_open+0x22f/0x340
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104607]  [<ffffffff811195b3>] ?
> > > > inode_permission+0x93/0xd0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.104614]  [<ffffffffa013cdc4>] ?
> > > > sd_ioctl+0xa4/0x120 [sd_mod]
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.105009]  [<ffffffff811c0798>] ?
> > > > __blkdev_driver_ioctl+0x98/0xe0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.105410]  [<ffffffff811c0c75>] ?
> > > > blkdev_ioctl+0x1f5/0x7b0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.105815]  [<ffffffff81113d30>] ?
> > > > cp_new_stat+0xe0/0x100
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.106230]  [<ffffffff8113b4f7>] ?
> > > > block_ioctl+0x37/0x40
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.106647]  [<ffffffff8111e985>] ?
> > > > vfs_ioctl+0x35/0xd0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.107064]  [<ffffffff8111ef08>] ?
> > > > do_vfs_ioctl+0x88/0x560
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.107490]  [<ffffffff8111402e>] ?
> > > > sys_newfstat+0x2e/0x50
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.107919]  [<ffffffff8111f460>] ?
> > > > sys_ioctl+0x80/0xa0
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.108003]  [<ffffffff81002e2b>] ?
> > > > system_call_fastpath+0x16/0x1b
> > > >
> > > > > Jul 16 00:38:05 boris kernel: [   20.108003] ---[ end trace
> > > > > e8ea9c22d6b28439 ]---
> > > > >
> > > > > Other than this stack trace, it seems to work fine.
> > > > >
> > > > > > > At some point though I really hope this gets fixed. I'm still
> > > > willing
> > > > to help test any new versions, just that I can't keep my box down for
> > > > an extended period.
> > > >
> > > > > > > Thanks.
> > > > > I forgot to post, but here are the kernel messages I get when trying
> > > > to
> > > >
> > > > use the kernel's included mvsas driver:
> > > > > Jul 15 22:42:41 boris kernel: [  208.816129] sd 0:0:3:0: [sdf]
> > > > Unhandled
> > > > error code
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.816809] sd 0:0:3:0: [sdf]
> Result:
> > > > hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.817470] sd 0:0:3:0: [sdf] CDB:
> > > > Read(10): 28 00 3a 45 c1 08 00 04 00 00
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.818853] sd 0:0:1:0: [sdd]
> > > > Unhandled
> > > > error code
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.819508] sd 0:0:1:0: [sdd]
> Result:
> > > > hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.820179] sd 0:0:1:0: [sdd] CDB:
> > > > Read(10): 28 00 3a 45 be 58 00 02 b0 00
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.821558] sd 0:0:2:0: [sde]
> > > > Unhandled
> > > > error code
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.822201] sd 0:0:2:0: [sde]
> Result:
> > > > hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.822836] sd 0:0:2:0: [sde] CDB:
> > > > Read(10): 28 00 3a 45 c1 08 00 04 00 00
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.824157] sd 0:0:4:0: [sdg]
> > > > Unhandled
> > > > error code
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.824784] sd 0:0:4:0: [sdg]
> Result:
> > > > hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
> > > >
> > > > > Jul 15 22:42:41 boris kernel: [  208.825407] sd 0:0:4:0: [sdg] CDB:
> > > > Read(10): 28 00 3a 45 c1 08 00 04 00 00
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.737334] md1_raid5     D
> > > > > 0000000000000001     0  6120      2 0x00000000
> > > > > Jul 15 22:43:13 boris kernel: [  240.737948]  ffff88012c94c420
> > > > > 0000000000000046 ffff880100000000 ffff88012f65b680
> > > > > Jul 15 22:43:13 boris kernel: [  240.738570]  00000000000134c0
> > > > > ffff88012e6effd8 00000000000134c0 ffff88012c94c420
> > > > > Jul 15 22:43:13 boris kernel: [  240.739196]  ffff88012e6effd8
> > > > > ffff88012e6effd8 00000000000134c0 00000000000134c0
> > > > > Jul 15 22:43:13 boris kernel: [  240.739821] Call Trace:
> > > > > Jul 15 22:43:13 boris kernel: [  240.740458]  [<ffffffffa018d17e>] ?
> > > > md_super_wait+0xae/0xd0 [md_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.741100]  [<ffffffff810671b0>] ?
> > > > autoremove_wake_function+0x0/0x30
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.741729]  [<ffffffffa018d748>] ?
> > > > md_update_sb+0x268/0x3d0 [md_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.742361]  [<ffffffffa018fcd2>] ?
> > > > md_check_recovery+0x232/0x520 [md_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.742982]  [<ffffffffa0421833>] ?
> > > > raid5d+0x23/0x4f0 [raid456]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.743602]  [<ffffffff8137883d>] ?
> > > > schedule_timeout+0x23d/0x310
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.744221]  [<ffffffff8103aee4>] ?
> > > > finish_task_switch+0x34/0xb0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.744861]  [<ffffffffa018ce43>] ?
> > > > md_thread+0x53/0x120 [md_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.745489]  [<ffffffff810671b0>] ?
> > > > autoremove_wake_function+0x0/0x30
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.746121]  [<ffffffffa018cdf0>] ?
> > > > md_thread+0x0/0x120 [md_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.746743]  [<ffffffff81066c9e>] ?
> > > > kthread+0x8e/0xa0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.747367]  [<ffffffff81003bd4>] ?
> > > > kernel_thread_helper+0x4/0x10
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.748000]  [<ffffffff81066c10>] ?
> > > > kthread+0x0/0xa0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.748639]  [<ffffffff81003bd0>] ?
> > > > kernel_thread_helper+0x0/0x10
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.750521] mount         D
> > > > > 0000000000000001     0  6405   6403 0x00000000
> > > > > Jul 15 22:43:13 boris kernel: [  240.751158]  ffff88012eb8f3d0
> > > > > 0000000000000082 ffff88012e50c600 ffff88012f65d1c0
> > > > > Jul 15 22:43:13 boris kernel: [  240.751805]  00000000000134c0
> > > > > ffff88012dc0bfd8 00000000000134c0 ffff88012eb8f3d0
> > > > > Jul 15 22:43:13 boris kernel: [  240.752452]  ffff88012dc0bfd8
> > > > > ffff88012dc0bfd8 00000000000134c0 00000000000134c0
> > > > > Jul 15 22:43:13 boris kernel: [  240.753108] Call Trace:
> > > > > Jul 15 22:43:13 boris kernel: [  240.753761]  [<ffffffffa0020990>] ?
> > > > scsi_done+0x0/0x20 [scsi_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.754409]  [<ffffffff8137883d>] ?
> > > > schedule_timeout+0x23d/0x310
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.755053]  [<ffffffff811ba097>] ?
> > > > blk_peek_request+0x127/0x1e0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.755708]  [<ffffffffa0020c8d>] ?
> > > > scsi_dispatch_cmd+0x18d/0x2b0 [scsi_mod]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.756358]  [<ffffffff81377af2>] ?
> > > > wait_for_common+0xd2/0x180
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.757023]  [<ffffffff8103da50>] ?
> > > > default_wake_function+0x0/0x20
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.757672]  [<ffffffffa041f486>] ?
> > > > unplug_slaves+0x86/0xc0 [raid456]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.758363]  [<ffffffffa048ed8d>] ?
> > > > xlog_bread_noalign+0xbd/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.759046]  [<ffffffffa04a38c0>] ?
> > > > xfs_buf_iowait+0x40/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.759730]  [<ffffffffa048ed8d>] ?
> > > > xlog_bread_noalign+0xbd/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.760423]  [<ffffffffa048edf5>] ?
> > > > xlog_bread+0x35/0x80 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.761124]  [<ffffffffa0491b9f>] ?
> > > > xlog_find_verify_cycle+0xbf/0x170 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.761813]  [<ffffffffa0492558>] ?
> > > > xlog_find_head+0x168/0x3a0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.762495]  [<ffffffffa04927b7>] ?
> > > > xlog_find_tail+0x27/0x3d0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.763178]  [<ffffffffa0492b75>] ?
> > > > xlog_recover+0x15/0x90 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.763858]  [<ffffffffa048b9c4>] ?
> > > > xfs_log_mount+0x134/0x170 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.764528]  [<ffffffffa0495b8f>] ?
> > > > xfs_mountfs+0x38f/0x720 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.765214]  [<ffffffffa04a090b>] ?
> > > > kmem_alloc+0x7b/0xc0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.765888]  [<ffffffffa04a09fb>] ?
> > > > kmem_zalloc+0x2b/0x40 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.766559]  [<ffffffffa04ad985>] ?
> > > > xfs_fs_fill_super+0x225/0x3b0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.767203]  [<ffffffff81112c03>] ?
> > > > get_sb_bdev+0x1a3/0x1e0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.767877]  [<ffffffffa04ad760>] ?
> > > > xfs_fs_fill_super+0x0/0x3b0 [xfs]
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.768533]  [<ffffffff81112633>] ?
> > > > vfs_kern_mount+0x83/0x1f0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.769174]  [<ffffffff81112813>] ?
> > > > do_kern_mount+0x53/0x120
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.769806]  [<ffffffff8112abfa>] ?
> > > > do_mount+0x28a/0x8a0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.770441]  [<ffffffff81128960>] ?
> > > > copy_mount_options+0xe0/0x180
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.771073]  [<ffffffff8112b2aa>] ?
> > > > sys_mount+0x9a/0xf0
> > > >
> > > > > Jul 15 22:43:13 boris kernel: [  240.771695]  [<ffffffff81002e2b>] ?
> > > > system_call_fastpath+0x16/0x1b
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.769363] md1_raid5     D
> > > > > 0000000000000001     0  6120      2 0x00000000
> > > > > Jul 15 22:45:13 boris kernel: [  360.770006]  ffff88012c94c420
> > > > > 0000000000000046 ffff880100000000 ffff88012f65b680
> > > > > Jul 15 22:45:13 boris kernel: [  360.770648]  00000000000134c0
> > > > > ffff88012e6effd8 00000000000134c0 ffff88012c94c420
> > > > > Jul 15 22:45:13 boris kernel: [  360.771298]  ffff88012e6effd8
> > > > > ffff88012e6effd8 00000000000134c0 00000000000134c0
> > > > > Jul 15 22:45:13 boris kernel: [  360.771946] Call Trace:
> > > > > Jul 15 22:45:13 boris kernel: [  360.772620]  [<ffffffffa018d17e>] ?
> > > > md_super_wait+0xae/0xd0 [md_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.773265]  [<ffffffff810671b0>] ?
> > > > autoremove_wake_function+0x0/0x30
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.773911]  [<ffffffffa018d748>] ?
> > > > md_update_sb+0x268/0x3d0 [md_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.774550]  [<ffffffffa018fcd2>] ?
> > > > md_check_recovery+0x232/0x520 [md_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.775180]  [<ffffffffa0421833>] ?
> > > > raid5d+0x23/0x4f0 [raid456]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.775804]  [<ffffffff8137883d>] ?
> > > > schedule_timeout+0x23d/0x310
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.776424]  [<ffffffff8103aee4>] ?
> > > > finish_task_switch+0x34/0xb0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.777064]  [<ffffffffa018ce43>] ?
> > > > md_thread+0x53/0x120 [md_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.777679]  [<ffffffff810671b0>] ?
> > > > autoremove_wake_function+0x0/0x30
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.778302]  [<ffffffffa018cdf0>] ?
> > > > md_thread+0x0/0x120 [md_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.778919]  [<ffffffff81066c9e>] ?
> > > > kthread+0x8e/0xa0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.779534]  [<ffffffff81003bd4>] ?
> > > > kernel_thread_helper+0x4/0x10
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.780148]  [<ffffffff81066c10>] ?
> > > > kthread+0x0/0xa0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.780776]  [<ffffffff81003bd0>] ?
> > > > kernel_thread_helper+0x0/0x10
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.782623] mount         D
> > > > > 0000000000000001     0  6405   6403 0x00000000
> > > > > Jul 15 22:45:13 boris kernel: [  360.783248]  ffff88012eb8f3d0
> > > > > 0000000000000082 ffff88012e50c600 ffff88012f65d1c0
> > > > > Jul 15 22:45:13 boris kernel: [  360.783883]  00000000000134c0
> > > > > ffff88012dc0bfd8 00000000000134c0 ffff88012eb8f3d0
> > > > > Jul 15 22:45:13 boris kernel: [  360.784536]  ffff88012dc0bfd8
> > > > > ffff88012dc0bfd8 00000000000134c0 00000000000134c0
> > > > > Jul 15 22:45:13 boris kernel: [  360.785184] Call Trace:
> > > > > Jul 15 22:45:13 boris kernel: [  360.785829]  [<ffffffffa0020990>] ?
> > > > scsi_done+0x0/0x20 [scsi_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.786465]  [<ffffffff8137883d>] ?
> > > > schedule_timeout+0x23d/0x310
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.787098]  [<ffffffff811ba097>] ?
> > > > blk_peek_request+0x127/0x1e0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.787740]  [<ffffffffa0020c8d>] ?
> > > > scsi_dispatch_cmd+0x18d/0x2b0 [scsi_mod]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.788361]  [<ffffffff81377af2>] ?
> > > > wait_for_common+0xd2/0x180
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.788988]  [<ffffffff8103da50>] ?
> > > > default_wake_function+0x0/0x20
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.789612]  [<ffffffffa041f486>] ?
> > > > unplug_slaves+0x86/0xc0 [raid456]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.790277]  [<ffffffffa048ed8d>] ?
> > > > xlog_bread_noalign+0xbd/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.790933]  [<ffffffffa04a38c0>] ?
> > > > xfs_buf_iowait+0x40/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.791597]  [<ffffffffa048ed8d>] ?
> > > > xlog_bread_noalign+0xbd/0xf0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.792258]  [<ffffffffa048edf5>] ?
> > > > xlog_bread+0x35/0x80 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.792935]  [<ffffffffa0491b9f>] ?
> > > > xlog_find_verify_cycle+0xbf/0x170 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.793598]  [<ffffffffa0492558>] ?
> > > > xlog_find_head+0x168/0x3a0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.794258]  [<ffffffffa04927b7>] ?
> > > > xlog_find_tail+0x27/0x3d0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.794910]  [<ffffffffa0492b75>] ?
> > > > xlog_recover+0x15/0x90 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.795565]  [<ffffffffa048b9c4>] ?
> > > > xfs_log_mount+0x134/0x170 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.796216]  [<ffffffffa0495b8f>] ?
> > > > xfs_mountfs+0x38f/0x720 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.796879]  [<ffffffffa04a090b>] ?
> > > > kmem_alloc+0x7b/0xc0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.797527]  [<ffffffffa04a09fb>] ?
> > > > kmem_zalloc+0x2b/0x40 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.798171]  [<ffffffffa04ad985>] ?
> > > > xfs_fs_fill_super+0x225/0x3b0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.798785]  [<ffffffff81112c03>] ?
> > > > get_sb_bdev+0x1a3/0x1e0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.799429]  [<ffffffffa04ad760>] ?
> > > > xfs_fs_fill_super+0x0/0x3b0 [xfs]
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.800046]  [<ffffffff81112633>] ?
> > > > vfs_kern_mount+0x83/0x1f0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.800678]  [<ffffffff81112813>] ?
> > > > do_kern_mount+0x53/0x120
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.801292]  [<ffffffff8112abfa>] ?
> > > > do_mount+0x28a/0x8a0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.801910]  [<ffffffff81128960>] ?
> > > > copy_mount_options+0xe0/0x180
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.802531]  [<ffffffff8112b2aa>] ?
> > > > sys_mount+0x9a/0xf0
> > > >
> > > > > Jul 15 22:45:13 boris kernel: [  360.803152]  [<ffffffff81002e2b>] ?
> > > > system_call_fastpath+0x16/0x1b
> > > >
> > > > > I'm pretty sure most of that is due to the driver not responding for
> > > > > 4
> > > > of
> > > >
> > > > > the drives (the first few messages)
> > > > >
> > > > > Thanks again.
> > > > >
> > > > > --
> > > > > Thomas Fjellstrom
> > > > > tfjellstrom@xxxxxxxxxxxxxxx
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe
> > > > > linux-scsi"
> > > > in
> > > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > >
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe
> > > > linux-kernel" in
> > > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > Please read the FAQ at  http://www.tux.org/lkml/
> > > --
> > > Thomas Fjellstrom
> > > tfjellstrom@xxxxxxxxxxxxxxx
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-scsi"
> > > in the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> > in the body of a message to majordomo@xxxxxxxxxxxxxxx
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/

--- Begin Message ---

• To: Jelle de Jong <jelledejong@xxxxxxxxxxxxx>
• Subject: Re: Still havind major MVSAS issues.
• From: Konstantinos Skarlatos <k.skarlatos@xxxxxxxxx>
• Date: Sun, 06 Jun 2010 16:35:03 +0300
• Cc: "Srinivas Naga Venkatasatya Pasagadugula - ERS, HCL Tech" <satyasrinivasp@xxxxxx>, Caspar Smit <c.smit@xxxxxxxxxx>, "linux-scsi@xxxxxxxxxxxxxxx" <linux-scsi@xxxxxxxxxxxxxxx>, "Yugendranath R Singh - ERS, HCL Tech" <yugendranath.s@xxxxxx>, "Geetha Gopalan, ERS-HCLTech" <geethagp@xxxxxx>, "Mohan Kumar TG, ERS,HCLTech" <mohankumartg@xxxxxx>
• In-reply-to: <4C0B90DC.6070800@xxxxxxxxxxxxx>
• User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100526 Thunderbird/3.1

 Hello all,

I have also got severe problems with mvsas, but have managed to at least make it usable. My config is a AOC-SASLP-MV8 with an HP SAS expander, with WD and Seagate sata disks connected to the expander using a norco 4020 case that does not have any on board expanders. Kernel version is 2.6.33 with latest srinivas patch. I also dont raid, the 13 disks have their own filesystem.

My experiences are:

The only filesystem that works is JFS. XFS and btrfs crash the controller making all the disks unreadable and that can only be solved by rebooting. (mkfs succeeds, crashes happen only after mounting or when fscking) JFS works ok as long as i access it via samba or make copies and moves with cp and mv. If I try file operations with Thunar file manager, the controller crashes. Smartctl has not caused any problems so far, and works ok.

Attached are some kernel logs captured from those crashes.

Kind regards,
Konstantinos Skarlatos





On 6/6/2010 3:13 μμ, Jelle de Jong wrote:

Dear Srini,

I spent a few weeks catering information and did some intensive
testing the last few days.

Srinivas Naga Venkatasatya Pasagadugula wrote, on 06-05-10 08:01:

1. Is this is the problem with only WD SATA drives? (I don't have WD SATA drives to reproduce this issue.)
2. Whether the problem is with direct attached SATA drives or drives connected in expanders also?
3. Could you please provide me the "dmesg" log or "/var/log/messages" log.
4. How much the capacity of SATA drives connected to controller?
5. Your HBA is having 6440 chipset?

With a few tricks I managed to boot my OS from the mvsas controller. I
got eleven different sata disks attached from a 4-port mini-sas
backplate without expander to the two Marvel 88SE63xx/64xx mvsas
controllers in my system.

I managed to add five mdadm raid1 arrays without added the actual
active sync device (so one disk for each array) I build my lvm systems
on top of this and did a lot of file transfers for testing.

This worked stable enough, there are HDIO_GET_IDENTITY errors during
boot and operation, but the hard disks seem to be working.

So to debug if issues are related to some brand of harddisk, I started
to add WDC, Hitachi, SAMSUNG, Maxtor and Seagate disks to the
respected raid1 array, the disks are of different sizes (320GB, 500GB,
1TB.)

The sync starts and will fail directly or a while later on failures in
the mvsas driver. I attached an failure example as attachment.

The failures are of grave severity, I have lost complete lvm2 volumes
and raid arrays during testing.

Do you also have a SuperMicro AOC-SASLP-MV8 controller for testing?

I would love to use the controllers for production, but they are
currently unstable. I hope this information helps to solve the mvsas
issues?

With kind regards,

Jelle de Jong

------------[ cut here ]------------
WARNING: at drivers/ata/libata-core.c:5186 ata_qc_issue+0x31f/0x330 [libata]()
Hardware name:
Modules linked in: ipv6 hwmon_vid jfs cpufreq_powersave fan cpufreq_ondemand edac_core powernow_k8 firewire_ohci psmouse firewire_core freq_table serio_raw pcspkr k8temp thermal crc_itu_t evdev edac_mce_amd skge processor button i2c_nforce2 sg forcedeth i2c_core fuse rtc_cmos rtc_core rtc_lib ext2 mbcache dm_crypt dm_mod ses enclosure sd_mod usb_storage ohci_hcd mvsas libsas sata_sil ehci_hcd scsi_transport_sas sata_nv usbcore pata_amd sata_via ata_generic pata_via pata_acpi libata scsi_mod
Pid: 3308, comm: smartctl Not tainted 2.6.33-ARCH #1
Call Trace:
 [<ffffffff810528c8>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8105290f>] warn_slowpath_null+0xf/0x20
 [<ffffffffa002c14f>] ata_qc_issue+0x31f/0x330 [libata]
 [<ffffffffa0006fae>] ? scsi_init_sgtable+0x4e/0x90 [scsi_mod]
 [<ffffffffa0033cd0>] ? ata_scsi_pass_thru+0x0/0x2f0 [libata]
 [<ffffffffa00310c6>] ata_scsi_translate+0xa6/0x180 [libata]
 [<ffffffffa0000b10>] ? scsi_done+0x0/0x20 [scsi_mod]
 [<ffffffffa0000b10>] ? scsi_done+0x0/0x20 [scsi_mod]
 [<ffffffffa0034369>] ata_sas_queuecmd+0x139/0x2b0 [libata]
 [<ffffffffa00f3098>] sas_queuecommand+0x98/0x300 [libsas]
 [<ffffffffa0000c25>] scsi_dispatch_cmd+0xf5/0x230 [scsi_mod]
 [<ffffffffa0006ba2>] scsi_request_fn+0x322/0x3e0 [scsi_mod]
 [<ffffffff811b72bd>] __generic_unplug_device+0x2d/0x40
 [<ffffffff811bcbf8>] blk_execute_rq_nowait+0x68/0xb0
 [<ffffffff811bccc1>] blk_execute_rq+0x81/0xf0
 [<ffffffff811b4d0b>] ? blk_rq_bio_prep+0x2b/0xd0
 [<ffffffff811bc866>] ? blk_rq_map_kern+0xd6/0x150
 [<ffffffffa0007ee7>] scsi_execute+0xf7/0x160 [scsi_mod]
 [<ffffffffa0033167>] ata_cmd_ioctl+0x177/0x320 [libata]
 [<ffffffffa0033467>] ata_sas_scsi_ioctl+0x157/0x2b0 [libata]
 [<ffffffffa00f25f7>] sas_ioctl+0x47/0x50 [libsas]
 [<ffffffffa0002225>] scsi_ioctl+0xd5/0x390 [scsi_mod]
 [<ffffffffa0134d3e>] sd_ioctl+0xce/0xe0 [sd_mod]
 [<ffffffff811be35f>] __blkdev_driver_ioctl+0x8f/0xb0
 [<ffffffff811be82e>] blkdev_ioctl+0x22e/0x820
 [<ffffffff8114fdf7>] block_ioctl+0x37/0x40
 [<ffffffff81131ac8>] vfs_ioctl+0x38/0xd0
 [<ffffffff81131c70>] do_vfs_ioctl+0x80/0x560
 [<ffffffff811cff46>] ? __up_read+0xa6/0xd0
 [<ffffffff81077c29>] ? up_read+0x9/0x10
 [<ffffffff811321d1>] sys_ioctl+0x81/0xa0
 [<ffffffff8100a002>] system_call_fastpath+0x16/0x1b
---[ end trace 115ad6bf347654e7 ]---
 sdm: sdm1
 sdn: sdn1
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
INFO: task smbd:3348 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd          D ffff88000180f948     0  3348   2949 0x00000000
 ffff88002dfe38b8 0000000000000086 0000000000000000 ffffffffa000692b
 000000013d124bd0 0000000000011250 ffff88003caff938 ffff88003cf36690
 000000010061522b ffff88002dfe3fd8 ffff88002dfe2000 ffff88002dfe2000
Call Trace:
 [<ffffffffa000692b>] ? scsi_request_fn+0xab/0x3e0 [scsi_mod]
 [<ffffffff810dbd00>] ? sync_page+0x0/0x50
 [<ffffffff8135970e>] io_schedule+0x6e/0xb0
 [<ffffffff810dbd3d>] sync_page+0x3d/0x50
 [<ffffffff81359d32>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dbce2>] __lock_page+0x62/0x70
 [<ffffffff81073090>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc3d9>] do_read_cache_page+0x159/0x180
 [<ffffffffa02e11e0>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dc434>] read_cache_page_async+0x14/0x20
 [<ffffffff810dc449>] read_cache_page+0x9/0x20
 [<ffffffffa02e1d85>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffffa02d4fb5>] diRead+0x155/0x200 [jfs]
 [<ffffffffa02c8d08>] jfs_iget+0x38/0x160 [jfs]
 [<ffffffffa02cb461>] jfs_lookup+0x71/0x140 [jfs]
 [<ffffffff81110000>] ? calculate_sizes+0x220/0x4a0
 [<ffffffff81359d53>] ? __wait_on_bit_lock+0x73/0xb0
 [<ffffffff8135a45d>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff8112bafb>] do_lookup+0x1db/0x270
 [<ffffffff8112e127>] link_path_walk+0x6b7/0xf10
 [<ffffffff810e1b28>] ? free_hot_page+0x28/0x90
 [<ffffffff8112eb1c>] path_walk+0x5c/0xc0
 [<ffffffff8112ecb3>] do_path_lookup+0x53/0xa0
 [<ffffffff8112f8f2>] user_path_at+0x52/0xa0
 [<ffffffff8115f01e>] ? locks_free_lock+0x3e/0x60
 [<ffffffff8115fb74>] ? fcntl_setlk+0x64/0x350
 [<ffffffff811260a7>] vfs_fstatat+0x37/0x70
 [<ffffffff81126206>] vfs_stat+0x16/0x20
 [<ffffffff8112622f>] sys_newstat+0x1f/0x50
 [<ffffffff81131370>] ? sys_fcntl+0x160/0x5d0
 [<ffffffff8100a002>] system_call_fastpath+0x16/0x1b
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 30 b4 cb ff 00 04 00 00
end_request: I/O error, dev sdc, sector 817155071
sd 10:0:12:0: [sdm] Unhandled error code
sd 10:0:12:0: [sdm] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:12:0: [sdm] CDB: cdb[0]=0x2a: 2a 00 6e 04 66 c8 00 04 00 00
end_request: I/O error, dev sdm, sector 1845782216
sd 10:0:2:0: [sdd] Unhandled error code
sd 10:0:2:0: [sdd] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:2:0: [sdd] CDB: cdb[0]=0x2a: 2a 00 ae 97 38 ff 00 00 08 00
end_request: I/O error, dev sdd, sector 2929146111
sd 10:0:2:0: [sdd] Unhandled error code
sd 10:0:2:0: [sdd] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:2:0: [sdd] CDB: cdb[0]=0x28: 28 00 68 6a 4d 57 00 00 40 00
end_request: I/O error, dev sdd, sector 1751797079
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5

--- End Message ---

--- Begin Message ---

• To: Jelle de Jong <jelledejong@xxxxxxxxxxxxx>
• Subject: Re: Still havind major MVSAS issues.
• From: Konstantinos Skarlatos <k.skarlatos@xxxxxxxxx>
• Date: Sun, 06 Jun 2010 16:43:47 +0300
• Cc: "Srinivas Naga Venkatasatya Pasagadugula - ERS, HCL Tech" <satyasrinivasp@xxxxxx>, Caspar Smit <c.smit@xxxxxxxxxx>, "linux-scsi@xxxxxxxxxxxxxxx" <linux-scsi@xxxxxxxxxxxxxxx>, "Yugendranath R Singh - ERS, HCL Tech" <yugendranath.s@xxxxxx>, "Geetha Gopalan, ERS-HCLTech" <geethagp@xxxxxx>, "Mohan Kumar TG, ERS,HCLTech" <mohankumartg@xxxxxx>
• In-reply-to: <4C0B90DC.6070800@xxxxxxxxxxxxx>
• User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100526 Thunderbird/3.1

Here is one more log with crash information. All of my disks are 1.5TB, Seagates and WD. The expander did not affect the crashes, i had the same issues with direct attached drives.

On 6/6/2010 3:13 μμ, Jelle de Jong wrote:

Dear Srini,

I spent a few weeks catering information and did some intensive
testing the last few days.

Srinivas Naga Venkatasatya Pasagadugula wrote, on 06-05-10 08:01:

1. Is this is the problem with only WD SATA drives? (I don't have WD SATA drives to reproduce this issue.)
2. Whether the problem is with direct attached SATA drives or drives connected in expanders also?
3. Could you please provide me the "dmesg" log or "/var/log/messages" log.
4. How much the capacity of SATA drives connected to controller?
5. Your HBA is having 6440 chipset?

With a few tricks I managed to boot my OS from the mvsas controller. I
got eleven different sata disks attached from a 4-port mini-sas
backplate without expander to the two Marvel 88SE63xx/64xx mvsas
controllers in my system.

I managed to add five mdadm raid1 arrays without added the actual
active sync device (so one disk for each array) I build my lvm systems
on top of this and did a lot of file transfers for testing.

This worked stable enough, there are HDIO_GET_IDENTITY errors during
boot and operation, but the hard disks seem to be working.

So to debug if issues are related to some brand of harddisk, I started
to add WDC, Hitachi, SAMSUNG, Maxtor and Seagate disks to the
respected raid1 array, the disks are of different sizes (320GB, 500GB,
1TB.)

The sync starts and will fail directly or a while later on failures in
the mvsas driver. I attached an failure example as attachment.

The failures are of grave severity, I have lost complete lvm2 volumes
and raid arrays during testing.

Do you also have a SuperMicro AOC-SASLP-MV8 controller for testing?

I would love to use the controllers for production, but they are
currently unstable. I hope this information helps to solve the mvsas
issues?

With kind regards,

Jelle de Jong

drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 1c b2 86 6f 00 03 00 00
end_request: I/O error, dev sdc, sector 481461871
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 03 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442630912
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 ff 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442629888
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 fb 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442628864
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 f7 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442627840
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 f3 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442626816
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 ef 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442625792
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 00 00 c8 c8 00 00 08 00
end_request: I/O error, dev sdf, sector 51400
metapage_write_end_io: I/O error
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 00 00 db 90 00 00 58 00
end_request: I/O error, dev sdf, sector 56208
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
INFO: task filezilla:18572 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla     D ffffffff810dc2d0     0 18572   2063 0x00000000
 ffff8800175b3768 0000000000000082 00000000000530d7 0000000000000008
 0000000000800020 ffff88003ca52cb0 ffff8800175b3798 ffffffff811b6fa4
 ffff8800175b3740 ffff8800175b3fd8 ffff8800175b2000 ffff8800175b2000
Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f87eb>] ? xtLookup+0x18b/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task filezilla:18574 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla     D ffffffff810dc2d0     0 18574   2063 0x00000000
 ffff88003c967768 0000000000000082 0000000000045ca7 0000000000000008
 0000000000800020 ffff88003ca52cb0 ffff88003c967798 ffffffff811b6fa4
 ffff88003c967740 ffff88003c967fd8 ffff88003c966000 ffff88003c966000
Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f86ed>] ? xtLookup+0x8d/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18658 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd          D ffff88000190f948     0 18658  16782 0x00000000
 ffff8800328cd608 0000000000000086 0000000000000000 ffffffff812772b2
 ffff8800328cd5a8 ffffffffa000694a ffff880000000000 ffff88003c921830
 00000001061958d0 ffff8800328cdfd8 ffff8800328cc000 ffff8800328cc000
Call Trace:
 [<ffffffff812772b2>] ? put_device+0x12/0x20
 [<ffffffffa000694a>] ? scsi_request_fn+0xaa/0x4d0 [scsi_mod]
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffffa0309851>] dtSearch+0x781/0xb00 [jfs]
 [<ffffffffa02f7503>] jfs_lookup+0x103/0x150 [jfs]
 [<ffffffff81137648>] ? d_alloc+0x158/0x1c0
 [<ffffffff8112c64d>] __lookup_hash+0xed/0x150
 [<ffffffff8112cb95>] lookup_one_len+0x75/0xb0
 [<ffffffffa03a114d>] vfsub_lookup_one_len+0x1d/0x50 [aufs]
 [<ffffffffa03a7ae2>] au_lkup_one+0x32/0xf0 [aufs]
 [<ffffffff810f9619>] ? handle_mm_fault+0x6d9/0xa30
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff8135d0c5>] ? __down_write_nested+0xd5/0xe0
 [<ffffffffa03a526f>] au_wh_test+0x1f/0xd0 [aufs]
 [<ffffffff8135bbc1>] ? mutex_lock+0x11/0x30
 [<ffffffffa03a8286>] au_lkup_dentry+0x336/0x530 [aufs]
 [<ffffffffa03af3cf>] aufs_lookup+0xef/0x230 [aufs]
 [<ffffffff8112c4cb>] do_lookup+0x1db/0x270
 [<ffffffff8112eb59>] link_path_walk+0x679/0xe50
 [<ffffffff8112f502>] path_walk+0x62/0xe0
 [<ffffffff8112f6c3>] do_path_lookup+0x53/0xa0
 [<ffffffff8112f735>] kern_path+0x25/0x50
 [<ffffffff811cb30b>] ? cpumask_any_but+0x2b/0x40
 [<ffffffff810fdbb0>] ? unmap_region+0x170/0x1a0
 [<ffffffff81151629>] lookup_bdev+0x39/0xb0
 [<ffffffff8112e26e>] ? getname+0x1ce/0x230
 [<ffffffff81174b03>] sys_quotactl+0xb3/0x3d0
 [<ffffffff81077df9>] ? up_write+0x9/0x10
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
sd 10:0:6:0: [sdh] Unhandled error code
sd 10:0:6:0: [sdh] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:6:0: [sdh] CDB: cdb[0]=0x28: 28 00 27 63 5d 6f 00 00 08 00
end_request: I/O error, dev sdh, sector 660823407
metapage_read_end_io: I/O error
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
INFO: task filezilla:18572 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla     D ffffffff810dc2d0     0 18572   2063 0x00000000
 ffff8800175b3768 0000000000000082 00000000000530d7 0000000000000008
 0000000000800020 ffff88003ca52cb0 ffff8800175b3798 ffffffff811b6fa4
 ffff8800175b3740 ffff8800175b3fd8 ffff8800175b2000 ffff8800175b2000
Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f87eb>] ? xtLookup+0x18b/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task filezilla:18574 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla     D ffffffff810dc2d0     0 18574   2063 0x00000000
 ffff88003c967768 0000000000000082 0000000000045ca7 0000000000000008
 0000000000800020 ffff88003ca52cb0 ffff88003c967798 ffffffff811b6fa4
 ffff88003c967740 ffff88003c967fd8 ffff88003c966000 ffff88003c966000
Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f86ed>] ? xtLookup+0x8d/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18659 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd          D ffff88000180f948     0 18659  16782 0x00000000
 ffff8800328e9b18 0000000000000086 0000000000000000 ffff8800328e9db8
 0000000020000000 0000000000000000 0000000000000000 ffff88003c9209f0
 000000010619a86e ffff8800328e9fd8 ffff8800328e8000 ffff8800328e8000
Call Trace:
 [<ffffffff8135d085>] __down_write_nested+0x95/0xe0
 [<ffffffff8135d0db>] __down_write+0xb/0x10
 [<ffffffff8135c6d9>] down_write+0x9/0x10
 [<ffffffffa03a74c7>] di_write_lock+0x27/0x50 [aufs]
 [<ffffffffa03983f6>] aufs_read_lock+0x106/0x110 [aufs]
 [<ffffffff81077e09>] ? up_read+0x9/0x10
 [<ffffffffa03afdd3>] ? aufs_permission+0x313/0x3d0 [aufs]
 [<ffffffffa03a8ae1>] aufs_d_revalidate+0x31/0x450 [aufs]
 [<ffffffff8112eab2>] link_path_walk+0x5d2/0xe50
 [<ffffffff8112f502>] path_walk+0x62/0xe0
 [<ffffffff8112f6c3>] do_path_lookup+0x53/0xa0
 [<ffffffff81130302>] user_path_at+0x52/0xa0
 [<ffffffff8113db14>] ? mntput_no_expire+0x24/0x100
 [<ffffffff811d1276>] ? __up_read+0xa6/0xd0
 [<ffffffff81077e09>] ? up_read+0x9/0x10
 [<ffffffff81126927>] vfs_fstatat+0x37/0x70
 [<ffffffff8113db14>] ? mntput_no_expire+0x24/0x100
 [<ffffffff81126a86>] vfs_stat+0x16/0x20
 [<ffffffff81126aaf>] sys_newstat+0x1f/0x50
 [<ffffffff8112c0ec>] ? path_put+0x2c/0x40
 [<ffffffff81120ee1>] ? sys_chdir+0x51/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18660 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd          D ffff88000190f948     0 18660  16782 0x00000000
 ffff88003a767d18 0000000000000086 0000000000000000 ffff88003a767d08
 000001c73a767cb8 000001c8000001c8 000000083a767dd8 ffff88003c922d90
 000000010619fccf ffff88003a767fd8 ffff88003a766000 ffff88003a766000
Call Trace:
 [<ffffffff8135d175>] __down_read+0x95/0xd6
 [<ffffffff8135c6e9>] down_read+0x9/0x10
 [<ffffffffa03a76b6>] di_read_lock+0x26/0x90 [aufs]
 [<ffffffffa03af5f2>] aufs_getattr+0xe2/0x4a0 [aufs]
 [<ffffffff8113030d>] ? user_path_at+0x5d/0xa0
 [<ffffffff811268bc>] vfs_getattr+0x4c/0x80
 [<ffffffff81126948>] vfs_fstatat+0x58/0x70
 [<ffffffff811269c9>] vfs_lstat+0x19/0x20
 [<ffffffff811269ef>] sys_newlstat+0x1f/0x50
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 12 a0 00 04 00 00
end_request: I/O error, dev sdf, sector 442634912
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 0e a0 00 04 00 00
end_request: I/O error, dev sdf, sector 442633888
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 0b 00 00 03 a0 00
end_request: I/O error, dev sdf, sector 442632960
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 07 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442631936
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 1c b2 87 6f 00 00 08 00
end_request: I/O error, dev sdc, sector 481462127
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5
drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 00 04 5c a7 00 00 08 00
end_request: I/O error, dev sdc, sector 285863
metapage_read_end_io: I/O error
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 00 05 30 d7 00 00 08 00
end_request: I/O error, dev sdc, sector 340183
metapage_read_end_io: I/O error
sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06
sd 10:0:1:0: [sdc] CDB: cdb[0]=0x2a: 2a 00 ae a6 3d 67 00 00 08 00
end_request: I/O error, dev sdc, sector 2930130279

--- End Message ---