On Thu, 31 Jan 2013 15:17:28 +0530 Suresh Jayaraman <sjayaraman@xxxxxxxx> wrote: > On 01/30/2013 05:59 AM, Jeff Layton wrote: > > On Tue, 29 Jan 2013 17:54:21 +0530 > > Suresh Jayaraman <sjayaraman@xxxxxxxx> wrote: > > > >> Hi all, > >> > >> I'm looking into a report on 3.0 based kernel (plus stable fixes) > >> where writes greater than 64k to a NAS (Hitachi NAS) is failing > >> (simple dd). The problem was not seen with a 2.6.32-ish kernel. > >> Also, note that the problem is not seen with other Servers such > >> as Windows 2003 or Windows 8 Servers. > >> > >> The strace output shows the close() fall fails with -ENOSPC. > >> > >> The relevant cFYI snip > >> > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb: total_len 127044 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb: total_len 8260 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33 > >> Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28 > >> > >> > >> The problem seems to be that during close(), we try to flush the > >> buffers by calling cifs_flush() which in turn will call > >> filemap_write_and_wait() to wait on the pages under writeback to > >> complete. do_writepages() will invoke cifs_writepages() which is > >> perhaps returning -ENOSPC and it is propagated back. > >> > >> There are no quota restrictions or disk space problems. The > >> tcpdump output doesn't show any errors during the write or close. > >> > >> I'm not sure what could be causing the problem and would > >> appreciate any clues or debugging suggestions. > >> > >> Thanks > >> > > > > I'd look for "short" write replies. We have this in > > cifs_writev_callback: > > I think you are spot-on. The packet capture looks little odd: > > Client is requesting a WRITE of 126976 bytes > [DataLengthHigh -> 1 (multiple by 64k), DataLengthLow -> 61440, > WordCount -> 14] > > Server seems to have written only 65536 bytes > [CountHigh -> 1, CountLow -> 0, Remaining -> 65535] > However, [File RW Length] is set to 126976 > > There are many such writes. I don't have the full capture to verify > > Sounds similar to the problem fixed by commit > ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 (Some server not handling large > wsize correctly? > > From: Jeff Layton <jlayton@xxxxxxxxxx> > Date: Tue, 17 Jan 2012 16:08:51 -0500 > Subject: cifs: lower default wsize when unix extensions are not used > Yep, though it's only "fixed" in that that patch makes the defaults match what Windows does. You can still make the wsize larger (up to the 128k limit), but at that point we assume that you know what you're doing. :) > > > if (written < wdata->bytes) > > wdata->result = -ENOSPC; > > > > IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX > > responses, but I remember it not being 100% clear... > > > > Did you mean that the spec suggests that in case of -ENOSPC, the SMB/NT > status code should still be STATUS_SUCCESS and the POSIX error should be > set to -ENOSPC? > > See section 2.2.4.43.2. The "Error Codes" chart shows 2 entries at the top that seem to describe short writes. I'll note that they *do* indicate that the server ought to return a count of 0 in that situation. It does not indicate how the client should interpret a non-zero, but short write count. As Tom suggests, the current behavior in that situation is clearly wrong. We ought to just go back and redirty_page_for_writepage those pages that didn't get written out. If the wsize is larger than the server can handle though, it's likely to make performance suck. Doing writeback may take several passes through writepages since the client will have to go back and fill in the gaps. It would be nice to indicate that to the admin somehow, but I'm not sure how best to do so. -- Jeff Layton <jlayton@xxxxxxxxxx> -- To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
