On Sun, 2012-05-06 at 22:12 +0000, Daniel Pocock wrote: > > On 06/05/12 21:49, Myklebust, Trond wrote: > > On Sun, 2012-05-06 at 21:23 +0000, Daniel Pocock wrote: > >> > >> On 06/05/12 18:23, Myklebust, Trond wrote: > >>> On Sun, 2012-05-06 at 03:00 +0000, Daniel Pocock wrote: > >>>> > >>>> I've been observing some very slow nfs write performance when the server > >>>> has `sync' in /etc/exports > >>>> > >>>> I want to avoid using async, but I have tested it and on my gigabit > >>>> network, it gives almost the same speed as if I was on the server > >>>> itself. (e.g. 30MB/sec to one disk, or less than 1MB/sec to the same > >>>> disk over NFS with `sync') > >>>> > >>>> I'm using Debian 6 with 2.6.38 kernels on client and server, NFSv3 > >>>> > >>>> I've also tried a client running Debian 7/Linux 3.2.0 with both NFSv3 > >>>> and NFSv4, speed is still slow > >>>> > >>>> Looking at iostat on the server, I notice that avgrq-sz = 8 sectors > >>>> (4096 bytes) throughout the write operations > >>>> > >>>> I've tried various tests, e.g. dd a large file, or unpack a tarball with > >>>> many small files, the iostat output is always the same > >>> > >>> Were you using 'conv=sync'? > >> > >> No, it was not using conv=sync, just the vanilla dd: > >> > >> dd if=/dev/zero of=some-fat-file bs=65536 count=65536 > > > > Then the results are not comparable. > > If I run dd with conv=sync on the server, then I still notice that OS > caching plays a factor and write performance just appears really fast > > >>>> Looking at /proc/mounts on the clients, everything looks good, large > >>>> wsize, tcp: > >>>> > >>>> rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.x.x.x,mountvers=3,mountport=58727,mountproto=udp,local_lock=none,addr=192.x.x.x > >>>> 0 0 > >>>> > >>>> and > >>>> rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.x.x.x.,minorversion=0,local_lock=none,addr=192.x.x.x 0 0 > >>>> > >>>> and in /proc/fs/nfs/exports on the server, I have sync and wdelay: > >>>> > >>>> /nfs4/daniel > >>>> 192.168.1.0/24,192.x.x.x(rw,insecure,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9,sec=1) > >>>> /home/daniel > >>>> 192.168.1.0/24,192.x.x.x(rw,root_squash,sync,wdelay,no_subtree_check,uuid=aa2a6f37:9cc94eeb:bcbf983c:d6e041d9) > >>>> > >>>> Can anyone suggest anything else? Or is this really the performance hit > >>>> of `sync'? > >>> > >>> It really depends on your disk setup. Particularly when your filesystem > >>> is using barriers (enabled by default on ext4 and xfs), a lot of raid > >> > >> On the server, I've tried both ext3 and ext4, explicitly changing things > >> like data=writeback,barrier=0, but the problem remains > >> > >> The only thing that made it faster was using hdparm -W1 /dev/sd[ab] to > >> enable the write-back cache on the disk > > > > That should in principle be safe to do as long as you are using > > barrier=1. > > Ok, so the combination of: > > - enable writeback with hdparm > - use ext4 (and not ext3) > - barrier=1 and data=writeback? or data=? > > - is there a particular kernel version (on either client or server side) > that will offer more stability using this combination of features? Not that I'm aware of. As long as you have a kernel > 2.6.29, then LVM should work correctly. The main problem is that some SATA hardware tends to be buggy, defeating the methods used by the barrier code to ensure data is truly on disk. I believe that XFS will therefore actually test the hardware when you mount with write caching and barriers, and should report if the test fails in the syslogs. See http://xfs.org/index.php/XFS_FAQ#Write_barrier_support. > I think there are some other variations of my workflow that I can > attempt too, e.g. I've contemplated compiling C++ code onto a RAM disk > because I don't need to keep the hundreds of object files. You might also consider using something like ccache and set the CCACHE_DIR to a local disk if you have one. > >>> setups really _suck_ at dealing with fsync(). The latter is used every > >> > >> I'm using md RAID1, my setup is like this: > >> > >> 2x 1TB SATA disks ST31000528AS (7200rpm with 32MB cache and NCQ) > >> > >> SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI > >> mode] (rev 40) > >> - not using any of the BIOS softraid stuff > >> > >> Both devices have identical partitioning: > >> 1. 128MB boot > >> 2. md volume (1TB - 128MB) > >> > >> The entire md volume (/dev/md2) is then used as a PV for LVM > >> > >> I do my write tests on a fresh LV with no fragmentation > >> > >>> time the NFS client sends a COMMIT or trunc() instruction, and for > >>> pretty much all file and directory creation operations (you can use > >>> 'nfsstat' to monitor how many such operations the NFS client is sending > >>> as part of your test). > >> > >> I know that my two tests are very different in that way: > >> > >> - dd is just writing one big file, no fsync > >> > >> - unpacking a tarball (or compiling a large C++ project) does a lot of > >> small writes with many fsyncs > >> > >> In both cases, it is slow > >> > >>> Local disk can get away with doing a lot less fsync(), because the cache > >>> consistency guarantees are different: > >>> * in NFS, the server is allowed to crash or reboot without > >>> affecting the client's view of the filesystem. > >>> * in the local file system, the expectation is that on reboot any > >>> data lost is won't need to be recovered (the application will > >>> have used fsync() for any data that does need to be persistent). > >>> Only the disk filesystem structures need to be recovered, and > >>> that is done using the journal (or fsck). > >> > >> > >> Is this an intractable problem though? > >> > >> Or do people just work around this, for example, enable async and > >> write-back cache, and then try to manage the risk by adding a UPS and/or > >> battery backed cache to their RAID setup (to reduce the probability of > >> unclean shutdown)? > > > > It all boils down to what kind of consistency guarantees you are > > comfortable living with. The default NFS server setup offers much > > stronger data consistency guarantees than local disk, and is therefore > > likely to be slower when using cheap hardware. > > > > I'm keen for consistency, because I don't like the idea of corrupting > some source code or a whole git repository for example. > > How did you know I'm using cheap hardware? It is a HP MicroServer, I > even got the £100 cash-back cheque: > > http://www8.hp.com/uk/en/campaign/focus-for-smb/solution.html#/tab2/ > > Seriously though, I've worked with some very large arrays in my business > environment, but I use this hardware at home because of the low noise > and low heat dissipation rather than for saving money, so I would like > to try and get the most out of it if possible and I'm very grateful for > these suggestions. Right. All I'm saying is that when comparing local disk and NFS performance, then make sure that you are doing an apples-to-apples comparison. The main reason for wanting to use NFS in a home setup would usually be in order to simultaneously access the same data through several clients. If that is not a concern, then perhaps transforming your NFS server into an iSCSI target might fit your performance requirements better? -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@xxxxxxxxxx www.netapp.com ��.n��������+%������w��{.n�����{��w��� jg��� �����ݢj����G���� ���j:+v���w�m������w�������h�����٥
