On Thu, 2024-05-30 at 13:01 +0300, Sagi Grimberg wrote:
>
>
> On 30/05/2024 11:31, Jan Kara wrote:
> > On Thu 30-05-24 10:44:01, Sagi Grimberg wrote:
> > > On 29/05/2024 20:05, Trond Myklebust wrote:
> > > > On Wed, 2024-05-29 at 18:11 +0200, Jan Kara wrote:
> > > > > Hello,
> > > > >
> > > > > so I was investigating why random writes to a large file over
> > > > > NFS got
> > > > > noticeably slower. The workload we use to test this is this
> > > > > fio
> > > > > command:
> > > > >
> > > > > fio --direct=0 --ioengine=sync --thread --directory=/mnt --
> > > > > invalidate=1 \
> > > > > --group_reporting=1 --runtime=300 --fallocate=posix --
> > > > > ramp_time=10 \
> > > > > --name=RandomWrites-async-257024-4k-4 --new_group --
> > > > > rw=randwrite
> > > > > \
> > > > > --size=32000m --numjobs=4 --bs=4k --fsync_on_close=1 --
> > > > > end_fsync=1 \
> > > > > --filename_format='FioWorkloads.$jobnum'
> > > > >
> > > > > Eventually I've tracked down the regression to be caused by
> > > > > 6df25e58532b
> > > > > ("nfs: remove reliance on bdi congestion") which changed the
> > > > > congestion
> > > > > mechanism from a generic bdi congestion handling to NFS
> > > > > private one.
> > > > > Before
> > > > > this commit the fio achieved throughput of 180 MB/s, after
> > > > > this
> > > > > commit only
> > > > > 120 MB/s. Now part of the regression was actually caused by
> > > > > inefficient
> > > > > fsync(2) and the fact that more dirty data was cached at the
> > > > > time of
> > > > > the
> > > > > last fsync after commit 6df25e58532b. After fixing fsync [1],
> > > > > the
> > > > > throughput got to 150 MB/s so better but still not quite the
> > > > > throughput
> > > > > before 6df25e58532b.
> > > > >
> > > > > The reason for remaining regression is that bdi congestion
> > > > > handling
> > > > > was
> > > > > broken and the client had happily ~8GB of outstanding IO
> > > > > against the
> > > > > server
> > > > > despite the congestion limit was 256 MB. The new congestion
> > > > > handling
> > > > > actually works but as a result the server does not have
> > > > > enough dirty
> > > > > data
> > > > > to efficiently operate on and the server disk often gets idle
> > > > > before
> > > > > the
> > > > > client can send more.
> > > > >
> > > > > I wanted to discuss possible solutions here.
> > > > >
> > > > > Generally 256MB is not enough even for consumer grade
> > > > > contemporary
> > > > > disks to
> > > > > max out throughput. There is tunable
> > > > > /proc/sys/fs/nfs/nfs_congestion_kb.
> > > > > If I tweak it to say 1GB, that is enough to give the server
> > > > > enough
> > > > > data to
> > > > > saturate the disk (most of the time) and fio reaches 180MB/s
> > > > > as
> > > > > before
> > > > > commit 6df25e58532b. So one solution to the problem would be
> > > > > to
> > > > > change the
> > > > > default of nfs_congestion_kb to 1GB.
> > > > >
> > > > > Generally the problem with this tuning is that faster disks
> > > > > may need
> > > > > even
> > > > > larger nfs_congestion_kb, the NFS client has no way of
> > > > > knowing what
> > > > > the
> > > > > right value of nfs_congestion_kb is. I personally find the
> > > > > concept of
> > > > > client throttling writes to the server flawed. The *server*
> > > > > should
> > > > > push
> > > > > back (or throttle) if the client is too aggressively pushing
> > > > > out the
> > > > > data
> > > > > and then the client can react to this backpressure. Because
> > > > > only the
> > > > > server
> > > > > knows how much it can handle (also given the load from other
> > > > > clients). And
> > > > > I believe this is actually what is happening in practice
> > > > > (e.g. when I
> > > > > tune
> > > > > nfs_congestion_kb to really high number). So I think even
> > > > > better
> > > > > solution
> > > > > may be to just remove the write congestion handling from the
> > > > > client
> > > > > completely. The history before commit 6df25e58532b, when
> > > > > congestion
> > > > > was
> > > > > effectively ignored, shows that this is unlikely to cause any
> > > > > practical
> > > > > problems. What do people think?
> > > > I think we do still need a mechanism to prevent the client from
> > > > pushing
> > > > more writebacks into the RPC layer when the server throttling
> > > > is
> > > > causing RPC transmission queues to build up. Otherwise we end
> > > > up
> > > > increasing the latency when the application is trying to do
> > > > more I/O to
> > > > pages that are queued up for writeback in the RPC layer (since
> > > > the
> > > > latter will be write locked).
> > > Plus the server is likely serving multiple clients, so removing
> > > any type
> > > of congestion handling from the client may overwhelm the server.
> > I understand this concern but before commit 6df25e58532b we
> > effectively
> > didn't do any throttling for years and nobody complained.
Commit 6df25e58532b doesn't add throttling. It just converts from using
the bdi based mechanism to using a NFS-specific one.
That bdi based throttling mechanism dates at least back to 2007,
although there was code before that dating back to pre-git history.
>
> don't know about the history nor what people could have attributed
> problems.
>
> > So servers
> > apparently know how to cope with clients sending too much IO to
> > them.
>
> not sure how an nfs server would cope with this. nfsv4 can reduce
> slots,
> but not
> sure what nfsv3 server would do...
>
> btw, I think you meant that *slower* devices may need a larger queue
> to
> saturate,
> because if the device is fast, 256MB inflight is probably enough...
> So
> you are solving
> for the "consumer grade contemporary disks".
>
It is hard to do server side congestion control with UDP, since it does
not have a native congestion mechanism to leverage.
However connection based transports are essentially a queuing mechanism
as far as the server is concerned. It can trivially push back on the
client by slowing down the rate at which it pulls RPC calls from the
transport (or stopping them altogether). That's a mechanism that works
just fine for both TCP and RDMA.
Additionally, NFSv4 has the session slot mechanism, and while that can
be used as a throttling mechanism, it is more about providing safe
only-once replay semantics. A prudent server implementation would not
rely on it to replace transport level throttling.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@xxxxxxxxxxxxxxx
