Updated patch attached. It does seem to help - just tried an experiment
dd if=/mnt/test/1GBfile of=/dev/null bs=1M count=1024
to the same server, same share and compared mounting with rasize=6MB
vs. default (1MB to Azure)
(rw,relatime,vers=3.1.1,cache=strict,username=linuxsmb3testsharesmc,uid=0,noforceuid,gid=0,noforcegid,addr=20.150.70.104,file_mode=0777,dir_mode=0777,soft,persistenthandles,nounix,serverino,mapposix,mfsymlinks,nostrictsync,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1,multichannel,max_channels=2)
Got 391 MB/s with rasize=6MB, much faster than default (which ends up
as 1MB with current code) of 163MB/s
# dd if=/mnt/test/394.29520 of=/dev/null bs=1M count=1024 ; dd
if=/mnt/scratch/394.29520 of=/mnt/test/junk1 bs=1M count=1024 ;dd
if=/mnt/test/394.29520 of=/dev/null bs=1M count=1024 ; dd
if=/mnt/scratch/394.29520 of=/mnt/test/junk1 bs=1M count=1024 ;
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.06764 s, 264 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.5912 s, 85.3 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.0573 s, 351 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.58283 s, 125 MB/s
On Sat, Apr 24, 2021 at 9:36 PM Steve French <smfrench@xxxxxxxxx> wrote:
>
> Yep - good catch. It is missing part of my patch :(
>
> Ugh
>
> Will need to rerun and get real numbers
>
> On Sat, Apr 24, 2021 at 9:10 PM Matthew Wilcox <willy@xxxxxxxxxxxxx> wrote:
> >
> > On Sat, Apr 24, 2021 at 02:27:11PM -0500, Steve French wrote:
> > > Using the buildbot test systems, this resulted in an average improvement
> > > of 14% to the Windows server test target for the first 12 tests I
> > > tried (no multichannel)
> > > changing to 12MB rasize (read ahead size). Similarly increasing the
> > > rasize to 12MB to Azure (this time with multichannel, 4 channels)
> > > improved performance 37%
> > >
> > > Note that Ceph had already introduced a mount parameter "rasize" to
> > > allow controlling this. Add mount parameter "rasize" to cifs.ko to
> > > allow control of read ahead (rasize defaults to 4MB which is typically
> > > what it used to default to to the many servers whose rsize was that).
> >
> > I think something was missing from this patch -- I see you parse it and
> > set it in the mount context, but I don't see where it then gets used to
> > actually affect readahead.
>
>
>
> --
> Thanks,
>
> Steve
--
Thanks,
Steve
From 4d7974292cec43309a0aad92f76fde44bb1bafd3 Mon Sep 17 00:00:00 2001
From: Steve French <stfrench@xxxxxxxxxxxxx>
Date: Sat, 24 Apr 2021 21:46:23 -0500
Subject: [PATCH] smb3: add rasize mount parameter to improve readahead
performance
In some cases readahead of more than the read size can help
(to allow parallel i/o of read ahead which can improve performance).
Ceph introduced a mount parameter "rasize" to allow controlling this.
Add mount parameter "rasize" to allow control of amount of readahead
requested of the server. If rasize not set, rasize defaults to
negotiated rsize as before.
Signed-off-by: Steve French <stfrench@xxxxxxxxxxxxx>
---
fs/cifs/cifsfs.c | 9 +++++++--
fs/cifs/fs_context.c | 25 ++++++++++++++++++++++++-
fs/cifs/fs_context.h | 2 ++
3 files changed, 33 insertions(+), 3 deletions(-)
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index 39f4889a036b..5f2c139143a7 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -217,8 +217,11 @@ cifs_read_super(struct super_block *sb)
rc = super_setup_bdi(sb);
if (rc)
goto out_no_root;
- /* tune readahead according to rsize */
- sb->s_bdi->ra_pages = cifs_sb->ctx->rsize / PAGE_SIZE;
+ /* tune readahead according to rsize if readahead size not set on mount */
+ if (cifs_sb->ctx->rasize)
+ sb->s_bdi->ra_pages = cifs_sb->ctx->rasize / PAGE_SIZE;
+ else
+ sb->s_bdi->ra_pages = cifs_sb->ctx->rsize / PAGE_SIZE;
sb->s_blocksize = CIFS_MAX_MSGSIZE;
sb->s_blocksize_bits = 14; /* default 2**14 = CIFS_MAX_MSGSIZE */
@@ -649,6 +652,8 @@ cifs_show_options(struct seq_file *s, struct dentry *root)
seq_printf(s, ",rsize=%u", cifs_sb->ctx->rsize);
seq_printf(s, ",wsize=%u", cifs_sb->ctx->wsize);
seq_printf(s, ",bsize=%u", cifs_sb->ctx->bsize);
+ if (cifs_sb->ctx->rasize)
+ seq_printf(s, ",rasize=%u", cifs_sb->ctx->rasize);
if (tcon->ses->server->min_offload)
seq_printf(s, ",esize=%u", tcon->ses->server->min_offload);
seq_printf(s, ",echo_interval=%lu",
diff --git a/fs/cifs/fs_context.c b/fs/cifs/fs_context.c
index 74758e954035..3e0d016849e3 100644
--- a/fs/cifs/fs_context.c
+++ b/fs/cifs/fs_context.c
@@ -137,6 +137,7 @@ const struct fs_parameter_spec smb3_fs_parameters[] = {
fsparam_u32("min_enc_offload", Opt_min_enc_offload),
fsparam_u32("esize", Opt_min_enc_offload),
fsparam_u32("bsize", Opt_blocksize),
+ fsparam_u32("rasize", Opt_rasize),
fsparam_u32("rsize", Opt_rsize),
fsparam_u32("wsize", Opt_wsize),
fsparam_u32("actimeo", Opt_actimeo),
@@ -941,6 +942,26 @@ static int smb3_fs_context_parse_param(struct fs_context *fc,
ctx->bsize = result.uint_32;
ctx->got_bsize = true;
break;
+ case Opt_rasize:
+ /*
+ * readahead size realistically should never need to be
+ * less than 1M (CIFS_DEFAULT_IOSIZE) or greater than 32M
+ * (perhaps an exception should be considered in the
+ * for the case of a large number of channels
+ * when multichannel is negotiated) since that would lead
+ * to plenty of parallel I/O in flight to the server.
+ * Note that smaller read ahead sizes would
+ * hurt performance of common tools like cp and scp
+ * which often trigger sequential i/o with read ahead
+ */
+ if ((result.uint_32 > (8 * SMB3_DEFAULT_IOSIZE)) ||
+ (result.uint_32 < CIFS_DEFAULT_IOSIZE)) {
+ cifs_errorf(fc, "%s: Invalid rasize %d vs. %d\n",
+ __func__, result.uint_32, SMB3_DEFAULT_IOSIZE);
+ goto cifs_parse_mount_err;
+ }
+ ctx->rasize = result.uint_32;
+ break;
case Opt_rsize:
ctx->rsize = result.uint_32;
ctx->got_rsize = true;
@@ -1377,7 +1398,9 @@ int smb3_init_fs_context(struct fs_context *fc)
ctx->cred_uid = current_uid();
ctx->linux_uid = current_uid();
ctx->linux_gid = current_gid();
- ctx->bsize = 1024 * 1024; /* can improve cp performance significantly */
+ /* By default 4MB read ahead size, 1MB block size */
+ ctx->bsize = CIFS_DEFAULT_IOSIZE; /* can improve cp performance significantly */
+ ctx->rasize = 0; /* 0 = use default (ie negotiated rsize) for read ahead pages */
/*
* default to SFM style remapping of seven reserved characters
diff --git a/fs/cifs/fs_context.h b/fs/cifs/fs_context.h
index 56d7a75e2390..2a71c8e411ac 100644
--- a/fs/cifs/fs_context.h
+++ b/fs/cifs/fs_context.h
@@ -120,6 +120,7 @@ enum cifs_param {
Opt_dirmode,
Opt_min_enc_offload,
Opt_blocksize,
+ Opt_rasize,
Opt_rsize,
Opt_wsize,
Opt_actimeo,
@@ -235,6 +236,7 @@ struct smb3_fs_context {
/* reuse existing guid for multichannel */
u8 client_guid[SMB2_CLIENT_GUID_SIZE];
unsigned int bsize;
+ unsigned int rasize;
unsigned int rsize;
unsigned int wsize;
unsigned int min_offload;
--
2.27.0
