On 12/08/2015 06:41 AM, Jens Rosenboom wrote:
2015-11-27 17:06 GMT+01:00 Jens Axboe <axboe@xxxxxxxxx>:
On 11/26/2015 08:51 AM, Jens Rosenboom wrote:
From dbd950891a43a7104bce5ec935cd992c7e2b27ff Mon Sep 17 00:00:00 2001
From: Jens Rosenboom <j.rosenboom@xxxxxxxx>
Date: Thu, 26 Nov 2015 16:27:49 +0100
Subject: [PATCH] Add an option to flush RBD after opening
With recent Ceph releases the default value for
rbd cache writethrough until flush
has been changed to true, so the librbd client cache will stay in
writethrough mode for fio.
This patch adds an option to perform an rbd_flush() call after opening
the RBD, causing the client cache to activate writeback mode.
Signed-off-by: Jens Rosenboom <j.rosenboom@xxxxxxxx>
---
This works in my test enviroment and boosts 4k randwrites at iodepth=1
from 100 to 6000.
Still it would probably be good if other people could test this, too.
engines/rbd.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/engines/rbd.c b/engines/rbd.c
index 2be9b55..b3efedf 100644
--- a/engines/rbd.c
+++ b/engines/rbd.c
@@ -30,6 +30,7 @@ struct rbd_options {
char *pool_name;
char *client_name;
int busy_poll;
+ int flush_on_open;
};
static struct fio_option options[] = {
@@ -71,6 +72,16 @@ static struct fio_option options[] = {
.group = FIO_OPT_G_RBD,
},
{
+ .name = "flush_on_open",
+ .lname = "Flush on open",
+ .type = FIO_OPT_BOOL,
+ .help = "Flush on opening the RBD to activate
client cache",
+ .off1 = offsetof(struct rbd_options,
flush_on_open),
+ .def = "0",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_RBD,
+ },
+ {
.name = NULL,
},
};
@@ -140,6 +151,13 @@ static int _fio_rbd_connect(struct thread_data *td)
log_err("rbd_open failed.\n");
goto failed_open;
}
+ if (o->flush_on_open) {
+ r = rbd_flush(rbd->image);
+ if (r < 0) {
+ log_err("rbd_flush failed.\n");
+ goto failed_open;
+ }
+ }
return 0;
Looks good, but we should probably limit this to td_write() being true, and
not do this for read-only opens.
I'm not sure that it will be helpful to make this option more
complicated in behaviour and documentation. I agree that currently it
will only be useful for write jobs, but it should not do any harm for
others. And maybe in the future the behaviour of librbd will also
change for read operations, rather than having to add yet another flag
then, I'd vote for KISS.
I'm all for KISS, but issuing a flush on a read-only opened device is
not a great idea, period. And since the flush-to-enable-write-cache only
has an effect on writes, we should only do it for writes.
We should also rename the option to "write_cache_enable" or something
like that, flush_on_open tells us what a horrible interface rbd is, not
what the option actually does. The fact that this is the control
interface for write caching should not be imposed on the user, since
it's crazy. When you do that rename, the fact that it should only have
an effect on a writeable workload becomes more apparent.
--
Jens Axboe
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
