2016-01-11 12:09 GMT+01:00 Jens Rosenboom <j.rosenboom@xxxxxxxx>:
> 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@xxxxxxxx>:
>> When running fio-2.3 with the rbd engine, the tests themselves run
>> fine, but instead of the results being printed, I get:
>>
>> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic ==
>> 0x4d555445U' failed.
>>
>> The command I am using is
>>
>> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1
>> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1
>> --bs=1048576B --iodepth=64 --name=test1
>>
>> If I instead use another engine, everything is fine, e.g.:
>>
>> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10
>> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64
>> --name=test1
>>
>> Going back to fio-2.2.13 makes the error go away and running a git
>> bisect leads to the commit mentioned in the subject as the culprit,
>> the error turns into a Segmentation Fault at that point, though.
>
> Accidentally I found out that I can also reproduce the bug with libaio if I set
>
> LD_PRELOAD=../jemalloc/lib/libjemalloc.so
>
> compiled from https://github.com/jemalloc/jemalloc.git (tested with
> 3.6.0 and 4.0.4)
>
> So it seems there is some use-after-free issue in this commit. I'll
> try to dig into this a bit more when I get the time, but I won't be
> sad if someone else is faster. ;)
In the standalone case, sk_out_key is used uninitialized, which may
happen to work fine until some other library also uses a pthread_key,
like it seems both librbd and libjemalloc do.
Attached is a patch that fixes this.
From fd4f2df3243b215622fc35ed7c175c9e6c0d1f62 Mon Sep 17 00:00:00 2001
From: Jens Rosenboom <j.rosenboom@xxxxxxxx>
Date: Mon, 11 Jan 2016 14:41:41 +0100
Subject: [PATCH] Don't use uninitialized pthread_key
When running standalone instead of as a server, sk_out_key doesn't get
initialized, but still will be referenced lateron, which according to
POSIX is unspecified.
Signed-off-by: Jens Rosenboom <j.rosenboom@xxxxxxxx>
---
fio.c | 6 ++++--
server.c | 14 ++++++++++----
server.h | 2 +-
3 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/fio.c b/fio.c
index bd3e260..d59c567 100644
--- a/fio.c
+++ b/fio.c
@@ -56,8 +56,10 @@ int main(int argc, char *argv[], char *envp[])
if (fio_start_all_clients())
goto done;
ret = fio_handle_clients(&fio_client_ops);
- } else
- ret = fio_backend(NULL);
+ } else {
+ if (!server_create_sk_out_key())
+ ret = fio_backend(NULL);
+ }
done:
deinitialize_fio();
diff --git a/server.c b/server.c
index eccd9d3..85174c2 100644
--- a/server.c
+++ b/server.c
@@ -2186,17 +2186,23 @@ static void set_sig_handlers(void)
sigaction(SIGINT, &act, NULL);
}
-static int fio_server(void)
-{
- int sk, ret;
-
+int server_create_sk_out_key(void) {
if (pthread_key_create(&sk_out_key, NULL)) {
log_err("fio: can't create sk_out backend key\n");
return -1;
}
pthread_setspecific(sk_out_key, NULL);
+ return 0;
+}
+static int fio_server(void)
+{
+ int sk, ret;
+
+
+ if (server_create_sk_out_key())
+ return -1;
dprint(FD_NET, "starting server\n");
if (fio_handle_server_arg())
diff --git a/server.h b/server.h
index 9205ae6..3ae55d4 100644
--- a/server.h
+++ b/server.h
@@ -222,5 +222,5 @@ extern int fio_net_send_quit(int sk);
extern int exit_backend;
extern int fio_net_port;
-
+extern int server_create_sk_out_key(void);
#endif
--
2.4.10
