This patch makes fio --client JSON output work better when many server threads are used by removing interleaved output from fio servers with --output-format=json .
Background: Red Hat is testing large numbers of OpenStack guests with Cinder block devices backed by Ceph storage using fio. Originally we were using ssh to start independent fio processes on the guests, but we ran into problems with threads not starting and stopping at the same time, despite using options to use fixed elapsed time for all threads. But when we started using --client, we noticed that thread time skew is greatly reduced, because fio is *already running* on the server and we only have to connect to it and start the test. This is great, but for large numbers of fio servers, it becomes impossible to parse the output.
For example, here's a sample of per-VM JSON output:
Disk stats (read/write):
<vm4> {
<vm4> "fio version" : "fio-2.2.4",
<vm4> "timestamp" : 1421677173,
<vm4> "time" : "Mon Jan 19 09:19:33 2015",
<vm4> "jobs" : [
<vm4>
<vm4> ],
<vm4> "disk_util" : [
...
<vm4> ]
<vm4> }
https://s3.amazonaws.com/ben.england/client.c.diff
https://s3.amazonaws.com/ben.england/fio-results.2015-01-20-11-50.json.gz
The above output works fine for a few threads and is not hard to filter out, but the various guests' output sections get mixed together for large thread counts such as 256 client threads. if you look at the previous URL fio-results*json.gz containing garbled output, the cause of this output is the handle_text() routine, which is called by fio whenever a FIO_NET_CMD_TEXT PDU is received by the fio client (in fio_handle_client()). If you use --debug=all you can see that these PDUs are processed in whatever order they are received from the fio server processes, essentially in random order. But almost all of this info is in the final json object output at the end of the file (except for the timestamp field which I wish was there!) so there is no reason to output twice, except as a debugging tool perhaps. The patch is to V2.2.4.
[ben@localhost fio-master]$ diff -u client.{c.sav2,c}
--- client.c.sav2 2015-01-28 11:30:02.657073505 -0500
+++ client.c 2015-01-28 11:33:02.248373685 -0500
@@ -968,12 +968,6 @@
const char *name;
int fio_unused ret;
- name = client->name ? client->name : client->hostname;
-
- if (!client->skip_newline)
- fprintf(f_out, "<%s> ", name);
- ret = fwrite(buf, pdu->buf_len, 1, f_out);
- fflush(f_out);
client->skip_newline = strchr(buf, '\n') == NULL;
}
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
