Hello,
This is v2 of this patchset which includes changes for the feedback
thus far.
When testing storage platforms that have integrated compression it is
useful to use specifically compressible real world data. The existing
buffer_compress_percentage option doesn't work well for testing the
performance of the compression algorithm portion of the stack seeing it
results in random data (which can slow down many compression algorithms)
followed by trivially compressible data (which can often be compressed
at a higher performance than real world data).
Instead of the crude compressibility approximation when using
buffer_compress_percentage, it would be useful to specify specific test
data for benchmarking purposes. Using real world data or well-known
test data like the Calgary Corpus would help to make the benchmarks
closer to real world applications.
The existing buffer_pattern option would almost work for this, except it
is limited to a maximum length of 512B and will be repeated in the
IO buffer and thus can't be used for any arbitrary compression
size seeing repeated data is easily compressed.
This patch set proposes allowing much larger buffer patterns,
that may be loaded from a file. The first patch provides the
infrastructure for passing arbitrary sized buffers in the client/server
protocol (which means the server version number must be bumped), the
second patch adds a method for pre-calculating the buffer size
in parse_and_fill_pattern(), the third patch supports properly
reading large files and the final patch enables dynamic allocation of the
buffers themselves.
The buffer_pattern will still be truncated to the block size of
the IO and repeated for every IO, but this should still be acceptable
for testing compression in storage which tends to operate on
fixed size chunks anyway.
An arbitrary limit on the pattern buffer of 128MB is still enforced to
avoid issues that would crop up with the maximum limit of the
client/server commands (FIO_SERVER_MAX_CMD_MB). This should be more than
enough seeing it isn't useful to specify a buffer greater than the block
size and blocksizes that large are not very practical.
Thanks,
Logan
--
Changes since v1:
- Added comment on the patterns member of struct thread_options_pack,
per Vincent
- Added a test in the suite to test a 16KB buffer and verify pattern
--
Logan Gunthorpe (5):
cconv: Support pattern buffers of arbitrary size
lib/pattern: Support NULL output buffer in parse_and_fill_pattern()
lib/pattern: Support short repeated read calls when loading from file
options: Support arbitrarily long pattern buffers
test: add large pattern test
cconv.c | 86 ++++++++++++++++++++++++++++++-----------
client.c | 17 +++++---
gclient.c | 12 ++++--
lib/pattern.c | 96 ++++++++++++++++++++++++++++++++++++++--------
lib/pattern.h | 21 +++++++---
options.c | 10 ++---
server.c | 23 ++++++-----
server.h | 2 +-
stat.h | 1 -
t/jobs/t0027.fio | 14 +++++++
t/run-fio-tests.py | 29 ++++++++++++++
thread_options.h | 15 +++++---
12 files changed, 253 insertions(+), 73 deletions(-)
create mode 100644 t/jobs/t0027.fio
base-commit: 07c8fe21021681f86fbfd3c3d63b88a5ebd4e557
--
2.30.2
