On 10/26/22 15:40, Logan Gunthorpe wrote:
Hello, When testing storage platforms that have integrated compression it is useful to use specifically compressible real world data. The existing buffer_compress_percentage option doesn't work well for testing the performance of the compression algorithm portion of the stack seeing it results in random data (which can slow down many compression algorithms) followed by trivially compressible data (which can often be compressed at a higher performance than real world data). Instead of the crude compressibility approximation when using buffer_compress_percentage, it would be useful to specify specific test data for benchmarking purposes. Using real world data or well-known test data like the Calgary Corpus would help to make the benchmarks closer to real world applications. The existing buffer_pattern option would almost work for this, except it is limited to a maximum length of 512B and will be repeated in the IO buffer and thus can't be used for any arbitrary compression size seeing repeated data is easily compressed. This patch set proposes allowing much larger buffer patterns, that may be loaded from a file. The first patch provides the infrastructure for passing arbitrary sized buffers in the client/server protocol (which means the server version number must be bumped), the second patch adds a method for pre-calculating the buffer size in parse_and_fill_pattern(), the third patch supports properly reading large files and the final patch enables dynamic allocation of the buffers themselves. The buffer_pattern will still be truncated to the block size of the IO and repeated for every IO, but this should still be acceptable for testing compression in storage which tends to operate on fixed size chunks anyway. An arbitrary limit on the pattern buffer of 128MB is still enforced to avoid issues that would crop up with the maximum limit of the client/server commands (FIO_SERVER_MAX_CMD_MB). This should be more than enough seeing it isn't useful to specify a buffer greater than the block size and blocksizes that large are not very practical. Thanks, Logan -- Logan Gunthorpe (4): cconv: Support pattern buffers of arbitrary size lib/pattern: Support NULL output buffer in parse_and_fill_pattern() lib/pattern: Support short repeated read calls when loading from file options: Support arbitrarily long pattern buffers cconv.c | 86 +++++++++++++++++++++++++++++++------------ client.c | 17 ++++++--- gclient.c | 12 ++++-- lib/pattern.c | 96 ++++++++++++++++++++++++++++++++++++++++-------- lib/pattern.h | 21 ++++++++--- options.c | 10 ++--- server.c | 23 +++++++----- server.h | 2 +- stat.h | 1 - thread_options.h | 12 +++--- 10 files changed, 207 insertions(+), 73 deletions(-) base-commit: c4704c081a54160621227b42238f6e439c28fba3 -- 2.30.2
I've looked through the patches, done some testing and see no problems apart from the small changes already mentioned.
It would be nice to have an addition to our test suite that covers this functionality.
Vincent
