Hi,
I used https://github.com/dvassallo/s3-benchmark to measure some performance values for the rgws and got some unexpected results.
Everything above 64K has excellent performance but below it drops down to a fraction of the speed and responsiveness resulting in even 256K objects being faster than anything below 64K.
Does anyone observe similar effects while running this benchmark? Is it the benchmarks fault or are there some options to tweak performance for low object sizes?
s3-benchmark results below.
Best,
Christian
$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=4 -payloads-max=15 -samples=1000 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
--- SETUP --------------------------------------------------------------------------------------------------------------------
Uploading 8 KB objects
100% |████████████████████████████████████████| [0s:0s]
[...munged...]
--- BENCHMARK ----------------------------------------------------------------------------------------------------------------
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 1.8 MB/s | 32 1 7 41 42 42 42 43 | 32 1 7 41 42 42 42 43 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 16 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 3.7 MB/s | 30 1 5 41 41 42 43 45 | 30 2 5 41 41 42 43 45 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 6.3 MB/s | 34 2 41 41 41 42 43 43 | 35 2 41 41 42 42 43 43 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 156.2 MB/s | 2 1 2 2 3 4 5 5 | 3 2 2 3 3 5 5 6 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 128 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 238.8 MB/s | 2 1 2 2 3 4 5 5 | 4 3 3 4 4 5 7 7 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 256 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 346.0 MB/s | 3 1 2 2 3 5 6 7 | 5 3 4 5 6 8 10 10 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 512 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 427.8 MB/s | 4 2 2 3 4 7 9 9 | 9 4 6 9 10 13 16 16 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 1 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 631.6 MB/s | 4 2 3 4 5 8 12 14 | 12 6 9 11 14 19 26 26 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 2 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 707.3 MB/s | 7 3 5 6 7 13 19 20 | 22 8 16 20 23 34 44 51 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 4 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 519.0 MB/s | 20 4 6 8 11 32 214 214 | 51 19 27 31 38 64 313 318 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 8 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 901.4 MB/s | 15 6 9 12 16 27 39 50 | 69 37 51 62 68 112 144 147 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 16 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 992.7 MB/s | 14 6 8 10 13 23 38 61 | 125 63 98 112 132 168 287 307 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
--- CLEANUP ------------------------------------------------------------------------------------------------------------------
$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=4 -payloads-max=15 -samples=1000 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
--- SETUP --------------------------------------------------------------------------------------------------------------------
Uploading 8 KB objects
100% |████████████████████████████████████████| [0s:0s]
[...munged...]
--- BENCHMARK ----------------------------------------------------------------------------------------------------------------
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 1.8 MB/s | 32 1 7 41 42 42 42 43 | 32 1 7 41 42 42 42 43 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 16 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 3.7 MB/s | 30 1 5 41 41 42 43 45 | 30 2 5 41 41 42 43 45 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 6.3 MB/s | 34 2 41 41 41 42 43 43 | 35 2 41 41 42 42 43 43 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 156.2 MB/s | 2 1 2 2 3 4 5 5 | 3 2 2 3 3 5 5 6 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 128 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 238.8 MB/s | 2 1 2 2 3 4 5 5 | 4 3 3 4 4 5 7 7 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 256 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 346.0 MB/s | 3 1 2 2 3 5 6 7 | 5 3 4 5 6 8 10 10 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 512 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 427.8 MB/s | 4 2 2 3 4 7 9 9 | 9 4 6 9 10 13 16 16 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 1 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 631.6 MB/s | 4 2 3 4 5 8 12 14 | 12 6 9 11 14 19 26 26 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 2 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 707.3 MB/s | 7 3 5 6 7 13 19 20 | 22 8 16 20 23 34 44 51 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 4 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 519.0 MB/s | 20 4 6 8 11 32 214 214 | 51 19 27 31 38 64 313 318 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 8 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 901.4 MB/s | 15 6 9 12 16 27 39 50 | 69 37 51 62 68 112 144 147 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 16 MB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 992.7 MB/s | 14 6 8 10 13 23 38 61 | 125 63 98 112 132 168 287 307 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
--- CLEANUP ------------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
