On Tue, Nov 19, 2019 at 9:34 AM Christian <syphdias+ceph@xxxxxxxxx> wrote:
Hi,I used https://github.com/dvassallo/s3-benchmark to measure some performance values for the rgws and got some unexpected results.Everything above 64K has excellent performance but below it drops down to a fraction of the speed and responsiveness resulting in even 256K objects being faster than anything below 64K.Does anyone observe similar effects while running this benchmark? Is it the benchmarks fault or are there some options to tweak performance for low object sizes?I have an update on the performance issues with reading rgw objects smaller than 64K. If I reduce the sample size to a lower number (number of my threads in this case) I get good values for the first objects.In this example for 32K objects: The more samples I use the worse my average and my percentiles get. I interpret this as: The first 32K objects are read fast but the consecutive reads are getting slower the more objects are read.
Here are 3 examples with 8 threads, 32K and 64K with 8, 12, 16 and 128 samples to demonstrate how the performance degrades with more samples.Notice how the performance is deminished drastically for 32K objects but not as badly for 63K objects.Is there a config option that can be tuned to get better performance for 32K objects? Like a cache or buffer setting I didn't find yet...Can anyone confirm the same issue?$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=8 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 6.1 MB/s | 8 2 2 3 4 4 4 41 | 8 2 2 3 4 4 4 41 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 122.5 MB/s | 3 2 3 3 3 4 4 4 | 3 3 3 3 3 4 4 4 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]
$./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=12 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 7.8 MB/s | 20 3 3 4 41 44 44 44 | 20 3 3 4 41 44 44 44 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 106.3 MB/s | 4 2 3 4 5 5 5 5 | 4 2 3 4 5 5 5 5 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]
$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=16 -endpoint http://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 6.0 MB/s | 25 2 3 41 42 42 42 42 | 25 2 3 41 42 42 42 42 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 88.4 MB/s | 4 1 2 3 5 6 9 10 | 4 1 2 4 5 6 9 10 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=128 -endpoint http://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false[...section SETUP was deleted...]
Download performance with 32 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 7.4 MB/s | 32 1 40 41 41 41 42 42 | 32 1 40 41 41 42 42 42 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
Download performance with 64 KB objects
+-------------------------------------------------------------------------------------------------+
| Time to First Byte (ms) | Time to Last Byte (ms) |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads | Throughput | avg min p25 p50 p75 p90 p99 max | avg min p25 p50 p75 p90 p99 max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| 8 | 196.3 MB/s | 2 1 2 2 3 3 5 5 | 2 1 2 2 3 3 5 5 |
+---------+----------------+------------------------------------------------+------------------------------------------------+[...section CLEANUP was deleted...]
Did you make progress on this? We have a ton of < 64K objects as well and are struggling to get good performance out of our RGW. Sometimes we have RGW instances that are just gobbling up CPU even when there are no requests to them, so it seems like things are getting hung up somewhere. There is nothing in the logs and I haven't had time to do more troubleshooting.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
