Re: RGW performance with low object sizes

[Christian <syphdias+ceph@xxxxxxxxx>]

Hi,

I used https://github.com/dvassallo/s3-benchmark to measure some performance values for the rgws and got some unexpected results.

Everything above 64K has excellent performance but below it drops down to a fraction of the speed and responsiveness resulting in even 256K objects being faster than anything below 64K.

Does anyone observe similar effects while running this benchmark? Is it the benchmarks fault or are there some options to tweak performance for low object sizes?

I have an update on the performance  issues with reading rgw objects smaller than 64K. If I reduce the sample size to a lower number (number of my threads in this case) I get good values for the first objects.

In this example for 32K objects: The more samples I use the worse my average and my percentiles get. I interpret this as: The first 32K objects are read fast but the consecutive reads are getting slower the more objects are read.

Here are 3 examples with 8 threads, 32K and 64K with 8, 12, 16 and 128 samples to demonstrate how the performance degrades with more samples.

Notice how the performance is deminished drastically for 32K objects but not as badly for 63K objects.

Is there a config option that can be tuned to get better performance for 32K objects? Like a cache or buffer setting I didn't find yet...

Can anyone confirm the same issue?

$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=8 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       6.1 MB/s |    8     2     2     3     4     4     4    41 |    8     2     2     3     4     4     4    41 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 64 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |     122.5 MB/s |    3     2     3     3     3     4     4     4 |    3     3     3     3     3     4     4     4 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]

$./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=12 -endpoint https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       7.8 MB/s |   20     3     3     4    41    44    44    44 |   20     3     3     4    41    44    44    44 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 64 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |     106.3 MB/s |    4     2     3     4     5     5     5     5 |    4     2     3     4     5     5     5     5 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]

$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=16 -endpoint http://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false
[...section SETUP was deleted...]
Download performance with 32 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       6.0 MB/s |   25     2     3    41    42    42    42    42 |   25     2     3    41    42    42    42    42 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 64 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |      88.4 MB/s |    4     1     2     3     5     6     9    10 |    4     1     2     4     5     6     9    10 |
+---------+----------------+------------------------------------------------+------------------------------------------------+
[...section CLEANUP was deleted...]

$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8 -payloads-min=6 -payloads-max=7 -samples=128 -endpoint http://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false

[...section SETUP was deleted...]
Download performance with 32 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |       7.4 MB/s |   32     1    40    41    41    41    42    42 |   32     1    40    41    41    42    42    42 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

Download performance with 64 KB objects
                           +-------------------------------------------------------------------------------------------------+
                           |            Time to First Byte (ms)             |            Time to Last Byte (ms)              |
+---------+----------------+------------------------------------------------+------------------------------------------------+
| Threads |     Throughput |  avg   min   p25   p50   p75   p90   p99   max |  avg   min   p25   p50   p75   p90   p99   max |
+---------+----------------+------------------------------------------------+------------------------------------------------+
|       8 |     196.3 MB/s |    2     1     2     2     3     3     5     5 |    2     1     2     2     3     3     5     5 |
+---------+----------------+------------------------------------------------+------------------------------------------------+

[...section CLEANUP was deleted...]

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com