On 4/21/2015 9:21 AM, Holger.Friedrich-Fa-Trivadis@xxxxxxxxx wrote:
Hello list, Exactly what constitutes „reproducible“ values from pgbench? I keep getting a range between 340 tps and 440 tps or something like that using the same command line on the same machine. Is that reproducible enough? The docs state that one should verify that the numbers are reproducible, so I repeat any test run ten times before believing the results. I’ve tried increasing the test duration (-T) from one minute to five minutes, then turning off autovacuum (in postgresql.conf) as recommended by the docs, but the range of results is not getting any narrower. So what does “reproducible” mean as applied to pgbench? Obviously I could be doing something wrong, such as missing some vital configuration option… Thanks in advance for any insights. Cheers, Holger Friedrich
I think its common to get different timings. I think its ok because things are changing (files, caches, indexes, etc).
If you run three to five short runs, they should all be withing the same range (say 340 to 440). If you are planning hardware, you might take the worst case and purchase based on that. If you are planning schedules you might use the average case. If you are bragging on the newsgroups use the best case :-).
If you want more realistic then keep vacuum enabled and run for 24 hours. In the real world, you are going to vacuum, so benchmark it too.
If you are playing with postgres.conf settings, then three runs of a few minutes each will give you an average, and you can compare different settings based on that.
As Qingqing said, a read-only test should be more stable, because you are comparing apples to apples. A read-write test is changing under the hood so expect some differences.
Also, if your test data is small, or large, you are benchmarking different things. (lock speed, cpu speed, disk io, etc)
pgbench is good for a first test, but its going to act different than your real world work load.
-Andy -- Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
