On Thu, 14 Dec 2006, Tom Lane wrote:
The pgbench app itself becomes the bottleneck at high transaction rates. Awhile back I rewrote it to improve its ability to issue commands concurrently, but then desisted from submitting the changes --- if we change the app like that, future numbers would be incomparable to past ones, which sort of defeats the purpose of a benchmark no?
Not at all. Here's an example from the PC hardware benchmarking landscape. Futuremark Corporation has a very popular benchmark for 3D hardware called 3DMark. Every year, they release a new version, and numbers from it are completely different from those produced by the previous year's version. That lets them rev the entire approach taken by the benchmark to reflect current practice. So when the standard for high-end hardware includes, say, acceleration of lighting effects, the new version will include a lighting test. In order to break 1000 points (say) on that test, you absolutely have to have lighting acceleration, even though on the previous year's test you could score that high without it.
That is not an isolated example; every useful PC benchmark gets updated regularly, completely breaking backward compatibility, to reflect the capabilities of current hardware and software. Otherwise we'd still be testing how well DOS runs on new processors. Right now everyone is (or has already) upgraded their PC benchmarking code such that you need a dual-core processor to do well on some of the tests.
If you have a pgbench version with better concurrency features, I for one would love to see it. I'm in the middle of patching that thing up right now anyway but hadn't gotten that far (yet--I just spent some of yesterday staring at how it submits into libpq trying to figure out how to improve that). I would be happy to take your changes, my changes, changes to the base code since you forked it, and reconcile everything together into a pgbench2007--whose results can't be directly compared to the earlier version, but are more useful on current gen multi-processor/core systems.
-- * Greg Smith gsmith@xxxxxxxxxxxxx http://www.gregsmith.com Baltimore, MD
