Announcing the public availability of the TPCx-V prototype

[Reza Taheri <rtaheri@xxxxxxxxxx>]

Hello PGSQL performance community,

You might remember me from these past postings:

·         In 2012, we announced that the TPC was using PostgreSQL in a benchmark for virtualized databases. Unlike the traditional TPC Enterprise benchmarks, which are defined in paper specifications, this Express benchmark comes with a complete end-to-end benchmarking kit, which at this point in time, runs on Linux VMs (we have tested it with PGSQL 9.3 on RHEL 7.1) on the X86 architecture.
On that occasion, I was looking for a PGSQL feature similar to MS SQL Server’s “clustered indexes” or Oracle’s “index-only-tables”. (I was told about “index-only scans” which helped but not as much as clustered indexes or IOT)

·         In 2013, I asked about a performance hit during checkpoints, which was quickly diagnosed as the well-known dirty_background_bytes problem

·         Last year, I asked about the high rate of transaction aborts due to serialization failures. It turned out that even dbt5, which implements TPC-E, was running into the same issue, and simply resubmitting the transaction was an acceptable solution

 

We are done with the benchmark development, and it has entered TPC’s “Public Review” phase. In parallel, the TPC council has approved the public release of a prototype of the benchmark kit to gather more experimental data, and to speed up the approval of the benchmark.

 

The Benchmark Specification, the Benchmark Kit, and the User’s Guide are available from tpc.org/tpcx-v. In addition to these components of the benchmark standard, the subcommittee members have developed two other tools to help benchmarkers get a fast start:  a downloadable VM in the form of an ovf-format VM template that contains a complete benchmarking environment with all the software already installed and pre-configured; and a PowerCLI script that, in the VMware vSphere environment, allows you to quickly clone a VM template into a large number of benchmark VMs.

 

The review period is open until Thursday, October 15th. The subcommittee intends to resolve the comments from the Formal Review by November 12th, and bring forward a motion to the Council to approve the Benchmark Standard.

 

We would love to get feedback from the PGSQL community. The subcommittee is taking feedback via FogBugz at www.tpc.org/bugdb under the project “TPC-Virt”.

 

Anticipating some of the questions/comments, here are some thoughts:

-          We wish we had a more representative workload; with multiple kits for multiple hardware architectures; etc. Those are questions for another day. Having an end-to-end database benchmarking kit for X86 virtualization with a workload derived from TPC-E is a pretty good first step.

-          You don’t have to run the full multi-VM configuration of TPCx-V if you are just playing with the kit and not intending to have an audited result. You can run the benchmark on a single database on a single VM (or even a native, un-virtualized server!) This would be very similar to a simple TPC-E config. If we can get feedback based on a single-VM config, we will still be grateful

-          Having said that, there are enough subtle differences between the TPC-E schema and the TPCx-V schema that running TPCx-V on a single VM doesn’t exactly take you to TPC-E.

 

Thanks,

Reza Taheri for the TPCx-V subcommittee