rgw/multisite teuthology suite

[Casey Bodley <cbodley@xxxxxxxxxx>]

Hi Yuval,

You can find the entrypoint for multisite tests in teuthology at 
qa/suites/rgw/multisite/tasks/test_multi.yaml. This runs the 'ceph' task 
twice to install two clusters, the 'rgw' task four times to put two 
gateways on each cluster, the 'rgw-multisite' task for multisite 
configuration, then 'rgw-multisite-tests' to run a version of the 
multisite test suite.

The python code for each of these tasks is under qa/tasks/, with a 
rgw_multisite_tests.py task that performs the same role as 
src/test/rgw/test_multi.py (with a qa/tasks/rgw_multi symlink to the 
same test cases under src/test/rgw).

The test suite currently runs under two different configurations - 
qa/suites/rgw/multisite/realms/three-zone.yaml provides the 
'rgw-multisite' configuration for three zones in a single zonegroup, and 
two-zonegroup.yaml creates two zonegroups with two zones each. Each of 
these realm configurations gets tested against both the civetweb and 
beast frontend, for a total of 4 jobs in the suite.

It looks like the rgw-multisite task only knows how to create 
RadosZones, and doesn't support the num_ps_zones option for PSZones from 
test_multi.py. If you can add that, then I think we can just add a 
pubsub zone to the existing 'three-zone.yaml' job since that only uses 3 
of the 4 gateways.