Re: Multimaster

konstantin knizhnik <k.knizhnik@xxxxxxxxxxxxxx> · Thu, 14 Apr 2016 12:14:40 +0300

On Apr 14, 2016, at 8:41 AM, Craig Ringer wrote:

On 1 April 2016 at 19:50, Konstantin Knizhnik <k.knizhnik@xxxxxxxxxxxxxx> wrote:

Right now the main problem is parallel apply: we need to apply
    changes concurrently to avoid unintended dependencies causing
    deadlocks and provide reasonable performance.

How do you intend to approach that?

Actually we already have working implementation of multimaster...
There is a pool of pglogical executors. pglogical_receiver just reads transaction body from connection and append it to ready-for-execution queue.
Any vacant worker form this pool can dequeue this work and proceed it.
To provide correct  replication session context, I have to setup replication session for each transaction. It looks dummy but fortunately has no effect on performance.

It is certainly possible with this approach that order of applying transactions can be not the same at different nodes.
But it is not a problem if we have DTM. The only exception is recovery of multimaster node. In this case we have to apply transaction exactly in the same order as them were applied at the original node performing recovery. It is done by applying changes in recovery mode by pglogical_receiver itself.

You should be able to do parallel apply between nodes trivially, i.e. node A applies changes in parallel from nodes B C and D. 

Doing parallel apply of multiple changes from node A to node B is much harder. I wrote about parallel logical apply separately so I won't repeat it here; search the archives for the notes if interested.

We also need 2PC support but this code was sent to you by Stas, so I
    hope that sometime it will be included in PostgreSQL core and
    pglogical plugin.

I never got a response to my suggestion that testing of upstream DDL is needed for that. I want to see more on how you plan to handle DDL on the upstream side that changes the table structure and acquires strong locks. Especially when it's combined with row changes in the same prepared xacts. 

We are now replicating DDL in the way similar with one used in BDR: DDL statements are inserted in special table and are replayed at destination node as part of transaction.
We have also alternative implementation done by Artur Zakirov <a.zakirov@xxxxxxxxxxxxxx> 
which is using custom WAL records: https://gitlab.postgrespro.ru/pgpro-dev/postgrespro/tree/logical_deparse
Patch for custom WAL records was committed in 9.6, so we are going to switch to this approach.

Right now we are trying to run all Postgres regression tests for multimaster. Currently about 50 tests are failed. There are a lot of different issues: with temporary tables, GUC variables, ...
Some we have fixed, investigation of others is in progress...

I'd really like separate patches for the core changes and the pglogical support for them, too.

There are also some minor technical issues which lead us to making
    few changes in pglogical code but we tried to do our best to keep
    original versions unchanged, so we can switch to public version in
    future.

Details?

Now is exactly the time to address those points.

Well, recently I have made attempt to merge our code with the latest version of pglogical plugin (because our original implementation of multimaster was based on the code partly taken fro BDR) but finally have to postpone most of changes. My primary intention was to support metadata caching. But presence of multiple apply workers make it not possible to implement it in the same way as it is done node in pglogical plugin. 

Also now pglogical plugin contains a lot of code which performs mapping between source and target database schemas. So it it is assumed that them may be different.
But it is not true in case of multimaster and I do not want to pay extra cost for the functionality we do not need.

Frankly speaking I do not wan to spent much time on integration of multimaster with pglogical plugin right now,  because  both of them are actively developed now. We should definitely do it, but  when we have stable version, so there is no need to repeat this work once and once again.

We can try to prepare our "wish list" for pglogical plugin.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services