As I tersely noted on [1], we have a bit of issue with regards to tabled
endpoint high availability, even with BDB replication+failover.
For all HTTP methods that update the database, we return a
RedirectClient error with HTTP Location header. Even ignoring the FIXME
in cli_err() [tabled/server/server.c], and even though this behavior is
within the S3 API, it presents problems as we have currently implemented
things.
If a site implements a single endpoint "tabled.example.com" that returns
multiple A/AAAA records in DNS, then the client could potentially cycle
through a large number of redirects from tabled slaves, before finally
hitting the tabled master... with no guarantee of -ever- finding the
master. And the larger your tabled cell, the more redirects each client
must suffer before finding a master.
If a site implements distinct endpoints for each tabled node
("t1.example.com", "t2.example.com", etc.) then redirects should result
in directing clients to the current master, assuming that slaves have a
deterministic manner of discovering the current master.
Such a setup also makes use of IP Virtual Server impossible.
But that brings us to our second problem, a common problem in computer
science: the thundering herd.
When a tabled endpoint crashes or loses its master status, clients must
move en masse to the new master. As client counts increase, this
becomes a "thundering herd" DDoS'ing the new target machine.
Third, our current setup that concentrates writes on the master really
limits parallelism. It is the _BDB database_ that must only write on
the master, but due to our design, this also limits
client->tabled->chunkd writes to the master.
Ideally, we want to enable writing on every tabled node in a cell.
Given that the metadata is the only bit that _must_ be performed on the
master, it seems like the least-effort, least-cost solution for us is
for slaves to send a "write metadata" message to the master, and then
perform the data write itself.
BDB documentation[2] hints that the database replication infrastructure
may be used to send application-specific messages between BDB slaves and
the BDB master. That sounds worth investigating.
Jeff
[1] http://hail.wiki.kernel.org/index.php/Extended_status
[2] file:///usr/share/doc/db4-devel-4.7.25/api_c/rep_transport.html
--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
