tabled vs. BDB high availability

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




As I tersely noted on [1], we have a bit of issue with regards to tabled endpoint high availability, even with BDB replication+failover.

For all HTTP methods that update the database, we return a RedirectClient error with HTTP Location header. Even ignoring the FIXME in cli_err() [tabled/server/server.c], and even though this behavior is within the S3 API, it presents problems as we have currently implemented things.

If a site implements a single endpoint "tabled.example.com" that returns multiple A/AAAA records in DNS, then the client could potentially cycle through a large number of redirects from tabled slaves, before finally hitting the tabled master... with no guarantee of -ever- finding the master. And the larger your tabled cell, the more redirects each client must suffer before finding a master.

If a site implements distinct endpoints for each tabled node ("t1.example.com", "t2.example.com", etc.) then redirects should result in directing clients to the current master, assuming that slaves have a deterministic manner of discovering the current master.

Such a setup also makes use of IP Virtual Server impossible.

But that brings us to our second problem, a common problem in computer science: the thundering herd.

When a tabled endpoint crashes or loses its master status, clients must move en masse to the new master. As client counts increase, this becomes a "thundering herd" DDoS'ing the new target machine.

Third, our current setup that concentrates writes on the master really limits parallelism. It is the _BDB database_ that must only write on the master, but due to our design, this also limits client->tabled->chunkd writes to the master.

Ideally, we want to enable writing on every tabled node in a cell. Given that the metadata is the only bit that _must_ be performed on the master, it seems like the least-effort, least-cost solution for us is for slaves to send a "write metadata" message to the master, and then perform the data write itself.

BDB documentation[2] hints that the database replication infrastructure may be used to send application-specific messages between BDB slaves and the BDB master. That sounds worth investigating.

	Jeff


[1] http://hail.wiki.kernel.org/index.php/Extended_status
[2] file:///usr/share/doc/db4-devel-4.7.25/api_c/rep_transport.html



--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Fedora Clound]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux