Synchronization

Bedrock’s primary feature is its ability to seamlessly synchronize data between multiple nodes, manage distributed commits across those nodes, and automatically failover (and recover) in response to failure. This enables for very reliable configurations across multiple datacenters (or availability zones). A general description of how this works follows:

Any number of nodes starts up, each of which has a unique priority (except for “permafollowers”, which have 0 priority and are invisible from a quorum perspective, but provide additional read capacity).
All nodes attempt to connect to all other nodes.
During this process all nodes SYNCHRONIZE from their peers, which means they broadcast “My most recent transaction has commitCount X, and the hash of every transaction up to that point is Y”. Anybody who has newer data will respond with the missing transactions, which are all committed in the same order on every node via a private blockchain.
Any two nodes that disagree on what the hash of a given transaction should be will immediately disconnect from each other. This means that any node that has “forked” away from the cluster will be excluded from participation.
The Paxos distributed consensus algorithm is used to identify which of the connected nodes has the highest priority. If enough of the configured nodes are online and agree, the highest node will stand up as LEADER. All other nodes begin FOLLOWING to that leader. (On the other hand, if too few of the configured nodes are able to connect so as to achieve quorum, then nobody will stand up, thereby avoiding the “split brain” problem.)
Once a node begins LEADING or FOLLOWING, it opens up its external port to begin accepting traffic from clients (typically webservers). Clients are typically configured to connect to the “nearest” node from a latency perspective, but all nodes appear equally capable from the outside – the client has no awareness of who is or isn’t the leader.
Each node processes read requests from its local database. By default it will respond based on the latest data. However, the client can optionally provide a commitCount, which if larger than the current commit count of that node’s database, will cause the node to hold off on responding until the database has been synchronized up to that point. In this way, clients can avoid inconsistency by querying two different nodes with different states (though in practice, clients should attempt to query the same node repeatedly to avoid any unnecessary delay). All of this is provided “out of the box” by Bedrock’s PHP client library.
Write commands are escalated to the leader, which commits them to its own database and then streams them to the followers. The leader does not wait for followers to acknowledge a transaction before committing it, which is what allows write throughput to exceed 1/median(rtt) transactions per second.
The cost of not waiting is that the leader can “race ahead” of the cluster: if it crashes with commits that haven’t reached anyone else, those commits are lost. A cluster is therefore only as durable as the leader’s ability to hand its commits off before it dies. There is no per-transaction way to ask for a stronger guarantee.
After a write transaction is processed, the response is returned to the node that escalated it, and then back to the client.
If the leader dies before an escalated command has been processed, the follower will re-escalate the command to the new leader once elected. Furthermore, followers will continue accepting commands during the period of leader failover, thereby ensuring that the client sees no “downtime” and merely a short delay (typically imperceptible).
When the leader returns to operation, the leader will synchronize any transactions it missed while down, and then stand back up and take over control from the interim leader seamlessly.