Hard Failover
Graceful failover can be used to remove a node from a cluster when the node is unavailable or unstable.
Understanding Hard Failover
Hard failover is the process of removing a cluster-node that has become unavailable or unstable. The exact sequence of actions taken by the Cluster Manager depends on which services are running on the node to be removed.
Hard failover can be performed on any node in the cluster. However, if all cluster-nodes are available, either graceful failover or node removal should be used instead.
Hard failover can be initiated manually by means of Couchbase Web Console, the CLI, or the REST API. It can also be initiated by means of automatic failover.
Hard Failover on Data Nodes
If the node to be failed over has been running the Data Service, and active vBuckets have therefore become unavailable, the Cluster Manager promotes replica vBuckets to active status on the remaining cluster-nodes. A revised cluster-map is then communicated to all clients.
When the process is complete, no vBuckets remain on the failed-over node, but the node has not yet been fully removed from the cluster. The cluster is at this point considered to be in a degraded state, in terms of availability; as is now contains a reduced number of replica vBuckets, and so lacks resources to handle a further node-outage.
Hard Failover on Index Nodes
When Global Secondary Indexes (GSI) are defined, each is by default created on only one node, which is running the Index Service. If that node fails, the indexes therefore become unavailable. If the node is subsequently repaired and added back into the cluster by means of Delta node recovery, the indexes are updated, and become available again. If the node is replaced, the indexes need to be recreated: in this case, before rebalance of the new node into the cluster, the failed node must be removed by means of hard failover.
Hard Failover on Query Nodes
Query Service nodes are stateless, and can be added and removed from the cluster with no consequence to data. However, ongoing queries on those nodes are interrupted by hard failover, and therefore produce errors.
As long as one Query Service node continues to be available, the cluster continues to support querying, although potentially with reduced performance.
Hard Failover on Search Nodes
If multiple nodes run the Search Service, Full Text Indexes are partitioned, and are automatically distributed across those nodes. If a Search Service node is failed over, it stops taking traffic. If no other nodes run the Search Service, all building of Full Text Indexes stops, and searches fail. If at least one other node is running the Search Service, this continues to handle queries and return partial results.
When rebalance occurs following failover:
-
If replica Full Text Indexes have been configured, the Search Service promotes the replicas to active on the remaining nodes of the cluster.
-
If replica Full Text Indexes have not been configured, the Search Service rebuilds the indexes on the remaining nodes of the cluster, using stored index definitions.
Note that the Search Service is not supported by Delta Node Recovery.
Returning the Cluster to a Stable State
If or when the failed node is repaired and ready, it can be added back to the cluster via Delta or Full recovery. Alternatively, an entirely new node can be added instead.
-
If Delta Node Recovery can be performed, the Cluster Manager recognizes the node as a previous member of the cluster. If the administrator is using the Couchbase Web Console, they are prompted to perform the Delta Node Recovery. In CLI is being used, the operation is either performed successfully; or fails, and inform the administrator that Full recovery is required.
When a node is added back to the cluster using Delta recovery, the replica vBuckets on the failed-over node are considered to be trusted but behind on data. The Cluster Manager therefore resynchronizes the vBuckets, so that their data becomes current. When this operation is complete, vBuckets are promoted to active status as appropriate, and the cluster map is updated.
-
If the node is added back using Full recovery, the node is treated as an entirely new node: it is reloaded with data, and requires rebalance.
-
If the node cannot be added back, a new node can be added, and the cluster rebalanced.
Prior to rebalance, a cluster should always be restored to an appropriate size and topology. Note that a rebalance performed prior to re-adding a failed over node prevents Delta recovery.
Hard Failover Example
Given:
-
A cluster containing four nodes, each of which runs the Data Service
-
A single replica configured per bucket, such that 256 active and 256 replica vBuckets therefore reside on each node
-
Node 4 of the cluster, on which vBucket #762 resides, offline and apparently unrecoverable
The following occur:
-
Clients attempting reads and writes on node 4 receive errors or timeouts.
-
Hard failover is initiated, either manually or automatically, to remove node 4.
-
The Cluster Manager promotes the replica vBucket 762 to active status, on node 2. The cluster now has no replica for vBucket 762.
-
The Cluster Cap is updated, so that clients' subsequent reads and writes will go to the correct location for vBucket 762, now node #2.
-
The same process is repeated for the remaining 255 vBuckets. It is then repeated for the remaining 255 vBuckets of the bucket, one bucket at a time.
-
Hard Failover and Multiple Nodes
Unless Server Group Awareness is in operation, multiple nodes should be note failed over simultaneously unless enough replica vBuckets exist on the remaining nodes to support required promotions to active status, and when the number of remaining nodes is otherwise sufficient in terms of resources to allow continued cluster-operation. If two nodes are to be failed over, two replicas per bucket are required, to prevent data-loss.