Rebalancing Nodes
Nodes are rebalanced with the POST /controller/rebalance HTTP method and URI.
Description
To start a rebalance process, provide the ejectedNodes and knownNodes parameters.
These parameters contain the list of nodes that have been marked to be ejected and the list of nodes that are known within the cluster.
Retrieve information about ejected and known nodes by getting the current node configuration. This ensures that the client making the REST API request is aware of the current cluster configuration. Nodes should have been previously added or marked for removal as appropriate.
HTTP method and URI
The information must be supplied via the ejectedNodes and knownNodes parameters as a POST operation to the /controller/rebalance endpoint.
POST /controller/rebalance
Syntax
curl -v -X POST -u [admin]:[password] http://[localhost]:8091/controller/rebalance [-d ejectedNodes | -d knownNodes]
Examples
Curl request example for rebalancing new nodes in to a cluster:
curl -v -X POST -u Administrator:password \ 'http://192.168.0.77:8091/controller/rebalance'\ -d 'knownNodes=ns_1@192.168.0.77,ns_1@192.168.0.56'
Curl request example for rebalancing nodes out of a cluster:
curl -v -X POST -u Administrator:password \ 'http://192.168.0.77:8091/controller/rebalance'\ -d 'ejectedNodes=ns_1@192.168.0.56'
Raw HTTP request example:
POST /controller/rebalance HTTP/1.1 Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4= User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5 Host: 192.168.0.77:8091 Accept: */* Content-Length: 63 Content-Type: application/x-www-form-urlencoded
Getting rebalance progress
HTTP method and URI
There are two endpoints for rebalance progress.
One is a general request which outputs high-level percentage completion at /pools/default/rebalanceProgress.
The second possible endpoint is one corresponds to the detailed rebalance report available in the web console.
GET /pools/default/rebalanceProgress
Example
Curl request example that returns a JSON structure containing the current progress information:
curl -u Administrator:password \ '192.168.0.77:8091/pools/default/rebalanceProgress'
Raw HTTP request example:
GET /pools/default/rebalanceProgress HTTP/1.1 Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4= User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5 Host: 192.168.0.77:8091 Accept: */*
Response
The response data packet contains a JSON structure showing the rebalance progress for each node. The progress figure is provided as a percentage (shown as a floating point value between 0 and 1).
{
"status":"running",
"ns_1@192.168.0.56":{"progress":0.2734375},
"ns_1@192.168.0.77":{"progress":0.09114583333333337}
}
Example: detailed
Curl request example with more details about the rebalance:
curl -u Administrator:password \ 'http://192.168.0.77:8091/pools/default/tasks'
Raw HTTP request example:
GET /pools/default/rebalanceProgress HTTP/1.1 Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4= User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5 Host: 192.168.0.77:8091 Accept: */*
Response: detailed
The response data packet contains a JSON structure showing detailed progress:
{
type: "rebalance",
recommendedRefreshPeriod: 0.25,
status: "running",
progress: 9.049479166666668,
perNode: {
ns_1@10.3.3.61: {
progress: 13.4765625
},
ns_1@10.3.2.55: {
progress: 4.6223958333333375
}
},
detailedProgress: {
bucket: "default",
bucketNumber: 1,
bucketsCount: 1,
perNode: {
ns_1@10.3.3.61: {
ingoing: {
docsTotal: 0,
docsTransferred: 0,
activeVBucketsLeft: 0,
replicaVBucketsLeft: 0
},
outgoing: {
docsTotal: 512,
docsTransferred: 69,
activeVBucketsLeft: 443,
replicaVBucketsLeft: 511
}
},
ns_1@10.3.2.55: {
ingoing: {
docsTotal: 512,
docsTransferred: 69,
activeVBucketsLeft: 443,
replicaVBucketsLeft: 0
},
outgoing: {
docsTotal: 0,
docsTransferred: 0,
activeVBucketsLeft: 0,
replicaVBucketsLeft: 443
}
}
}
}
}
This response shows percentage complete for each individual node undergoing rebalance. For each specific node, it provides the current number of docs transferred and other items. For details and definitions of these items.
If rebalance fails, the following response error occurs:
[
{
"type": "rebalance",
"status": "notRunning",
"errorMessage": "Rebalance failed. See logs for detailed reason. You can try rebalance again."
}
]
Adjusting rebalance during compaction
Description
If a rebalance is performed while a node is undergoing index compaction, rebalance delays may be experienced.
The parameter, rebalanceMovesBeforeCompaction, is used to improve rebalance performance.
If this selection is made, index compaction performance is reduced which can result in larger index file size.
This needs to be made as POST request to the /internalSettings endpoint.
By default, this setting is 64 which specifies the number of vBuckets which are moved per node until all vBucket movements pauses.
After this pause, the system triggers index compaction.
Index compaction is not performed while vBuckets are being moved, so if a larger value is specified, it means that the server spends less time compacting the index, which results in larger index files that take up more disk space.