Troubleshooting

The most common problems that occur in the context of the Couchbase-Elasticsearch data-replication system are data-transfer failure and data-indexing failure. This section describes procedures whereby you can monitor system-health, determine causes of failure, and fix problems.

Monitoring

You should monitor all of the following:

Outbound XDCR operations. In Couchbase Web Console, left-click on the Data Buckets tab; then, left-click on the bucket-name; and finally, open the Outbound XDCR Operations tab, which is located towards the bottom of the UI. This provides statistics related to data replication to Elasticsearch, including the rate of transfer:
Log files. The Elasticsearch Plug-in uses the same directory as file as Elasticsearch itself; which is located under the Elasticsearch install directory, and named logs.
Document-counts. The number of documents in Couchbase can be compared with the number in Elasticsearch. This assumes that Couchbase has a static number of items. To get the number for Couchbase, in the Couchbase Web Console, left-click on the Data Buckets tab; then, examine the figure for each bucket in the Item Count column. For example:

To get the number from Elasticsearch, send an HTTP request, formed as follows:
```
curl http://[elasticsearch_host]:9200/[index_name]/couchbaseDocument/_count
```
A successful response might appear as follows:
```
{
    "count":7303,"_shards":
        {
            "Total":5,"successful":5,"failed":0
    }
}
```
In this response, the count has the same value as the number of items shown in the Couchbase Web Console, indicating that all items have been successfully transferred and indexed.

Troubleshooting Data-Transfer Issues

If, by means of the monitoring procedures described above, you find evidence of data-transfer failure, proceed as follows:

Check your version of Elasticsearch. Note that a plug-in version-incompatibility may produce an error message such as the following:
```
Attention - Failed to grab remote bucket info from any of known nodes
```
Additionally, the Elasticsearch head console may display an error message such as the following:
```
[2012-12-19 05:50:41758][WARN ][org.eclipse.jetty.servlet.ServletHandler]
Error for /pools/default/buckets java.lang.NoSuchMethodError:
```
Check destination-cluster references. If such references were created in Couchbase Web Console, and the replication itself was created and started subsequently, IP addresses potentially underwent change in the interim: this occurrence is especially likely if a laptop is used as the administration-platform.

In such a case, the following error appears in the Status area of the Ongoing Replications tab, on the XDCR view provided by Couchbase Web Console:
```
2013-01-28 15:07:10 - Error replicating vbucket 56: {badmatch, {error,all_nodes_failed,
<<"Failed to grab remote bucket info from Any of known nodes">>}}
```
To resolve the problem, check remote Elasticsearch references in XDCR, and correct IP addresses where appropriate.

Troubleshooting Indexing Issues

If you find that Couchbase items do not get indexed, or that unexpected items appear in Elasticsearch query-responses, proceed as follows:

Change the refresh_interval-setting to a reduced number of seconds. See Update Indices Settings in the Elasticsearch online documentation.
Check Elasticsearch mappings. Such mappings are automatically generated by default; and this involves assumptions about data types and structures within the imported documents. In consequence, objects may be omitted: for example, objects within an array. See Mapping, in the Elasticsearch online documentation.
Check documents, to ensure that all are well-formed JSON. Note that whereas, when JSON is not well-formed, the plug-in both accepts binary data and logs error messages, Elasticsearch itself cannot index any such data.

Enabling Logging

You can configure the Couchbase Transport logging facility by editing the file ./config/logging.yml, and adding the following lines to the logger section:

Transport.couchbase: TRACE com.couchbase: TRACE

For information, see Logging, in the Elasticsearch online documentation.