Manage Logging

The Logging facility allows a record to be maintained of important events that occur on Couchbase Server.

Logging Overview

The Couchbase Logging facility records important events, and saves the details to log files, on disk. Additionally, subsets of information are provided on the Logs screen, in Couchbase Web Console. This may appear as follows:

loggingScreenBasic

By default, on Linux systems, log files are saved to /opt/couchbase/var/lib/couchbase/logs; on MacOS, to /Users/username/Library/Application Support/Couchbase/var/lib/couchbase/logs; and on Windows, to C:\Program Files\Couchbase\Server\var\lib\couchbase\logs.

Collecting Information

On each node within a Couchbase Server-cluster, logging is performed continuously. A subset of the results can be reviewed in the Couchbase Web Console Logs screen; while all details are saved to the logs directory, as described above. (Note that the logs directory may include audit.log. This is a special log file, used to manage cluster-security, and is handled separately from the other log files. The information provided throughout the remainder of this page — on collecting, uploading, redacting, and more — does not apply to audit.log. For information on audit.log, see Auditing.)

Additionally, explicit logging can be performed by the user. This allows comprehensive and fully updated information to be generated as required. The output includes everything currently on disk, together with additional data that is gathered in real time. Explicit logging can either be performed for all nodes in the cluster, or for one or more individual nodes. The results are saved as zip files: each zip file contains the log-data generated for an individual node.

Explicit logging can be performed by means of the Couchbase CLI utility cbcollect_info. The documentation for this utility, provided here, includes a complete list of the log files that can be created, and a description of the contents of each.

Explicit logging can also be performed by means of Couchbase Web Console: on the Logs page, left-click on the Collect Information tab, located near the top.

collectInfo

This brings up the Collect Information screen:

collectInformationScreen

This allows logs and diagnostic information to be collected either from all or from selected nodes within the cluster. It also allows a Log Redaction Level to be specified (this is described in Applying Redaction, below). The Specify custom temp directory checkbox can be checked to specify the absolute pathname of a directory into which data is temporarily saved, during the collection process. The Specify custom destination directory can be checked to specify the absolute pathname of a directory into which the competed zip files are saved.

The Upload to Couchbase checkbox is described in Uploading Log Files, below.

To start the collection-process, left-click on the Start Collection button. A notification is displayed, indicating that the collection-process is running. When the process has completed, the following information is displayed:

collectInformationComplete

As this indicates, a set of log files has been created for each node in the cluster. Each file is saved as a zip file in the stated temporary location.

Uploading Log Files

Log files can be uploaded to Couchbase, for inspection by Couchbase Support.

For information on performing upload at the command-prompt, see cbcollect_info. To upload by means of Couchbase Web Console, before starting the collection-process, check the Upload to Couchbase checkbox:

uploadToCouchbaseCheckbox

The display changes to the following:

uploadToCouchbaseDialogBasic

The dialog now features an Upload to Host field, which contains the server-location to which the customer-data is uploaded. Fields are also provided for Customer Name (required) and Ticket Number (optional). The Upload Proxy field optionally takes the hostname of a remote system, which contains the directory specified by the pathname.

Left-click on the Start Collection button. When collection is complete, a notification provides the URL of the uploaded zip file.

Understanding Redaction

Optionally, log files can be redacted. This means that user-data, considered to be private, is removed. Such data includes:

  • Key/value pairs in JSON documents

  • Usernames

  • Query-fields that reference key/value pairs and/or usernames

  • Names and email addresses retrieved during product registration

  • Extended attributes

This redaction of user-data is referred to as partial redaction. (Full redaction, which will be available in a forthcoming version of Couchbase Server, additionally redacts meta-data.)

In each modified log file, encrypted text (achieved with SHA1) is substituted for redacted text. For example, the following log file fragment displays private data — a Couchbase username:

0ms [I0] {2506} [INFO] (instance - L:421) Effective connection string:
couchbase://127.0.0.1?username=Administrator&console_log_level=5&;.
Bucket=default

The redacted version of the log file might appear as follows:

0ms [I0] {2506} [INFO] (instance - L:421) Effective connection string:
<UD>e07a9ca6d84189c1d91dfefacb832a6491431e95</UD>.
Bucket=<UD>e16d86f91f9fd0b110be28ad00e348664b435e9e</UD>

Note that redaction may eliminate some parameters containing non-private data, as well as all parameters containing private.

Note also that redaction of log files may have one or both of the following consequences:

  • Logged issues will be found harder to diagnose, by both the user and Couchbase Support.

  • Log-collection is significantly more time-consumptive, since redaction is performed at collection-time.

Applying Redaction

Redaction of log files saved on the cluster can be applied as required, when performing explicit logging, by means of either cbcollect_info or the Logs facility of Couchbase Web Console.

For information on performing explicit logging with redaction at the command-prompt, see cbcollect_info.

To perform explicit logging with redaction by means of Couchbase Web Console, before starting the collection-process, access the Log Redaction Level panel, on the Collect Information screen. This features two radio-buttons, labelled None and Partial Redaction. Make sure the Partial Redaction radio-button is selected. Guidance on redaction is displayed below it:

partialRedactionSelection

Left-click on the Start Collection button. A notification explains that the collection-process is now running. When the process has completed, a further notification appears, specifying the location (local or remote) of each created zip file. Note that, when redaction has been specified, two zip files are provided for each node: one file containing redacted data, the other unredacted data.

Redacting Log Files Outside the Cluster

Certain Couchbase technologies — such as cbbackupmgr, the SDK, connectors, and Mobile — create log files saved outside the Couchbase Cluster. These can be redacted by means of the command-line tool cblogredaction. Multiple log files can be specified simultaneously. Each file must be specified as plain text. Optionally, the salt to be used in encryption can be automatically generated.

For example:

$ cblogredaction /Users/username/testlog.log -g -o /Users/username -vv
2018/07/17T11:27:06 WARNING: Automatically generating salt. This will make it difficult to cross reference logs
2018/07/17T11:27:07 DEBUG: /Users/username/testlog.log - Starting redaction file size is 19034284 bytes
2018/07/17T11:27:07 DEBUG: /Users/usernae/testlog.log - Log redacted using salt: <ud>COeAtexHB69hGEf3</ud>
2018/07/17T11:27:07 INFO: /Users/username/testlog.log - Finished redacting, 50373 lines processed, 740 tags redacted, 0 lines with unmatched tags

For more information, see the corresponding man page, or run the command with the --h (help) option.

Log File Locations

Couchbase Server creates log files in the following locations.

Platform Location

Linux

/opt/couchbase/var/lib/couchbase/logs

Windows

C:\Program Files\Couchbase\Server\var\lib\couchbase\logs

Assumes default installation location

Mac OS X

/Users/couchbase/Library/Application Support/Couchbase/var/lib/couchbase/logs

Log File Listing

The following table lists the log files to be found on Couchbase Server.

File Log Contents

audit

Security audit log for administrators.

babysitter

Troubleshooting log for the babysitter process which is responsible for spawning all Couchbase Server processes and respawning them where necessary.

couchdb

Troubleshooting log for the couchdb subsystem which underlies map-reduce and spatial views

crash-log.bin

Used to pass service crash reports from the babysitter to the ns_server. For example, if the ns_server is available, any crash of the babysitter’s child is passed directly to the special crash logger service within the ns_server. If the logger service is not attached to the babysiter, then the babysitter saves that crash report to the disk and the ns_server can later obtain and log it even if the babysitter is restarted. This is not a log file in itself.

debug

Debug-level troubleshooting for the cluster management component.

error

Error-level troubleshooting log for the cluster management component.

fts

Troubleshooting logs for the full-text search service.

goxdcr

Troubleshooting log for the Cross Data Center Replication (XDCR) component used in Couchbase Server versions after 4.0.

http_access

The admin access log records server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.

http_access_internal

The admin access log records internal server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.

indexer

Troubleshooting log for the indexing and storage subsystem.

info

Info-level troubleshooting log for the cluster management component.

mapreduce_errors

JavaScript and other view-processing errors are reported in this file.

memcached

Contains information relating to the core memcached component, including DCP stream requests and slow operations.

metakv

Troubleshooting log for the metakv store, a cluster-wide metadata store.

ns_couchdb

Contains information related to starting up the CouchDB subsystem.

projector

Troubleshooting log for the projector process which is responsible for sending appropriate mutations from Data nodes to Index nodes.

reports

Contains progress and crash reports for the Erlang processes. Due to the nature of Erlang, processes crash and restart upon an error.

ssl_proxy

Troubleshooting log for the ssl proxy spawned by the cluster manager.

stats

Contains periodic statistic dumps from the cluster management component.

views

Troubleshooting log for the view engine, predominantly focusing on the changing of partition states.

xdcr

Troubleshooting log for the Cross Data Center Replication (XDCR) component used in Couchbase Server versions prior to 4.0.

xdcr_errors

Error-level troubleshooting log for the Cross Data Center Replication (XDCR) component used in Couchbase Server versions prior to 4.0.

xcdr_trace

Trace-level troubleshooting log for the Cross Data Center Replication (XDCR) component used in Couchbase Server versions prior to 4.0. Unless trace-level logging is explicitly turned on this log is empty.

Log File Rotation

The memcached log file is rotated whenever memcached is restarted. Other logs are automatically rotated after they have reached 40MB in size; five rotations being maintained: the current file, plus four compressed rotations:

-rw-rw---- 1 couchbase couchbase 12M Feb 2 16:15 couchdb.log
        -rw-rw---- 1 couchbase couchbase 4.8M Feb 2 16:13 couchdb.log.1.gz
        -rw-rw---- 1 couchbase couchbase 4.5M Jan 30 17:35 couchdb.log.2.gz
        -rw-rw---- 1 couchbase couchbase 3.9M Jan 30 17:34 couchdb.log.3.gz
        -rw-rw---- 1 couchbase couchbase 5.7M Jan 30 17:30 couchdb.log.4.gz

To provide custom rotation-settings for each component, add the following to the static_config file:

{disk_sink_opts_disk_debug,
        [{rotation, [{size, 10485760},
        {num_files, 10}]}]}.

This rotates the debug.log at 10MB, and keeps ten copies of the log: the current log and nine compressed logs.

Changing Log File Locations

The default log location on Linux systems is /opt/couchbase/var/lib/couchbase/logs. To change this location, proceed as follows:

  1. Log in as root or sudo and navigate to the directory where Couchbase Server is installed. For example: /opt/couchbase/etc/couchbase/static_config.

  2. Edit the static_config file: change the error_logger_mf_dir variable, specifying a different directory. For example: {error_logger_mf_dir, "/home/user/cb/opt/couchbase/var/lib/couchbase/logs"}

  3. Stop and restart Couchbase Server. See Startup and Shutdown.

Changing Log File Levels

The default logging level for all log files is debug, except for couchdb, which is set to info. Either persistent or dynamic changes can be made to logging levels.

Persistent Changes

Persistent means that changes continue to be implemented, should a Couchbase Server reboot occur. To make a persistent change on Linux systems, proceed as follows:

  1. Log in as root or sudo, and navigate to the directory where you installed Couchbase. For example: /opt/couchbase/etc/couchbase/static_config.

  2. Edit the static_config file and change the desired log component. (Parameters with the loglevel_ prefix establish logging levels.)

  3. Stop and restart Couchbase Server. See Startup and Shutdown.

Dynamic Changes

Dynamic means that if a Couchbase Server reboot occurs, the changed logging levels revert to the default. To make a dynamic change, execute a curl POST command, using the following syntax:

curl -X POST -u adminName:adminPassword HOST:PORT/diag/eval \
              -d ‘ale:set_loglevel(<log_component>,<logging_level>).’

  • log_component: The default log level (except couchdb) is debug; for example ns_server. The available loggers are ns_server, couchdb, user, Menelaus, ns_doctor, stats, rebalance, cluster, views, mapreduce_errors , xdcr and error_logger.

  • logging_level: The available log levels are debug, info, warn, and error.

    curl -X POST -u Administrator:password http://127.0.0.1:8091/diag/eval \
                -d 'ale:set_loglevel(ns_server,error).

Collecting Logs Using the CLI

To collect logs, use the CLI command cbcollect_info.

To start and stop log-collection, and to collect log-status, use:

Collecting Logs Using the REST API

The Logs REST API provides the endpoints for retrieving log and diagnostic information.

To retrieve log information use the /diag and /sasl_logs REST endpoints.