cbstats

Retrieve lower level statistics with the cbstats utility, which provides deep insight into what occurs within a cluster.

Syntax

The basic syntax is:

cbstats host:port [common options] command [command options]

Description

The cbstats tool queries differently depending on the port:

  • When using the Couchbase data port 11210, it will give you operations per node per bucket:

    /opt/couchbase/bin/cbstats -b beer-sample localhost:11210 all | grep \ curr_items:
    curr_items:                                 3711
  • When using the Couchbase legacy port 11211, it will query via MOXI and provide cluster-wide stats:

    /opt/couchbase/bin/cbstats -b beer-sample localhost:11211 all | grep \ curr_items:
    curr_items:                                 7303

If a named bucket is not provided, the server applies the setting to any default bucket that exists at the specified node.

The tool is found in the following locations:

Platform Location

Linux

/opt/couchbase/bin/cbstats

Windows

C:\Program Files\Couchbase\Server\bin\cbstats.exe

Mac OS X

/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbstats

This following table shows top-level statistics:

Table 1. Top-level statistics
Stat Description

uuid*

The unique identifier for the bucket.

ep_version

Version number of ep_engine.

ep_storage_age

Seconds since most recently stored object was initially queued.

ep_storage_age_highwat

ep_storage_age high water mark.

ep_startup_time

System-generated engine startup time.

ep_data_age

Seconds since most recently stored object was modified.

ep_data_age_highwat

ep_data_age high water mark.

ep_too_young

Number of times an object was not stored due to being too young.

ep_too_old

Number of times an object was stored after being dirty too long.

ep_total_enqueued

Total number of items queued for persistence.

ep_total_new_items

Total number of persisted new items.

ep_total_del_items

Total number of persisted deletions.

ep_total_persisted

Total number of items persisted.

ep_item_flush_failed

Number of times an item failed to flush due to storage errors.

ep_item_commit_failed

Number of times a transaction failed to commit due to storage errors.

ep_item_begin_failed

Number of times a transaction failed to start due to storage errors.

ep_expired_access

Number of times an item was expired on application access.

ep_expired_pager

Number of times an item was expired by ep engine item pager.

ep_item_flush_expired

Number of times an item is not flushed due to the expiry of the item.

ep_queue_size

Number of items queued for storage.

ep_flusher_todo

Number of items currently being written.

ep_flusher_state

Current state of the flusher thread.

ep_commit_num

Total number of write commits.

ep_commit_time

Number of milliseconds of most recent commit.

ep_commit_time_total

Cumulative milliseconds spent committing.

ep_vbucket_del

Number of vBucket deletion events.

ep_vbucket_del_fail

Number of failed vBucket deletion events.

ep_vbucket_del_max_walltime

Max wall time (µs) spent by deleting a vBucket.

ep_vbucket_del_avg_walltime

Avg wall time (µs) spent by deleting a vBucket.

ep_flush_duration_total

Cumulative seconds spent flushing.

ep_flush_all

True if disk flush_all is scheduled.

ep_num_ops_get_meta

Number of getMeta operations.

ep_num_ops_set_meta

Number of setWithMeta operations.

ep_num_ops_del_meta

Number of delWithMeta operations..

ep_num_ops_set_meta_res_failed

Number of setWithMeta ops that failed conflict resolution.

ep_num_ops_del_meta_res_failed

Number of delWithMeta ops that failed conflict resolution.

ep_num_ops_set_ret_meta

Number of setRetMeta operations.

ep_num_ops_del_ret_meta

Number of delRetMeta operations.

curr_items

Num items in active vbuckets (temp + live).

curr_temp_items

Num temp items in active vbuckets.

curr_items_tot

Num current items including those not active (replica, dead and pending states).

ep_kv_size

Memory used to store item metadata, keys and values, no matter the vBucket’s state. If an item’s value is ejected, this stats will be decremented by the size of the item’s value.

ep_value_size

Memory used to store values for resident keys

ep_overhead

Extra memory used by transient data like persistence queues, replication queues, checkpoints, etc.

ep_mem_low_wat

Low water mark for auto-evictions.

ep_mem_high_wat

High water mark for auto-evictions.

ep_total_cache_size

The total byte size of all items, no matter the vBucket’s state, no matter if an item’s value is ejected.

ep_oom_errors

Number of times unrecoverable OOMs happened while processing operations.

ep_tmp_oom_errors

Number of times temporary OOMs happened while processing operations.

ep_mem_tracker_enabled

True if memory usage tracker is enabled.

ep_bg_fetched

Number of items fetched from disk.

ep_bg_meta_fetched

Number of meta items fetched from disk.

ep_bg_remaining_jobs

Number of remaining bg fetch jobs.

ep_max_bg_remaining_jobs

Max number of remaining bg fetch jobs that we have seen in the queue so far.

ep_tap_bg_fetched

Number of tap disk fetches.

ep_tap_bg_fetch_requeued

Number of times a tap bg fetch task is requeued.

ep_num_pager_runs

Number of times we ran pager loops to seek additional memory.

ep_num_expiry_pager_runs

Number of times we ran expiry pager loops to purge expired items from memory/disk.

ep_num_access_scanner_runs

Number of times we ran access scanner to snapshot working set.

ep_access_scanner_num_items

Number of items that last access scanner task swept to access log.

ep_access_scanner_task_time

Time of the next access scanner task (GMT).

ep_access_scanner_last_runtime

Number of seconds that last access scanner task took to complete.

ep_items_rm_from_checkpoints

Number of items removed from closed unreferenced checkpoints.

ep_num_value_ejects

Number of times item values got ejected from memory to disk.

ep_num_eject_failures

Number of items that could not be ejected.

ep_num_not_my_vbuckets

Number of times Not My vBucket exception happened during runtime.

ep_tap_keepalive

Tap keepalive time.

ep_dbname

DB path.

ep_io_num_read

Number of I/O read operations.

ep_io_num_write

Number of I/O write operations.

ep_io_read_bytes

Number of bytes read (key + values).

ep_io_write_bytes

Number of bytes written (key + values).

ep_io_total_read_bytes

Number of bytes read (total, including Couchstore B-Tree and other overheads).

ep_io_total_write_bytes

Number of bytes written (total, including Couchstore B-Tree and other overheads).

ep_io_compaction_read_bytes

Number of bytes read (compaction only, including Couchstore B-Tree and other overheads).

ep_io_compaction_write_bytes

Number of bytes written (compaction only, including Couchstore B-Tree and other overheads).

ep_pending_ops

Number of ops awaiting pending vbuckets.

ep_pending_ops_total

Total blocked pending ops since reset.

ep_pending_ops_max

Max ops seen awaiting 1 pending vBucket.

ep_pending_ops_max_duration

Max time (µs) used waiting on pending vbuckets.

ep_bg_num_samples

The number of samples included in the average.

ep_bg_min_wait

The shortest time (µs) in the wait queue.

ep_bg_max_wait

The longest time (µs) in the wait queue.

ep_bg_wait_avg

The average wait time (µs) for an item before it’s serviced by the dispatcher.

ep_bg_min_load

The shortest load time (µs).

ep_bg_max_load

The longest load time (µs).

ep_bg_load_avg

The average time (µs) for an item to be loaded from the persistence layer.

ep_num_non_resident

The number of non-resident items.

ep_bg_wait

The total elapse time for the wait queue.

ep_bg_load

The total elapse time for items to be loaded from the persistence layer.

ep_allow_data_loss_during_shutdown

Whether data loss is allowed during server shutdown.

ep_alog_block_size

Access log block size.

ep_alog_path

Path to the access log.

ep_alog_sleep_time

Interval between access scanner runs in minutes.

ep_alog_task_time

Hour in GMT time when access scanner task is scheduled to run.

ep_backend

The backend that is being used for data persistence.

ep_bg_fetch_delay

The amount of time to wait before doing a background fetch.

ep_chk_max_items

The number of items allowed in a checkpoint before a new one is created.

ep_chk_period

The maximum lifetime of a checkpoint before a new one is created.

ep_chk_persistence_remains

Number of remaining vbuckets for checkpoint persistence.

ep_chk_persistence_timeout

Timeout for vBucket checkpoint persistence.

ep_chk_remover_stime

The time interval for purging closed checkpoints from memory.

ep_config_file

The location of the ep-engine config file.

ep_couch_bucket

The name of this bucket.

ep_couch_host

The hostname that the CouchDB views server is listening on.

ep_couch_port

The port the CouchDB views server is listening on.

ep_couch_reconnect_sleeptime

The amount of time to wait before reconnecting to CouchDB.

ep_couch_response_timeout

Length of time to wait for a response from CouchDB before reconnecting.

ep_data_traffic_enabled

Whether or not data traffic is enabled for this bucket.

ep_degraded_mode

True if the engine is either warming up or data traffic is disabled.

ep_exp_pager_stime

The time interval for purging expired items from memory.

ep_failpartialwarmup

True if we want kill the bucket if warmup fails.

ep_flushall_enabled

True if this bucket enables the use of the flush_all command.

ep_getl_default_timeout

The default getl lock duration.

ep_getl_max_timeout

The maximum getl lock duration.

ep_ht_locks

The amount of locks per vb hashtable.

ep_ht_size

The initial size of each vb hashtable.

ep_item_num_based_new_chk

True if the number of items in the current checkpoint plays a role in a new checkpoint creation.

ep_keep_closed_chks

True if we want to keep the closed checkpoints for each vBucket unless the memory usage is above high water mark.

ep_max_checkpoints

The maximum amount of checkpoints that can be in memory per vBucket.

ep_max_item_size

The maximum value size.

ep_max_size

The maximum amount of memory this bucket can use.

ep_max_vbuckets

The maximum amount of vbuckets that can exist in this bucket.

ep_mutation_mem_threshold

The ratio of total memory available that we should start sending temp oom or oom message when hitting.

ep_pager_active_vb_pcnt

Active vbuckets paging percentage.

ep_tap_ack_grace_period

The amount of time to wait for a tap acks before disconnecting.

ep_tap_ack_initial_sequence_number

The initial sequence number for a tap ack when a tap stream is created.

ep_tap_ack_interval

The amount of messages a tap producer should send before requesting an ack.

ep_tap_ack_window_size

The maximum amount of ack requests that can be sent before the consumer sends a response ack. When the window is full the tap stream is paused..

ep_tap_backfill_resident

The resident ratio for deciding how to do backfill. If under the ratio we schedule full disk backfill. If above the ratio then we do bg fetches for non-resident items.

ep_tap_backlog_limit

The maximum amount of backfill items that can be in memory waiting to be sent to the tap consumer.

ep_tap_backoff_period

The number of seconds the tap connection.

ep_tap_bg_fetch_requeued

Number of times a tap bg fetch task is requeued.

ep_tap_bg_max_pending

The maximum number of bg jobs a tap connection may have.

ep_tap_noop_interval

Number of seconds between a noop is sent on an idle connection.

ep_tap_requeue_sleep_time

The amount of time to wait before a failed tap item is requeued.

ep_tap_throttle_cap_pcnt

Percentage of total items in write queue at which we throttle tap input.

ep_tap_throttle_queue_cap

Max size of a write queue to throttle incoming tap input.

ep_tap_throttle_threshold

Percentage of max mem at which we begin NAKing tap input.

ep_uncommitted_items

The amount of items that have not been written to disk.

ep_vb0

Whether vBucket 0 should be created by default.

ep_waitforwarmup

True if we should wait for the warmup process to complete before enabling traffic.

ep_warmup

Shows if warmup is enabled / disabled.

ep_warmup_batch_size

The size of each batch loaded during warmup.

ep_warmup_dups

Number of Duplicate items encountered during warmup.

ep_warmup_min_items_threshold

Percentage of total items warmed up before we enable traffic.

ep_warmup_min_memory_threshold

Percentage of max mem warmed up before we enable traffic.

ep_warmup_oom

The amount of oom errors that occurred during warmup.

ep_warmup_thread

The status of the warmup thread.

ep_warmup_time

The amount of time warmup took.

  • uuid is specified per bucket, therefore you must specify the bucket you are interested in. If you don’t specify a bucket (-b XXX), then you will get information for the default bucket, if it exists. If the default bucket does not exist, the output will be empty.

The following table shows replica vBucket statistics.

Stat Description

vb_replica_num

Number of replica vBuckets.

vb_replica_curr_items

Number of in memory items.

vb_replica_num_non_resident

Number of non-resident items.

vb_replica_perc_mem_resident

% memory resident.

vb_replica_eject

Number of times item values got ejected..

vb_replica_expired

Number of times an item was expired.

vb_replica_ht_memory

Memory overhead of the hashtable.

vb_replica_itm_memory

Total item memory.

vb_replica_meta_data_memory

Total metadata memory.

vb_replica_ops_create

Number of create operations.

vb_replica_ops_update

Number of update operations.

vb_replica_ops_delete

Number of delete operations.

vb_replica_ops_reject

Number of rejected operations.

vb_replica_queue_size

Replica items in disk queue.

vb_replica_queue_memory

Memory used for disk queue.

vb_replica_queue_age

Sum of disk queue item age in milliseconds.

vb_replica_queue_pending

Total bytes of pending writes.

vb_replica_queue_fill

Total enqueued items.

vb_replica_queue_drain

Total drained items.

The following table shows active vBucket statistics:

Stat Description

vb_active_num

Number of active vBuckets.

vb_active_curr_items

Number of in memory items.

vb_active_num_non_resident

Number of non-resident items.

vb_active_perc_mem_resident

% memory resident.

vb_active_eject

Number of times item values got ejected.

vb_active_expired

Number of times an item was expired.

vb_active_ht_memory

Memory overhead of the hashtable.

vb_active_itm_memory

Total item memory.

vb_active_meta_data_memory

Total metadata memory.

vb_active_ops_create

Number of create operations.

vb_active_ops_update

Number of update operations.

vb_active_ops_delete

Number of delete operations.

vb_active_ops_reject

Number of rejected operations.

vb_active_queue_size

Active items in disk queue.

vb_active_queue_memory

Memory used for disk queue.

vb_active_queue_age

Sum of disk queue item age in milliseconds.

vb_active_queue_pending

Total bytes of pending writes.

vb_active_queue_fill

Total enqueued items.

vb_active_queue_drain

Total drained items.

The following table shows pending vBucket statistics:

Stat Description

vb_pending_num

Number of pending vBuckets.

vb_pending_curr_items

Number of in memory items.

vb_pending_num_non_resident

Number of non-resident items.

vb_pending_perc_mem_resident

% memory resident.

vb_pending_eject

Number of times item values got ejected.

vb_pending_expired

Number of times an item was expired.

vb_pending_ht_memory

Memory overhead of the hashtable.

vb_pending_itm_memory

Total item memory.

vb_pending_meta_data_memory

Total metadata memory.

vb_pending_ops_create

Number of create operations.

vb_pending_ops_update

Number of update operations.

vb_pending_ops_delete

Number of delete operations.

vb_pending_ops_reject

Number of rejected operations.

vb_pending_queue_size

Pending items in disk queue.

vb_pending_queue_memory

Memory used for disk queue.

vb_pending_queue_age

Sum of disk queue item age in milliseconds.

vb_pending_queue_pending

Total bytes of pending writes.

vb_pending_queue_fill

Total enqueued items.

vb_pending_queue_drain

Total drained items.

The majority of cbstats commands are predominantly used by Couchbase internally and to help resolve customer support incidents.

Options

The following are the command options:

Options Description

-h, --help

Shows the help message and exits.

-a

Iterates over all buckets.

-j

Outputs the result in JSON format.

-8

Forces the use of UTF8 symbols in the output.

-b [BUCKETNAME]

The bucket to get the statistics from. Default: default.

-u [USERNAME]

The username to authenticate. Default: the bucket name.

-p [PASSWORD]

The password for the bucket, if one exists.

-S

Reads the password from stdin.

Examples

Example with two buckets, default and c:

$ cbstats localhost:11210 -b default uuid
	uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f
	$ cbstats localhost:11210 uuid
	uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f
	$ cbstats localhost:11210 -b c uuid
	uuid: 1de74b8dc0981abbc6481c5276b91be1

Example after the default bucket was deleted, and only the c bucket was left:

$ cbstats localhost:11210 -b default uuid
	Authentication error for default
	$ cbstats localhost:11210 uuid
	<nothing>
	$ cbstats localhost:11210 -b c uuid
	uuid: 1de74b8dc0981abbc6481c5276b91be1

Example for getting statistics for timings:

To get statistics, for example, timings on host 10.5.2.117:

cbstats 10.5.2.117:11210 -u Administrator -p password -b beer-sample timings

Response:

 disk_commit (1024 total)
    0 - 1s : (100.00%) 1024 ###################################################
    Avg    : (     1s)
 get_stats_cmd (30663276 total)
    0 - 1us       : (  0.05%)    14827
    1us - 2us     : (  6.56%)  1995778 ##
    2us - 4us     : ( 41.79%) 10804626 ##############
    4us - 8us     : ( 45.20%)  1044043 #
    8us - 16us    : ( 45.49%)    89929
    16us - 32us   : ( 45.90%)   124472
    32us - 64us   : ( 46.38%)   148935
    64us - 128us  : ( 56.17%)  2999690 ###
    128us - 256us : ( 68.57%)  3804009 ####
    256us - 512us : ( 69.91%)   411281
    512us - 1ms   : ( 78.77%)  2717402 ###
    1ms - 2ms     : ( 96.36%)  5391526 #######
    2ms - 4ms     : ( 99.05%)   826345 #
    4ms - 8ms     : ( 99.96%)   278727
    8ms - 16ms    : (100.00%)    11443
    16ms - 32ms   : (100.00%)      217
    32ms - 65ms   : (100.00%)       19
    65ms - 131ms  : (100.00%)        7
    Avg           : (  347us)
 disk_vbstate_snapshot (93280 total)
    32us - 64us   : ( 15.34%) 14308 ######
    64us - 128us  : ( 74.74%) 55413 #########################
    128us - 256us : ( 91.39%) 15532 #######
    256us - 512us : ( 95.69%)  4007 #
    512us - 1ms   : ( 99.49%)  3546 #
    1ms - 2ms     : ( 99.95%)   423
    2ms - 4ms     : ( 99.99%)    43
    4ms - 8ms     : (100.00%)     4
    2s - 4s       : (100.00%)     4
    Avg           : (  190us)
 notify_io (4 total)
    4us - 8us   : ( 25.00%) 1 ############
    16us - 32us : ( 75.00%) 2 ########################
    32us - 64us : (100.00%) 1 ############
    Avg         : (   17us)

Example for cbstats output:

The cbstats output can be used with other command-line tools to sort and filter the data, for example, the watch command.

watch --diff "cbstats \
    ip-10-12-19-81:11210 -b bucket_name -p bucket_password all | egrep 'item|mem|flusher|ep_queue|bg|eje|resi|warm'"

Example for disk_insert:

The following sample statistics show that disk_insert took 8–16µs 9,488 times, 16–32µs 290 times, and so on.

STAT disk_insert_8,16 9488
STAT disk_insert_16,32 290
STAT disk_insert_32,64 73
STAT disk_insert_64,128 86
STAT disk_insert_128,256 48
STAT disk_insert_256,512 2
STAT disk_insert_512,1024 12
STAT disk_insert_1024,2048 1

The same statistics displayed by the CLI cbstats tool looks like the following:

disk_insert (10008 total)
8us - 16us : ( 94.80%) 9488 ###########################################
16us - 32us : ( 97.70%) 290 #
32us - 64us : ( 98.43%) 73
64us - 128us : ( 99.29%) 86
128us - 256us : ( 99.77%) 4
256us - 512us : ( 99.79%) 2
512us - 1ms : ( 99.91%) 12
1ms - 2ms : ( 99.92%) 1