cbstats
Retrieve lower level statistics with the cbstats
utility, which provides deep insight into what occurs within a cluster.
Description
The cbstats
tool queries differently depending on the port:
-
When using the Couchbase data port 11210, it will give you operations per node per bucket:
/opt/couchbase/bin/cbstats -b beer-sample localhost:11210 all | grep \ curr_items: curr_items: 3711
-
When using the Couchbase legacy port 11211, it will query via MOXI and provide cluster-wide stats:
/opt/couchbase/bin/cbstats -b beer-sample localhost:11211 all | grep \ curr_items: curr_items: 7303
If a named bucket is not provided, the server applies the setting to any default bucket that exists at the specified node.
The tool is found in the following locations:
Platform | Location |
---|---|
Linux |
/opt/couchbase/bin/cbstats |
Windows |
C:\Program Files\Couchbase\Server\bin\cbstats.exe |
Mac OS X |
/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbstats |
This following table shows top-level statistics:
Stat | Description |
---|---|
uuid* |
The unique identifier for the bucket. |
ep_version |
Version number of ep_engine. |
ep_storage_age |
Seconds since most recently stored object was initially queued. |
ep_storage_age_highwat |
ep_storage_age high water mark. |
ep_startup_time |
System-generated engine startup time. |
ep_data_age |
Seconds since most recently stored object was modified. |
ep_data_age_highwat |
ep_data_age high water mark. |
ep_too_young |
Number of times an object was not stored due to being too young. |
ep_too_old |
Number of times an object was stored after being dirty too long. |
ep_total_enqueued |
Total number of items queued for persistence. |
ep_total_new_items |
Total number of persisted new items. |
ep_total_del_items |
Total number of persisted deletions. |
ep_total_persisted |
Total number of items persisted. |
ep_item_flush_failed |
Number of times an item failed to flush due to storage errors. |
ep_item_commit_failed |
Number of times a transaction failed to commit due to storage errors. |
ep_item_begin_failed |
Number of times a transaction failed to start due to storage errors. |
ep_expired_access |
Number of times an item was expired on application access. |
ep_expired_pager |
Number of times an item was expired by ep engine item pager. |
ep_item_flush_expired |
Number of times an item is not flushed due to the expiry of the item. |
ep_queue_size |
Number of items queued for storage. |
ep_flusher_todo |
Number of items currently being written. |
ep_flusher_state |
Current state of the flusher thread. |
ep_commit_num |
Total number of write commits. |
ep_commit_time |
Number of milliseconds of most recent commit. |
ep_commit_time_total |
Cumulative milliseconds spent committing. |
ep_vbucket_del |
Number of vBucket deletion events. |
ep_vbucket_del_fail |
Number of failed vBucket deletion events. |
ep_vbucket_del_max_walltime |
Max wall time (µs) spent by deleting a vBucket. |
ep_vbucket_del_avg_walltime |
Avg wall time (µs) spent by deleting a vBucket. |
ep_flush_duration_total |
Cumulative seconds spent flushing. |
ep_flush_all |
True if disk flush_all is scheduled. |
ep_num_ops_get_meta |
Number of getMeta operations. |
ep_num_ops_set_meta |
Number of setWithMeta operations. |
ep_num_ops_del_meta |
Number of delWithMeta operations.. |
ep_num_ops_set_meta_res_failed |
Number of setWithMeta ops that failed conflict resolution. |
ep_num_ops_del_meta_res_failed |
Number of delWithMeta ops that failed conflict resolution. |
ep_num_ops_set_ret_meta |
Number of setRetMeta operations. |
ep_num_ops_del_ret_meta |
Number of delRetMeta operations. |
curr_items |
Num items in active vbuckets (temp + live). |
curr_temp_items |
Num temp items in active vbuckets. |
curr_items_tot |
Num current items including those not active (replica, dead and pending states). |
ep_kv_size |
Memory used to store item metadata, keys and values, no matter the vBucket’s state. If an item’s value is ejected, this stats will be decremented by the size of the item’s value. |
ep_value_size |
Memory used to store values for resident keys |
ep_overhead |
Extra memory used by transient data like persistence queues, replication queues, checkpoints, etc. |
ep_mem_low_wat |
Low water mark for auto-evictions. |
ep_mem_high_wat |
High water mark for auto-evictions. |
ep_total_cache_size |
The total byte size of all items, no matter the vBucket’s state, no matter if an item’s value is ejected. |
ep_oom_errors |
Number of times unrecoverable OOMs happened while processing operations. |
ep_tmp_oom_errors |
Number of times temporary OOMs happened while processing operations. |
ep_mem_tracker_enabled |
True if memory usage tracker is enabled. |
ep_bg_fetched |
Number of items fetched from disk. |
ep_bg_meta_fetched |
Number of meta items fetched from disk. |
ep_bg_remaining_jobs |
Number of remaining bg fetch jobs. |
ep_max_bg_remaining_jobs |
Max number of remaining bg fetch jobs that we have seen in the queue so far. |
ep_tap_bg_fetched |
Number of tap disk fetches. |
ep_tap_bg_fetch_requeued |
Number of times a tap bg fetch task is requeued. |
ep_num_pager_runs |
Number of times we ran pager loops to seek additional memory. |
ep_num_expiry_pager_runs |
Number of times we ran expiry pager loops to purge expired items from memory/disk. |
ep_num_access_scanner_runs |
Number of times we ran access scanner to snapshot working set. |
ep_access_scanner_num_items |
Number of items that last access scanner task swept to access log. |
ep_access_scanner_task_time |
Time of the next access scanner task (GMT). |
ep_access_scanner_last_runtime |
Number of seconds that last access scanner task took to complete. |
ep_items_rm_from_checkpoints |
Number of items removed from closed unreferenced checkpoints. |
ep_num_value_ejects |
Number of times item values got ejected from memory to disk. |
ep_num_eject_failures |
Number of items that could not be ejected. |
ep_num_not_my_vbuckets |
Number of times |
ep_tap_keepalive |
Tap keepalive time. |
ep_dbname |
DB path. |
ep_io_num_read |
Number of I/O read operations. |
ep_io_num_write |
Number of I/O write operations. |
ep_io_read_bytes |
Number of bytes read (key + values). |
ep_io_write_bytes |
Number of bytes written (key + values). |
ep_io_total_read_bytes |
Number of bytes read (total, including Couchstore B-Tree and other overheads). |
ep_io_total_write_bytes |
Number of bytes written (total, including Couchstore B-Tree and other overheads). |
ep_io_compaction_read_bytes |
Number of bytes read (compaction only, including Couchstore B-Tree and other overheads). |
ep_io_compaction_write_bytes |
Number of bytes written (compaction only, including Couchstore B-Tree and other overheads). |
ep_pending_ops |
Number of ops awaiting pending vbuckets. |
ep_pending_ops_total |
Total blocked pending ops since reset. |
ep_pending_ops_max |
Max ops seen awaiting 1 pending vBucket. |
ep_pending_ops_max_duration |
Max time (µs) used waiting on pending vbuckets. |
ep_bg_num_samples |
The number of samples included in the average. |
ep_bg_min_wait |
The shortest time (µs) in the wait queue. |
ep_bg_max_wait |
The longest time (µs) in the wait queue. |
ep_bg_wait_avg |
The average wait time (µs) for an item before it’s serviced by the dispatcher. |
ep_bg_min_load |
The shortest load time (µs). |
ep_bg_max_load |
The longest load time (µs). |
ep_bg_load_avg |
The average time (µs) for an item to be loaded from the persistence layer. |
ep_num_non_resident |
The number of non-resident items. |
ep_bg_wait |
The total elapse time for the wait queue. |
ep_bg_load |
The total elapse time for items to be loaded from the persistence layer. |
ep_allow_data_loss_during_shutdown |
Whether data loss is allowed during server shutdown. |
ep_alog_block_size |
Access log block size. |
ep_alog_path |
Path to the access log. |
ep_alog_sleep_time |
Interval between access scanner runs in minutes. |
ep_alog_task_time |
Hour in GMT time when access scanner task is scheduled to run. |
ep_backend |
The backend that is being used for data persistence. |
ep_bg_fetch_delay |
The amount of time to wait before doing a background fetch. |
ep_chk_max_items |
The number of items allowed in a |
ep_chk_period |
The maximum lifetime of a checkpoint before a new one is created. |
ep_chk_persistence_remains |
Number of remaining vbuckets for checkpoint persistence. |
ep_chk_persistence_timeout |
Timeout for vBucket checkpoint persistence. |
ep_chk_remover_stime |
The time interval for purging closed checkpoints from memory. |
ep_config_file |
The location of the ep-engine config file. |
ep_couch_bucket |
The name of this bucket. |
ep_couch_host |
The hostname that the CouchDB views server is listening on. |
ep_couch_port |
The port the CouchDB views server is listening on. |
ep_couch_reconnect_sleeptime |
The amount of time to wait before reconnecting to CouchDB. |
ep_couch_response_timeout |
Length of time to wait for a response from CouchDB before reconnecting. |
ep_data_traffic_enabled |
Whether or not data traffic is enabled for this bucket. |
ep_degraded_mode |
True if the engine is either warming up or data traffic is disabled. |
ep_exp_pager_stime |
The time interval for purging expired items from memory. |
ep_failpartialwarmup |
True if we want kill the bucket if warmup fails. |
ep_flushall_enabled |
True if this bucket enables the use of the flush_all command. |
ep_getl_default_timeout |
The default getl lock duration. |
ep_getl_max_timeout |
The maximum getl lock duration. |
ep_ht_locks |
The amount of locks per vb hashtable. |
ep_ht_size |
The initial size of each vb hashtable. |
ep_item_num_based_new_chk |
True if the number of items in the current checkpoint plays a role in a new checkpoint creation. |
ep_keep_closed_chks |
True if we want to keep the closed checkpoints for each vBucket unless the memory usage is above high water mark. |
ep_max_checkpoints |
The maximum amount of checkpoints that can be in memory per vBucket. |
ep_max_item_size |
The maximum value size. |
ep_max_size |
The maximum amount of memory this bucket can use. |
ep_max_vbuckets |
The maximum amount of vbuckets that can exist in this bucket. |
ep_mutation_mem_threshold |
The ratio of total memory available that we should start sending temp oom or oom message when hitting. |
ep_pager_active_vb_pcnt |
Active vbuckets paging percentage. |
ep_tap_ack_grace_period |
The amount of time to wait for a tap acks before disconnecting. |
ep_tap_ack_initial_sequence_number |
The initial sequence number for a tap ack when a tap stream is created. |
ep_tap_ack_interval |
The amount of messages a tap producer should send before requesting an ack. |
ep_tap_ack_window_size |
The maximum amount of ack requests that can be sent before the consumer sends a response ack. When the window is full the tap stream is paused.. |
ep_tap_backfill_resident |
The resident ratio for deciding how to do backfill. If under the ratio we schedule full disk backfill. If above the ratio then we do bg fetches for non-resident items. |
ep_tap_backlog_limit |
The maximum amount of backfill items that can be in memory waiting to be sent to the tap consumer. |
ep_tap_backoff_period |
The number of seconds the tap connection. |
ep_tap_bg_fetch_requeued |
Number of times a tap bg fetch task is requeued. |
ep_tap_bg_max_pending |
The maximum number of bg jobs a tap connection may have. |
ep_tap_noop_interval |
Number of seconds between a noop is sent on an idle connection. |
ep_tap_requeue_sleep_time |
The amount of time to wait before a failed tap item is requeued. |
ep_tap_throttle_cap_pcnt |
Percentage of total items in write queue at which we throttle tap input. |
ep_tap_throttle_queue_cap |
Max size of a write queue to throttle incoming tap input. |
ep_tap_throttle_threshold |
Percentage of max mem at which we begin NAKing tap input. |
ep_uncommitted_items |
The amount of items that have not been written to disk. |
ep_vb0 |
Whether vBucket 0 should be created by default. |
ep_waitforwarmup |
True if we should wait for the warmup process to complete before enabling traffic. |
ep_warmup |
Shows if warmup is enabled / disabled. |
ep_warmup_batch_size |
The size of each batch loaded during warmup. |
ep_warmup_dups |
Number of Duplicate items encountered during warmup. |
ep_warmup_min_items_threshold |
Percentage of total items warmed up before we enable traffic. |
ep_warmup_min_memory_threshold |
Percentage of max mem warmed up before we enable traffic. |
ep_warmup_oom |
The amount of |
ep_warmup_thread |
The status of the warmup thread. |
ep_warmup_time |
The amount of time warmup took. |
-
uuid
is specified per bucket, therefore you must specify the bucket you are interested in. If you don’t specify a bucket (-b XXX
), then you will get information for thedefault
bucket, if it exists. If thedefault
bucket does not exist, the output will be empty.
The following table shows replica vBucket statistics.
Stat | Description |
---|---|
vb_replica_num |
Number of replica vBuckets. |
vb_replica_curr_items |
Number of in memory items. |
vb_replica_num_non_resident |
Number of non-resident items. |
vb_replica_perc_mem_resident |
% memory resident. |
vb_replica_eject |
Number of times item values got ejected.. |
vb_replica_expired |
Number of times an item was expired. |
vb_replica_ht_memory |
Memory overhead of the hashtable. |
vb_replica_itm_memory |
Total item memory. |
vb_replica_meta_data_memory |
Total metadata memory. |
vb_replica_ops_create |
Number of create operations. |
vb_replica_ops_update |
Number of update operations. |
vb_replica_ops_delete |
Number of delete operations. |
vb_replica_ops_reject |
Number of rejected operations. |
vb_replica_queue_size |
Replica items in disk queue. |
vb_replica_queue_memory |
Memory used for disk queue. |
vb_replica_queue_age |
Sum of disk queue item age in milliseconds. |
vb_replica_queue_pending |
Total bytes of pending writes. |
vb_replica_queue_fill |
Total enqueued items. |
vb_replica_queue_drain |
Total drained items. |
The following table shows active vBucket statistics:
Stat | Description |
---|---|
vb_active_num |
Number of active vBuckets. |
vb_active_curr_items |
Number of in memory items. |
vb_active_num_non_resident |
Number of non-resident items. |
vb_active_perc_mem_resident |
% memory resident. |
vb_active_eject |
Number of times item values got ejected. |
vb_active_expired |
Number of times an item was expired. |
vb_active_ht_memory |
Memory overhead of the hashtable. |
vb_active_itm_memory |
Total item memory. |
vb_active_meta_data_memory |
Total metadata memory. |
vb_active_ops_create |
Number of create operations. |
vb_active_ops_update |
Number of update operations. |
vb_active_ops_delete |
Number of delete operations. |
vb_active_ops_reject |
Number of rejected operations. |
vb_active_queue_size |
Active items in disk queue. |
vb_active_queue_memory |
Memory used for disk queue. |
vb_active_queue_age |
Sum of disk queue item age in milliseconds. |
vb_active_queue_pending |
Total bytes of pending writes. |
vb_active_queue_fill |
Total enqueued items. |
vb_active_queue_drain |
Total drained items. |
The following table shows pending vBucket statistics:
Stat | Description |
---|---|
vb_pending_num |
Number of pending vBuckets. |
vb_pending_curr_items |
Number of in memory items. |
vb_pending_num_non_resident |
Number of non-resident items. |
vb_pending_perc_mem_resident |
% memory resident. |
vb_pending_eject |
Number of times item values got ejected. |
vb_pending_expired |
Number of times an item was expired. |
vb_pending_ht_memory |
Memory overhead of the hashtable. |
vb_pending_itm_memory |
Total item memory. |
vb_pending_meta_data_memory |
Total metadata memory. |
vb_pending_ops_create |
Number of create operations. |
vb_pending_ops_update |
Number of update operations. |
vb_pending_ops_delete |
Number of delete operations. |
vb_pending_ops_reject |
Number of rejected operations. |
vb_pending_queue_size |
Pending items in disk queue. |
vb_pending_queue_memory |
Memory used for disk queue. |
vb_pending_queue_age |
Sum of disk queue item age in milliseconds. |
vb_pending_queue_pending |
Total bytes of pending writes. |
vb_pending_queue_fill |
Total enqueued items. |
vb_pending_queue_drain |
Total drained items. |
The majority of cbstats
commands are predominantly used by Couchbase internally and to help resolve customer support incidents.
Options
The following are the command options:
Options | Description |
---|---|
|
Shows the help message and exits. |
|
Iterates over all buckets. |
|
Outputs the result in JSON format. |
|
Forces the use of UTF8 symbols in the output. |
|
The bucket to get the statistics from.
Default: |
|
The username to authenticate. Default: the bucket name. |
|
The password for the bucket, if one exists. |
|
Reads the password from stdin. |
Examples
Example with two buckets, default and c:
$ cbstats localhost:11210 -b default uuid uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f $ cbstats localhost:11210 uuid uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f $ cbstats localhost:11210 -b c uuid uuid: 1de74b8dc0981abbc6481c5276b91be1
Example after the default bucket was deleted, and only the c bucket was left:
$ cbstats localhost:11210 -b default uuid Authentication error for default $ cbstats localhost:11210 uuid <nothing> $ cbstats localhost:11210 -b c uuid uuid: 1de74b8dc0981abbc6481c5276b91be1
Example for getting statistics for timings:
To get statistics, for example, timings on host 10.5.2.117:
cbstats 10.5.2.117:11210 -u Administrator -p password -b beer-sample timings
Response:
disk_commit (1024 total) 0 - 1s : (100.00%) 1024 ################################################### Avg : ( 1s) get_stats_cmd (30663276 total) 0 - 1us : ( 0.05%) 14827 1us - 2us : ( 6.56%) 1995778 ## 2us - 4us : ( 41.79%) 10804626 ############## 4us - 8us : ( 45.20%) 1044043 # 8us - 16us : ( 45.49%) 89929 16us - 32us : ( 45.90%) 124472 32us - 64us : ( 46.38%) 148935 64us - 128us : ( 56.17%) 2999690 ### 128us - 256us : ( 68.57%) 3804009 #### 256us - 512us : ( 69.91%) 411281 512us - 1ms : ( 78.77%) 2717402 ### 1ms - 2ms : ( 96.36%) 5391526 ####### 2ms - 4ms : ( 99.05%) 826345 # 4ms - 8ms : ( 99.96%) 278727 8ms - 16ms : (100.00%) 11443 16ms - 32ms : (100.00%) 217 32ms - 65ms : (100.00%) 19 65ms - 131ms : (100.00%) 7 Avg : ( 347us) disk_vbstate_snapshot (93280 total) 32us - 64us : ( 15.34%) 14308 ###### 64us - 128us : ( 74.74%) 55413 ######################### 128us - 256us : ( 91.39%) 15532 ####### 256us - 512us : ( 95.69%) 4007 # 512us - 1ms : ( 99.49%) 3546 # 1ms - 2ms : ( 99.95%) 423 2ms - 4ms : ( 99.99%) 43 4ms - 8ms : (100.00%) 4 2s - 4s : (100.00%) 4 Avg : ( 190us) notify_io (4 total) 4us - 8us : ( 25.00%) 1 ############ 16us - 32us : ( 75.00%) 2 ######################## 32us - 64us : (100.00%) 1 ############ Avg : ( 17us)
Example for cbstats output:
The cbstats
output can be used with other command-line tools to sort and filter the data, for example, the watch
command.
watch --diff "cbstats \ ip-10-12-19-81:11210 -b bucket_name -p bucket_password all | egrep 'item|mem|flusher|ep_queue|bg|eje|resi|warm'"
Example for disk_insert:
The following sample statistics show that disk_insert
took 8–16µs 9,488 times, 16–32µs 290 times, and so on.
STAT disk_insert_8,16 9488 STAT disk_insert_16,32 290 STAT disk_insert_32,64 73 STAT disk_insert_64,128 86 STAT disk_insert_128,256 48 STAT disk_insert_256,512 2 STAT disk_insert_512,1024 12 STAT disk_insert_1024,2048 1
The same statistics displayed by the CLI cbstats
tool looks like the following:
disk_insert (10008 total) 8us - 16us : ( 94.80%) 9488 ########################################### 16us - 32us : ( 97.70%) 290 # 32us - 64us : ( 98.43%) 73 64us - 128us : ( 99.29%) 86 128us - 256us : ( 99.77%) 4 256us - 512us : ( 99.79%) 2 512us - 1ms : ( 99.91%) 12 1ms - 2ms : ( 99.92%) 1