Memory

Memory-quota allocation on Couchbase Server occurs per service (except in the case of the Query Service, which does not require a specific allocation). This allows the availability of memory-resources to be tuned according to the assignment of services, node by node. Note that the Data Service must run on at least one node; and that on each of those nodes, quotas for buckets, specified at the time of bucket-creation, are subtracted from the quota allocated to the Data Service.

The memory-quota allocation specified for a given service applies to every instance of that service across the cluster. For example, if 2048 MB is specified for the Analytics Service, and the Analytics Service is run on three of a cluster’s five nodes, each of the three instances of the service is duly allocated 2048 MB. Note that it is not possible to have different memory allocations across multiple instances of the same service within a single cluster.

By default, Couchbase Server allows 80% of a node’s total available memory to be allocated to the server and its services. Consequently, if a node’s total available memory is 100 GB, any attempt to allocate memory beyond 80 GB produces an error.

Instructions on how to allocate memory quotas to services when initializing a new cluster can be found in the section Create a Cluster.

When a node is added to a cluster, the Default Configuration, as established by the set-up of the first node in the cluster, is available: this covers all configurable elements, including memory quotas. However, if insufficient memory for the default configuration is available on the new node, the default configuration is prohibited: in such cases, settings for the new node can be custom-configured, allowing an appropriate subset of services to be specified.

If, when the initial node of a cluster is set up, only a subset of services is assigned, additional nodes can subsequently be added, in order to host additional services: in which case, each new service can be given its initial memory allocation as its node is added.

Instructions on how to add a new node to a cluster can be found in the section xref:manage:manage-nodes/create-cluster.adoc.

The Service Memory Quotas panel on the Settings screen of Couchbase Web Console lists all services running on the cluster, and specifies the memory allocation for each. The panel is interactive, and allows the memory allocations to be changed and saved. If a modification attemptedly exceeds a permitted maximum or minimum, a notification of the error is displayed, and the modification is disallowed. See Cluster for further information.

Minimum-required memory-quotas are:

When Couchbase Server is restarted on a node, the node goes through a warmup process before it restarts the handling of data requests. During this warmup process, data on disk is sequentially reloaded into memory. The time required for the reload depends on the size and configuration of the system, the amount of data persisted on the node, and the ejection policy configured for the buckets.

Frequently used items are identified via a scanner process, which examines an access log, and obtains the appropriate keys. Corresponding items are then loaded with the highest priority. The scanner process is configurable, via the CLI utility cbepctl, with the flush_param parameter. This utility also provides the parameters warmup_min_memory_threshold and warmup_min_item_threshold, which can be used to schedule the resumption of traffic before all items have been reloaded into memory. See set flush_param.

If a bucket’s memory quota is exceeded, items may be ejected from the bucket by the Data Service. Different ejection methods are available, and are configured per bucket. Note that in some cases, ejection is configured not to occur. For detailed information, see Buckets.

For each bucket, available memory is managed according to two watermarks, which are mem_low_wat and mem_high_wat. If data is continuously loaded into the bucket, its quantity eventually increases to the value indicated by the mem_low_wat watermark. At this point, paging is triggered, so that memory-availability is continuously optimized. Then, as still more data is loaded, the data’s quantity increases to the value indicated by the mem_high_wat watermark. If, based on the bucket’s configuration, items can be ejected from the bucket, the Data Service ejects items from the bucket until the quantity of data has decreased to the mem_low_wat watermark. If the rate at which data continues to be loaded is greater than that at which it is being ejected, or if ejection cannot be performed, the system displays an insufficient memory notification until sufficient memory is available.

Items are selected for ejection based on metadata that each contains, indicating whether the item can be classified as Not Recently Used (NRU). If an item has not been recently used, it is a candidate for ejection.

The relationship of mem_low_wat and mem_high_wat to the bucket’s overall memory quota is illustrated as follows:

The default setting for mem_low_wat is 75%. The default setting for mem_high_wat is 85%. These settings give items from active vBuckets a 40% chance of ejection; and give items from replica vBuckets a 60% chance of ejection. The default settings can be changed by means of the cbepctl utility. See set flush_param.

Scans for items that have expired, and erases them from memory and disk; after which, a tombstone remains for a default period of 3 days. The expiry pager runs every 60 minutes by default: for information on changing the interval, see cbepctl set flush_param. For more information on item-deletion and tombstones, see Expiration.

Over time, Couchbase Server-memory can become fragmented. Each page in memory is typically responsible for holding documents of a specific size-range. Over time, if memory pages assigned to a specific size-range become sparsely populated (due to documents of that size being ejected, or to items changing in size), the unused space in those pages cannot be used for documents of other sizes, until a complete page is free, and that page is re-assigned to a new size. Such effects, which are highly workload-dependent, may result in memory that cannot be used efficiently.

Couchbase Server provides an Active Memory Defragmenter, which periodically scans the cache, to identify pages that are sparsely used. It then repacks the items on those pages, to free up space.