Storage Settings

A Secondary Index can be saved in either of two ways: standard or memory-optmized.

Memory-Optimized Index Storage

Memory-optimized index-storage allows high-speed maintenance and scanning; since the index is kept fully in memory at all times. A snapshot of the index is maintained on disk, to permit rapid recovery if node-failures are experienced. To be consistently beneficial, memory-optimized index-storage requires that all nodes running the Index Service have a memory quota sufficient for the number and size of their resident indexes, and for the frequency with which the indexes will be updated and scanned.

Memory-optimized index-storage may be less suitable for nodes where memory is constrained; since whenever the Index Service memory-quota is exceeded, indexes on the node can neither be updated nor scanned. In such circumstances, an error-notification is provided; and although the indexes remain in ONLINE state, traffic is routed away from the node. Before index-operations can resume, memory must be freed.

In cases where recovery requires an Index-Service node to be restarted, the node’s resident memory-optimized indexes are rebuilt from the snapshots retained on disk. Following the node’s restart, these indexes remain in the BUILDING state until all information has been read into memory: then, final updates are made with the indexes in ONLINE state. Note that once a rebuilt index is thus available, queries with consistency=request_plus or consistency=at_plus fail, if the specified timestamp exceeds the last timestamp processed by given index. However, queries with consistency=unbounded execute normally. For information on these settings, see Index Availability and Performance.

To rescue a node from an out-of-memory situation, consider taking one or more of the following actions:

  • Increase the index-memory quota, to give indexes additional memory for request-processing.

  • Remove less important indexes from the node, to free up memory.

  • Remove buckets with indexes: removing a bucket automatically removes all the dependent indexes.

  • Flush buckets that have indexes: flushing a bucket deletes all data in a bucket; and even if there are pending updates not yet processed, flushing causes all indexes to drop their own data.

    Note that attempting to delete bucket-data selectively during an out-of-memory condition does not succeed in decreasing memory-usage; since without memory, such requested deletions cannot themselves be processed.

Standard Index Storage

Standard is the default storage-setting for Secondary Indexes: the indexes are saved on disk; in a disk-optimized format that uses both memory and disk for index-update and scanning.

The performance of standard index storage depends on overall I/O performance. Each index saved with the standard option has two write modes:

  • Circular Write Mode: Writes changes to the end of the index-file, until the relative index fragmentation exceeds 65%. Block reuse is then triggered: new data is written into stale blocks where possible, rather than to the end of the file, so as to optimize I/O throughput. Full compaction runs in accordance with the value of the Circular write mode with day + time interval trigger setting: see Index Fragmentation. Note, however, that the index-fragmentation data-size is not significantly changed by compaction.

  • Append-only Write Mode: Writes changes to the end of the index-file, invalidating existing pages within the index file, and requiring frequent, full compaction.

By default, Couchbase Server uses Circular Write Mode for standard index storage. Append-only Write Mode is provided for backwards compatibility with previous versions. These and other storage-settings are described in detail in Configuring Auto-Compaction.

Changing Index-Storage Settings

Settings are established at cluster-initialization for all indexes on the cluster, across all buckets. Following cluster-initialization, to change from one setting to the other, all nodes running the Index Service must be removed. If the cluster is single-node, uninstall and reinstall Couchbase Server. If the cluster is multi-node, and only some of the nodes host the Index Service, proceed as follows:

  1. Identify the nodes running the Index Service.

  2. Remove each of the nodes running the Index Service. Note that as Index-Service nodes are removed, so are the indexes they contain; and in consequence, any ongoing queries fail.

  3. Perform a rebalance.

  4. Change the Index-Storage Settings for the cluster.

  5. Add new Index-Service nodes, and confirm the revised storage mode.

For information on adding and removing nodes, and on rebalancing a cluster, see Manage Nodes and Clusters.