Storage Settings
A Secondary Index can be saved in either of two ways: standard or memory-optmized.
Memory-Optimized Index Storage
Memory-optimized index-storage allows high-speed maintenance and scanning; since the index is kept fully in memory at all times. A snapshot of the index is maintained on disk, to permit rapid recovery if node-failures are experienced. To be consistently beneficial, memory-optimized index-storage requires that all nodes running the Index Service have a memory quota sufficient for the number and size of their resident indexes, and for the frequency with which the indexes will be updated and scanned.
Memory-optimized index-storage may be less suitable for nodes where memory is constrained; since whenever the Index Service memory-quota is exceeded, indexes on the node can neither be updated nor scanned.
In such circumstances, an error-notification is provided; and although the indexes remain in ONLINE
state, traffic is routed away from the node.
Before index-operations can resume, memory must be freed.
In cases where recovery requires an Index-Service node to be restarted, the node’s resident memory-optimized indexes are rebuilt from the snapshots retained on disk.
Following the node’s restart, these indexes remain in the BUILDING
state until all information has been read into memory: then, final updates are made with the indexes in ONLINE
state.
Note that once a rebuilt index is thus available, queries with consistency=request_plus
or consistency=at_plus
fail, if the specified timestamp exceeds the last timestamp processed by given index.
However, queries with consistency=unbounded
execute normally.
For information on these settings, see Index Availability and Performance.
To rescue a node from an out-of-memory situation, consider taking one or more of the following actions:
-
Increase the index-memory quota, to give indexes additional memory for request-processing.
-
Remove less important indexes from the node, to free up memory.
-
Remove buckets with indexes: removing a bucket automatically removes all the dependent indexes.
-
Flush buckets that have indexes: flushing a bucket deletes all data in a bucket; and even if there are pending updates not yet processed, flushing causes all indexes to drop their own data.
Note that attempting to delete bucket-data selectively during an out-of-memory condition does not succeed in decreasing memory-usage; since without memory, such requested deletions cannot themselves be processed.
Standard Index Storage
Standard is the default storage-setting for Secondary Indexes: the indexes are saved on disk; in a disk-optimized format that uses both memory and disk for index-update and scanning.
The performance of standard index storage depends on overall I/O performance. Each index saved with the standard option has two write modes:
-
Circular Write Mode: Writes changes to the end of the index-file, until the relative index fragmentation exceeds 65%. Block reuse is then triggered: new data is written into stale blocks where possible, rather than to the end of the file, so as to optimize I/O throughput. Full compaction runs in accordance with the value of the Circular write mode with day + time interval trigger setting: see Index Fragmentation. Note, however, that the index-fragmentation data-size is not significantly changed by compaction.
-
Append-only Write Mode: Writes changes to the end of the index-file, invalidating existing pages within the index file, and requiring frequent, full compaction.
By default, Couchbase Server uses Circular Write Mode for standard index storage. Append-only Write Mode is provided for backwards compatibility with previous versions. These and other storage-settings are described in detail in Configuring Auto-Compaction.
Changing Index-Storage Settings
Settings are established at cluster-initialization for all indexes on the cluster, across all buckets. Following cluster-initialization, to change from one setting to the other, all nodes running the Index Service must be removed. If the cluster is single-node, uninstall and reinstall Couchbase Server. If the cluster is multi-node, and only some of the nodes host the Index Service, proceed as follows:
-
Identify the nodes running the Index Service.
-
Remove each of the nodes running the Index Service. Note that as Index-Service nodes are removed, so are the indexes they contain; and in consequence, any ongoing queries fail.
-
Perform a rebalance.
-
Change the Index-Storage Settings for the cluster.
-
Add new Index-Service nodes, and confirm the revised storage mode.
For information on adding and removing nodes, and on rebalancing a cluster, see Manage Nodes and Clusters.