N1QL Application Continuity

In Couchbase Server 5.0, N1QL introduced multiple performance enhancements enabled by a new internal API between the Query and Index services, which guarantees the accuracy of query results between multiple versions of services while nodes are being upgraded.

Before doing any upgrade, you must have all the required RBAC roles of every Data, Index, and Query service involved in the upgrade. For details about RBAC permissions, see Roles.

This page refers to the new 5.0 API as "API-2" and the pre-5.0 API as "API-1".

API-1 and API-2 is used only between Index nodes and Query nodes since the Data nodes don’t use any API and are unaffected by the Index and Query versions.

While Index 4.x supports API-1 only, Index 5.0 supports both API-1 and API-2, so there is no impact.

As Index nodes and Query nodes are being upgraded, even after all the Query nodes are upgraded, they will continue using API-1 until all the Index nodes are upgraded.

While upgrading larger Couchbase cluster deployments, the cluster may be in a state of running different Couchbase versions of each service; so N1QL Application Continuity supports various upgrade scenarios, such as:

Scenario 1 (D → I → Q): Upgrade in the order of 1) Data nodes, 2) Index nodes, 3) Query nodes.
Scenario 2 (D → Q → I): Upgrade in the order of 1) Data nodes, 2) Query nodes, 3) Index nodes.
Scenario 3 (I → Q → D): Upgrade in the order of 1) Index nodes, 2) Query nodes, 3) Data nodes.
Scenario 4 (I + Q + D): Upgrade with Index+Query or Index+Query+Data on one node.

Consider the following example cluster of 4.5 nodes that applies to all 4.x versions: application continuity Scenario1 Step0

Scenario 1 (D → I → Q)

If you want to use the most common and straight-forward order of upgrading, use the following order: application continuity Scenario1 overview Step 1: Upgrade the Data Service nodes from 4.x to 5.0

Follow the current rolling upgrade procedure for all data nodes, one-by-one.
The applications directly fetching the data will continue without interruption.
Index will continue to get DCP stream and will keep the index up to date.
Query will use the index and data services as they did before 5.0.
Applications that use query and index continue uninterrupted.

Step 2: Upgrade the Index Service nodes from 4.x to 5.0

As you upgrade each index, the indexing client within the Query node will continue to use API-1 to interact with the Index nodes.
Index will continue to get updates from data via DCP and keep the indexes up to date.

Step 3: Upgrade one of the Query Service nodes from 4.x to 5.0

Upgrade the query nodes one-by-one.
As each query node comes online, the indexer client will return API-2, and the query service will use API-2.
You can have more than 1 query service node.
At this point, older (4.x) Query nodes will be using API-1 and new (5.0) Query nodes will be using API-2.
This is because Index nodes support API-1 and API-2 simultaneously.
The indexer client within all Query 5.0 nodes will return API-2.

If a CREATE INDEX with DESC key modifier is submitted to a Query 4.x node, you may get a syntax error.

application continuity Scenario1 Step3 circled Step 4: Upgrade the remaining Query Service nodes from 4.x to 5.0

You can submit new syntaxes (e.g. create index with DESC key) to query nodes version 5.0 only.
These indices will succeed but will not be visible to query nodes 4.x. They’ll be visible to these nodes only after they upgrade to 5.0.
The 5.0 replica feature is implemented in Index 5.0. If a CREATE INDEX is submitted with new options in the WITH clause to Query 4.x, it’ll be passed on to the Index service which can accept or raise an error.
These new features may not be used for query service until all the nodes are upgraded to 5.0.

Scenario 2 (D → Q → I)

If you want to upgrade the Query node before the Index node, use the following order: application continuity Scenario2 overview

Step 1: Upgrade the Data Service nodes from 4.x to 5.0

Follow the current rolling upgrade procedure for all data nodes, one-by-one.
The applications directly fetching the data will continue without interruption.
Index nodes will continue to get DCP stream and will keep the index up to date.
Query nodes will use the index and data services as usual.
Applications that use query and index continue uninterrupted.

application continuity Scenario2 Step1 circled

Step 2: Upgrade the Query Service nodes from 4.x to 5.0

Upgrade the query nodes one-by-one.
You can have more than 1 Query service node.
At this point, both older (4.x) Query nodes and new (5.0) nodes will be using API-1.
The indexer client within all Query nodes (Query 5.0) will continue to return the index API version as API-1.

application continuity Scenario2 Step2 circled

Step 3: Upgrade one of the Index Service nodes from 4.x to 5.0

Upgrade the index nodes one-by-one.
As the index nodes are upgraded, the index client within the query will continue to return API-1.
The newly upgraded Indexer nodes will return API-1 behavior until
- All indexer nodes are upgraded,
- There are no failed-over indexer nodes,
- And the cluster is not under a network partition
The query will dynamically exploit the new indexing features as more nodes are upgraded.
Indexer API (mainly concerned with CREATE INDEX) are capable of using API-2 (Indexer2).

If a CREATE INDEX with DESC key modifier is submitted to a Query 4.x node, you may get a syntax error.

application continuity Scenario2 Step3 circled

Step 4: Upgrade the remaining Index Service nodes from 4.x to 5.0

When the GSI client changes and returns the Index API (affecting query optimization and pushdown) version from API1 to API2, the query nodes will start using the API-2 for subsequent query optimizations.
Previously prepared queries will continue to use the OLD API-1 until they’re re-optimized.
Essentially, for index processing, query nodes will start using API-2 as soon as the indexer returns API-2 (Index2 interface). This is the most critical feature.

application continuity Scenario2 StepDone circled

Scenario 3 (I → Q → D)

If you want to upgrade the Index and Query nodes first, use the following order:

application continuity Scenario3 overview

Step 1: Upgrade the Index Service nodes from 4.x to 5.0

Index 4.x supports API-1 for interaction between index and query.
Index 5.0 uses both API-1 and API-2, so there is no impact.
As you upgrade each index, the indexing client within query will continue to think the indexing service is still API-1 and query service uses API-1 to interact with index.
Index will continue to get updates from data via DCP and keep the indexes up to date.

application continuity Scenario3 Step1 circled

Step 2: Upgrade the Query Service nodes from 4.x to 5.0

Upgrade the query nodes one-by-one.
As each query node comes online, the indexer client will return API-2, and the query service will use API-2.
You can have more than 1 query service node.
At this point, older (4.x) query nodes will be using API-1 and new (5.0) nodes will be using API-2.
This means, Indexes will support API-1 and API-2 simultaneously.
The indexer client within all query nodes (Query 5.0) will continue to return the index api version to be API-2.
You can submit new syntaxes (e.g. create index with DESC key) to query nodes version 5.0 only.
These indices will succeed but will not be visible to query nodes 4.x. They’ll be visible to these nodes only after they upgrade to 5.0.
These new features may not be used for query service until all the nodes are upgraded to 5.0.

If a CREATE INDEX with DESC key modifier is submitted to a Query 4.x node, you may get a syntax error.

The 5.0 replica feature is implemented in Index 5.0. If a CREATE INDEX is submitted with new options in the WITH clause to Query 4.x, it’ll be passed on to index service which might cause an error.

application continuity Scenario3 Step2 circled

Step 3: Upgrade the Data Service nodes from 4.x to 5.0

Follow the current rolling upgrade procedure for all data nodes, one-by-one.
The applications directly fetching the data will continue without interruption.
Index will continue to get DCP stream and will keep the index up to date.
Query will use the index and data services as usual.
Applications that use query and index continue uninterrupted.

application continuity Scenario3 StepDone circled

Scenario 4 (I+Q+D)

If your Index+Query services are on the same node, or if your Index+Query+Data services are on the same node, use the following order:

application continuity Scenario4 overview

Step 1: Upgrade Node 1’s Data Service, then Index Service, then Query Service

The Index node will not rebalance during the upgrade until after the full cluster is upgraded.

The user will have duplicate (equivalent) indexes on the other nodes.

You will have to create the indices manually on the available indexer service.

Upgrade the Data service. Afterwards, the data will automatically rebalance.
Upgrade the Index service. The Index will continue to return Index API-1 and use Spans. (4.x feature)
Upgrade the Query service. The query service will start using the indices in all three index nodes. For query processing, the Query service will use API-1 (returned by Index API).

application continuity Scenario4 Step1 circled node

Step 2: Upgrade Node 2’s Data Service, then Index Service, then Query Service

Upgrade the 2nd node to 5.0, just like Step 1.
Data: Once the second data service is upgraded, the data will get rebalanced.
Index: The second index will get upgraded. Index will continue to return Index API-1, so the spans and features of 4.x will be used.
Query: The query service will start using the indices in all three index nodes. For query processing, query will use API-1 (returned by Index API).

application continuity Scenario4 Step2 circled node

Step 3: Upgrade Node 3’s Data Service, then Index Service, then Query Service

Upgrade the 3rd node to 5.0, just like Step 1 and Step 2.
Data: Once the second data service is upgraded, the data will get rebalanced.
Index: The third index service will get upgraded. Once the index upgrade is complete, the index clients within query nodes will start to return index-API2 as well as all the query nodes.
Query: The query service will start using the indices in all three index nodes. For query processing, query services will use API-1 (returned by Index API).

application continuity Scenario4 Step3 circled node

Summary

In summary, this new Application Continuity feature ensures seamless interactions between N1QL queries and applications regardless of which 4.x or 5.0 version of the Query Services and Index Services are running, however, keep the following performance impact in mind:

Ver 5.0 Query with 5.0 Index

Will take full advantage of the latest features and performance enhancements.

Ver 4.x Query with 4.x Index

Ver 4.x Query with 5.0 Index

Ver 5.0 Query with 4.x Index

Slight behavior changes may be observed, such as a client connected to 5.0 query service may get different error messages compared to 4.x.
Will not fail and will still execute seamlessly with the previous behavior to ensure 100% backward-compatibility.

The N1QL clients are able to use the 5.0 feature and performance enhancements only when the issued queries are processed by 5.0 Query Services or Index services.

EXPLAIN Examples

The above explanation is evident in any EXPLAIN output of a query containing a WHERE clause.

Example 1a: Running 4.6 Query Service with 5.0 Index Service uses the standard IndexScan operator.

SELECT name FROM `travel-sample` WHERE type="hotel" ORDER BY name;

Then the EXPLAIN output would begin:

{
    "plan": {
      "#operator": "Sequence",
      "~children": [
        {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan",
              "index": "def_type",
              "index_id": "690c26a475406147",
              "keyspace": "travel-sample",
              "namespace": "default",
              "spans": [
                {

...

However, when both the query service and index service versions are 5.0, the higher-performance indexer IndexScan2 is used instead of IndexScan.

Example 1b: Running 5.0 Query Service with 5.0 Index Service uses the faster IndexScan2 operator.

SELECT name FROM `travel-sample` WHERE type="hotel" ORDER BY name;

Then the EXPLAIN output would begin:

{
    "plan": {
      "#operator": "Sequence",
      "~children": [
        {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan2",
              "index": "def_type",
              "index_id": "c3d31f22e3b5c798",
              "index_projection": {
                "primary_key": true
              },
...