Release Notes

Release notes for the Kafka Connector.

Couchbase Kafka Connector 3.4.0 (2018-10-16)

kafka-connect-couchbase-3.4.0.zip

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0</version>
</dependency>

New features

Version 3.4 introduces rollback mitigation in the form of persistence polling. Messages are published to Kafka only after the database events have been saved to disk on all Couchbase replicas. Rollback mitigation is enabled by default. It may be disabled by setting the persistence polling interval to 0.

The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

New configuration properties

couchbase.flow_control_buffer=128m
couchbase.persistence_polling_interval=100ms

Issues resolved

KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation
KAFKAC-127: [BUGFIX] Source connector can’t reconnect to Couchbase hosted on Docker Container
KAFKAC-124: [BUGFIX] Rollback mitigation causes data loss when opening DCP stream

Couchbase Kafka Connector 3.4.0-beta.1 (2018-07-06)

kafka-connect-couchbase-3.4.0-beta.1.zip

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0-beta.1</version>
</dependency>

Version 3.4.0-beta.1 has a bug that can cause data loss. When rollback mitigation is enabled, some events might not be published to the Kafka topic. See KAFKAC-124 for details. This issue is resolved in the 3.4.0 release.

When database changes are rolled back due to failover of a Couchbase Server node, the connector will now by default prevent the rolled-back changes from being published to the Kafka topic. This is done by buffering changes until they have been persisted to all replicas in the Couchbase cluster (and are unlikely to be rolled back). The connector polls the database at a configurable interval to determine which changes have been persisted. This feature is enabled by default, and may be disabled by setting the polling interval to 0.

The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

New configuration properties:

couchbase.flow_control_buffer=128m
couchbase.persistence_polling_interval=100ms

Issues resolved in this release:

KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation

Couchbase Kafka Connector 3.3.2 GA (2018-07-02)

kafka-connect-couchbase-3.3.2.zip

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.2</version>
</dependency>

The source connector now resumes streaming after a Couchbase Server rollback.

Issues resolved in this release:

KAFKAC-72: [BUGFIX] Connector stops streaming after rollback
KAFKAC-115: [ENHANCEMENT] Upgrade to DCP client 0.18.0

Couchbase Kafka Connector 3.3.1 GA (2018-06-19)

kafka-connect-couchbase-3.3.1.zip

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.1</version>
</dependency>

The sink connector can now set expiration times for documents, and can assign document IDs by composing values from multiple fields.

The source connector now continues streaming after a graceful failover.

Issues resolved in this release:

KAFKAC-104: [ENHANCEMENT] Support for setting TTL on items being written to Couchbase
KAFKAC-106: [ENHANCEMENT] Support assigning document ID from multiple fields
KAFKAC-109: [ENHANCEMENT] KAFKAC-109 Upgrade to Java client 2.5.9 and DCP client 0.17.0
JDCP-85: [BUGFIX] Client stops receiving some events after failover
JDCP-94: [BUGFIX] Should never pass null ChannelFlowController to event handlers

Couchbase Kafka Connector 3.3.0 GA (2018-05-15)

kafka-connect-couchbase-3.3.0.zip

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.0</version>
</dependency>

Compression is now enabled by default (requires Couchbase Server 5.5 or later).

IPv6 is now enabled by default. To disable IPv6 in a dual-stack environment, set the new couchbase.forceIPv4 config property to true.

Resolved a regression that caused the connector to need a restart after a Couchbase Server restart.

Couchbase password and SSL keystore password may be specified using via KAFKA_COUCHBASE_PASSWORD and KAFKA_COUCHBASE_SSL_KEYSTORE_PASSWORD environment variables.

The sink connector can now modify documents using the sub-document API (many thanks to community member Didier Caron).

Issues resolved in this release:

KAFKAC-97: [BUGFIX] Kafka Connector needs a restart after Couchbase Server restart
KAFKAC-102: [ENHANCEMENT] Enable compression by default
KAFKAC-100: [ENHANCEMENT] Enable IPv6 by default, add first-class config option
KAFKAC-96: [ENHANCEMENT] Support setting password via environment variable
KAFKAC-98: [ENHANCEMENT] Sink: Support sub-document access (Didier Caron)
KAFKAC-103: [ENHANCEMENT] Upgrade Couchbase dependencies

Couchbase Kafka Connector 3.2.3 GA (2018-02-20)

Keepalive is now enabled on the Couchbase Server connection. This prevents the server from dropping idle connections, and enables dead connection detection. Thanks to community member Patrik Nordebo (patrikn).

A new config key "couchbase.log_redaction" controls whether sensitive info in connector log messages is tagged for redaction. Options are "NONE", "PARTIAL", and "FULL".

A new source config key "couchbase.compression" can be used to enable compression when reading from Couchbase Server 4.5 and later. Options are "DISABLED", "ENABLED", and "FORCED".

Issues resolved in this release:

KAFKAC-89: [ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo)
KAFKAC-82: [FEATURE] Implement log redaction for Kafka Connector
KAFKAC-90: [FEATURE] Source: Add config settings to enable compression

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.3</version>
</dependency>

kafka-connect-couchbase-3.2.3.zip

Couchbase Kafka Connector 3.2.2 GA (19 December 2017)

The source connector now does a better job of reporting abnormal termination. Thanks to community member p_mx (tiny1990).

A new config key "couchbase.stream_from" lets you tell the source connector when in Couchbase history to start streaming from. Options are "BEGINNING", "NOW", "SAVED_OFFSET_OR_BEGINNING", and "SAVED_OFFSET_OR_NOW".

When the sink connector receives a Kafka message with a null value, it now deletes the Couchbase document whose ID matches the Kafka message key. (Previous versions would terminate when a null value was encountered.)

You can now specify durability requirements for the sink connector’s write operations via two new config keys: "couchbase.durability.persist_to" and "couchbase.durability.replicate_to".

Issues resolved in this release:

KAFKAC-84: [FEATURE] Sink: Allow setting durability requirements for Couchbase writes
KAFKAC-85: [FEATURE] Sink: Support deletion
KAFKAC-86: [FEATURE] Source: Restart from a given state / offset

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.2</version>
</dependency>

kafka-connect-couchbase-3.2.2.zip

Couchbase Kafka Connector 3.2.1 GA (8 November 2017)

Fixes a regression in 3.2.0 that prevented the sink connector from working.

Issues resolved in this release:

KAFKAC-83: [BUGFIX] Sink connector fails due to UnsupportedOperationException in JsonBinaryTranscoder.newDocument.
KAFKAC-81: [ENHANCEMENT] Document how to customize source handler

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.1</version>
</dependency>

kafka-connect-couchbase-3.2.1.zip

Couchbase Kafka Connector 3.2.0 GA (17 October 2017)

The sink connector is now officially supported.

Multiple source connector instances reading from the same Couchbase bucket can now manage their state independently. To enable this feature, assign a unique name to each connector and set the new config property compat.connector_name_in_offsets to true.

The source connector has a new, more flexible SourceHandler extension point intended to replace the Converter interface. By providing a custom SourceHandler, a developer can filter events, route messages to a topic other than the configured default topic, and control the format of the Kafka message.

The default Kafka message format now includes bucket and vBucketUuid fields, which may be used along with the partition (vBucketID) and bySeqno fields to construct a Couchbase MutationToken.

The sink connector now allows the Couchbase document ID to come from a field of the document. This feature is controlled by two new config properties, couchbase.document.id and couchbase.remove.document.id.

The Couchbase client libraries are upgraded to the latest versions, with performance enhancements and other improvements for Couchbase Server 5.0.

The following classes are scheduled for removal in version 4.0.0 of the connector:

com.couchbase.connect.kafka.converter.Converter - Deprecated in favor of SourceHandler.
com.couchbase.connect.kafka.converter.SchemaConverter - Deprecated in favor of DefaultSchemaSourceHandler.

Issues resolved in this release:

KAFKAC-69: [FEATURE] Allow Source connector to split DCP stream and write into separate topics.
KAFKAC-70: [FEATURE] Allow using connector name in offset storage namespace
KAFKAC-77: [FEATURE] Allow setting document ID from message field.
KAFKAC-78: [ENHANCEMENT] Upgrade Couchbase java-client to version 2.5.1, dcp-client to version 0.12.0
KAFKAC-79: [ENHANCEMENT] Use custom doc transcoder to reduce memory copies.
KAFKAC-80: [FEATURE] MutationToken Enabled in Kafka Connector to N1QL at plus.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.0</version>
</dependency>

kafka-connect-couchbase-3.2.0.zip

Couchbase Kafka Connector 3.1.3 GA (31 May 2017)

Version 3.1.3 is maintenance release.

KAFKAC-71: Support for RBAC credentials on Couchbase Server 5+. Java DCP Client updated to 0.10.0.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.1.3</version>
</dependency>

kafka-connect-couchbase-3.1.3.zip

Couchbase Kafka Connector 3.1.2 GA (14 March 2017)

Version 3.1.2 is maintenance release.

KAFKAC-66: On backfilling from large bucket, it is possible to get OOM when internal queue is not drained quickly enough to relay the data into Kafka.

kafka-connect-couchbase-3.1.2.zip

Couchbase Kafka Connector 3.1.1 GA (21 February 2017)

Version 3.1.1 is maintenance release. It contains fixes for resuming DCP streams after restart.

KAFKAC-56: Session state might be left partially initialized, which leads to rolling back to sequence number zero (0) and starting from the beginning (duplicating events in Kafka topic).

kafka-connect-couchbase-3.1.1.zip

Couchbase Kafka Connector 3.1.0 GA (03 January 2017)

Version 3.1.0 is GA release.

KAFKAC-55: Sink Connector support

kafka-connect-couchbase-3.1.0.zip

Couchbase Kafka Connector 3.0.0 GA (14 December 2016)

Version 3.0.0 is GA release. It brings documentation update.

kafka-connect-couchbase-3.0.0.zip

Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)

Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings documentation update, feature enhancements and bug fixes

KAFKAC-52: Support for SSL connections
Update dependencies: dcp-client to 0.7.0, and confluent libraries up to versions shipped with 3.1.1
Cleanup various configuration workarounds for platform 3.0

kafka-connect-couchbase-3.0.0-BETA.zip

Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)

Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.

KAFKAC-54: Create example of using in Kafka Stream to process events from Couchbase
Rename internal classes, and make configuration more consistent with other connectors (e.g. instead of timeout_ms, use timeout.ms)
Allow to override internal convertor into SourceRecord, and allow to inject Filter class to skip events before writing into Kafka

kafka-connect-couchbase-3.0.0-DP4.zip

Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)

Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It implements new features and also includes bug fixes to previous release.

KAFKAC-50: Allow to buffer DCP snapshots for consistent writes.
KAFKAC-51: Specify key for SourceRecord. Allows to use multiple Kafka partitions.
KAFKAC-53: Node-aware distribution of partitions for Tasks. Reduces amount of resources allocated on the server.

kafka-connect-couchbase-3.0.0-DP3.zip

Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)

Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. It improves configuration. And now can maintain replication state, which allow to resume transmission.

kafka-connect-couchbase-3.0.0-DP2.zip

Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)

Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.

kafka-connect-couchbase-3.0.0-DP1.zip

Parent topic: Kafka Connector

Previous topic: Couchbase Sample with Kafka Streams