Release v2.3.1 of the Amazon Kinesis Client Library for Java

Latest Release 2.3.1 (October 20, 2020)

Milestone#53

• Introducing support for processing multiple kinesis data streams with the same KCL 2.x for java consumer application

  • To build a consumer application that can process multiple streams at the same time, you must implement a new
    interface called MultistreamTracker (https://github.com/awslabs/amazon-kinesis-client/blob/0c5042dadf794fe988438436252a5a8fe70b6b0b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/MultiStreamTracker.java)

  • MultistreamTracker will also publish various metrics around the current active streams being processed, the number
    of streams which are deleted at this time period or are pending deletion.

Release v1.14.0 of the Amazon Kinesis Client Library for Java

Release 1.14.0 (August 17, 2020)

Milestone#50

• Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.

• When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.

  • Currently, the following shard filters are supported:
    • AT_TRIM_HORIZON - the response includes all the shards that were open at TRIM_HORIZON.
    • AT_LATEST - the response includes only the currently open shards of the data stream.
    • AT_TIMESTAMP - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
  • ShardFilter is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at KinesisClientLibConfiguration#initialPositionInStreamExtended.
  • For more information about ShardFilter, see the official AWS documentation on ShardFilter.

• Introducing support for the ChildShards response of the GetRecords API to perform lease/shard synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.

  • For KCL 1.x applications, this uses the ChildShards response of the GetRecords API.
  • For more information, see the official AWS Documentation on GetRecords and ChildShard.

• KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.SHARD_END, PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When KinesisClientLibConfiguration#shardSyncStrategyType is set to ShardSyncStrategyType.PERIODIC, leasesRecoveryAuditorInconsistencyConfidenceThreshold is ignored.

  • New configuration options are available to configure PeriodicShardSyncManager in KinesisClientLibConfiguration

  Name	Default	Description
  leasesRecoveryAuditorInconsistencyConfidenceThreshold	3	Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for ShardSyncStrategyType.SHARD_END.

  • New CloudWatch metrics are also now emitted to monitor the health of PeriodicShardSyncManager:

  Name	Description
  NumStreamsWithPartialLeases	Number of streams that had holes in their hash ranges.
  NumStreamsToSync	Number of streams which underwent a full shard sync.

• Introducing deferred lease cleanup. Leases will be deleted asynchronously by LeaseCleanupManager upon reaching SHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.

  • New configuration options are available to configure LeaseCleanupManager.

  Name	Default	Description
  leaseCleanupIntervalMillis	1 minute	Interval at which to run lease cleanup thread.
  completedLeaseCleanupIntervalMillis	5 minutes	Interval at which to check if a lease is completed or not.
  garbageLeaseCleanupIntervalMillis	30 minutes	Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not.

• Including an optimization to KinesisShardSyncer to only create leases for one layer of shards.

• Changing default shard prioritization strategy to be NoOpShardPrioritization to allow prioritization of completed shards. Customers who are upgrading to this version and are reading from TRIM_HORIZON should continue using ParentFirstPrioritization while upgrading.

• Upgrading version of AWS SDK to 1.11.844.

• #719 Upgrading version of Google Protobuf to 3.11.4.

• #712 Allowing KCL to consider lease tables in UPDATING healthy.

Release 2.3.0 of the Amazon Kinesis Client Library for Java

Latest Release (2.3.0 - August 17, 2020)

• Milestone#52

• Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.

• When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.

  • Currently, the following shard filters are supported:
    • AT_TRIM_HORIZON - the response includes all the shards that were open at TRIM_HORIZON.
    • AT_LATEST - the response includes only the currently open shards of the data stream.
    • AT_TIMESTAMP - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
  • ShardFilter is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at RetrievalConfig#initialPositionInStreamExtended.
  • For more information about ShardFilter, see the official AWS documentation on ShardFilter.

• Introducing support for the ChildShards response of the GetRecords and the SubscribeToShard APIs to perform lease/shard synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.

  • For shared throughout consumer applications, this uses the ChildShards response of the GetRecords API. For dedicated throughput (enhanced fan-out) consumer applications, this uses the ChildShards response of the SubscribeToShard API.
  • For more information, see the official AWS Documentation on GetRecords, SubscribeToShard, and ChildShard.

• KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. PeriodicShardSyncManager is the new component that is responsible for running periodic lease/shard scans.

  • New configuration options are available to configure PeriodicShardSyncManager in LeaseManagementConfig

  Name	Default	Description
  leasesRecoveryAuditorExecutionFrequencyMillis	120000 (2 minutes)	Frequency (in millis) of the auditor job to scan for partial leases in the lease table. If the auditor detects any hole in the leases for a stream, then it would trigger shard sync based on leasesRecoveryAuditorInconsistencyConfidenceThreshold.
  leasesRecoveryAuditorInconsistencyConfidenceThreshold	3	Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync

  • New CloudWatch metrics are also now emitted to monitor the health of PeriodicShardSyncManager:

  Name	Description
  NumStreamsWithPartialLeases	Number of streams that had holes in their hash ranges.
  NumStreamsToSync	Number of streams which underwent a full shard sync.

• Introducing deferred lease cleanup. Leases will be deleted asynchronously by LeaseCleanupManager upon reaching SHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.

  • New configuration options are available to configure LeaseCleanupManager.

  Name	Default	Description
  leaseCleanupIntervalMillis	1 minute	Interval at which to run lease cleanup thread.
  completedLeaseCleanupIntervalMillis	5 minutes	Interval at which to check if a lease is completed or not.
  garbageLeaseCleanupIntervalMillis	30 minutes	Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not.

• Introducing experimental support for multistreaming, allowing a single KCL application to multiplex processing multiple streams.

  • New configuration options are available to enable multistreaming in RetrievalConfig#appStreamTracker.

• Fixing a bug in PrefetchRecordsPublisher restarting while it was already running.

• Including an optimization to HierarchicalShardSyncer to only create leases for one layer of shards.

• Adding support to prepare and commit lease checkpoints with arbitrary bytes.

  • This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size (currently 400 KB as of release), and can be used for recovery by passing a serialized byte buffer to RecordProcessorCheckpointer#prepareCheckpoint and RecordProcessorCheckpointer#checkpoint.

• Upgrading version of AWS SDK to 2.14.0.

• #725 Allowing KCL to consider lease tables in UPDATING healthy.

Release 2.2.11 of the Amazon Kinesis Client Library for Java

Release 2.2.11 (May 28, 2020)

Milestone#51

• Adjusting HTTP2 initial window size to 512 KB
  • PR#706
• Updating protobuf-java to version 3.11.4
  • PR#718
• Updating the AWS Java SDK to version 2.13.25
  • PR#722

Release 2.2.10 of the Amazon Kinesis Client Library for Java

Release 2.2.10 (March 26, 2020)

Milestone#48

• Fixing a bug in DynamoDB billing mode support for special regions.
  • PR#703
• Adding request id logging to ShardConsumerSubscriber.
  • PR#705

Release 1.13.3 of the Amazon Kinesis Client Library for Java

Release 1.13.3 (March 2, 2020)

[Milestone#49](https://github.com/awslabs/amazon-kinesis-client/milestone/49）

• Refactoring shard closure verification performed by ShutdownTask.
  • PR #684
• Fixing the bug in ShardSyncTaskManager to resolve the issue of new shards not being processed after resharding.
  • PR #694

Release 2.2.9 of the Amazon Kinesis Client Library for Java

Release 2.2.9 (Febuary 17, 2020)

Milestone#47

• Updating the AWS SDK version to 2.10.66.
  • PR#687
• Adding request id logging to the SubscribeToShard response.
  • PR#678

Release 2.2.8 of the Amazon Kinesis Client Library for Java

Release 2.2.8 (January 28, 2020)

Milestone#46

• Updating the AWS SDK version to 2.10.56.
  • PR#679
• Making ShardConsumerTest resilient to race conditions.
  • PR#668
• Updating integration test naming.
  • PR#667

Release 1.13.2 of the Amazon Kinesis Client Library for Java

Latest Release (1.13.2 Janurary 13, 2020)

• Adding backward compatible constructors that use the default DDB Billing Mode (#673)
  • PR #673

Release 1.13.1 of the Amazon Kinesis Client Library for Java

Release 1.13.1 (December 31, 2019)

Milestone#44

• Adding BillingMode Support to KCL 1.x. This enables the customer to specify if they want provisioned capacity for DDB, or pay per request.
  • PR #656
• Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending ShardEnd events.
  • PR #659
• Fix the LeaseManagementIntegrationTest failure.
  • PR #670