Skip to content

Prevent empty NSQ batches and bound HTTP waits - #3744

Draft
arthurianresolve wants to merge 4 commits into
NationalSecurityAgency:integrationfrom
arthurianresolve:bugfix/NSQMetricsReporter
Draft

Prevent empty NSQ batches and bound HTTP waits#3744
arthurianresolve wants to merge 4 commits into
NationalSecurityAgency:integrationfrom
arthurianresolve:bugfix/NSQMetricsReporter

Conversation

@arthurianresolve

@arthurianresolve arthurianresolve commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Improve NSQMetricsReporter reliability and make its HTTP timeout configurable through each supported setup path:

  • do not post empty metric batches
  • set connection-pool, connection, and response-inactivity timeouts
  • allow direct Java callers to override the timeout through NSQMetricsReporterFactory.Builder
  • bind metrics.reporter.timeout for Spring services
  • bind dw.metrics.reporter.timeout.millis for CDI services
  • fail clearly when an explicit timeout is used with a reporter that does not support it

Root cause

flush() sent a request even when its buffer was empty. The Apache HTTP client also had no connection-request, connection, or socket timeout, so a slow NSQ endpoint could delay later reporting cycles. Service configuration paths could not pass a timeout to the NSQ builder.

Impact

NSQ no longer receives empty /mpub requests. HTTP waits use a 10-second default limit, and a timed-out request does not prevent a later batch from being attempted. Java, Spring, and CDI deployments can override the shared limit.

Configuration

The default timeout is 10 seconds for the connection-pool wait, connection attempt, and response wait.

  • Direct Java: call withTimeout(...) on the NSQ builder.
  • Spring: set metrics.reporter.timeout with Spring duration syntax, such as 10s, 500ms, or a bare number of seconds.
  • CDI: set dw.metrics.reporter.timeout.millis to a positive millisecond value. The default -1 leaves the reporter's own default unchanged.

When the property is omitted, existing behavior uses the reporter's 10-second default. Setting it for an unsupported reporter fails with a clear configuration error. The original five-argument CDI producer overload remains available for source and binary compatibility.

Release sequencing

The new builder capability is in metrics-reporter:3.0.1-SNAPSHOT, and the root and starter reactors point to that version so this draft can build the producer and consumers together.

Before this PR is ready to merge, publish immutable metrics-reporter:3.0.1 and replace both snapshot version pins with 3.0.1. Clean reactors that do not include the opt-in metrics-reporter module must not depend on the snapshot.

Integration

This PR overlaps #3743 in NSQMetricsReporter. Whichever PR merges second should rebase and retain both #3743's UTF-8 byte sizing/writes and this PR's empty-batch and timeout changes, then rerun the metrics-reporter tests.

Test

Windows host: Java 11.0.31 and Maven 3.9.16.

Passed:

  • Full metrics-reporter module: 5 tests, 0 failures, 0 errors, 0 skipped.
  • Full spring-boot-starter-datawave module: 43 tests, 0 failures, 0 errors, 0 skipped.
  • Full datawave-ws-common module: 68 tests, 0 failures, 0 errors, 0 skipped.
  • CDI compatibility reactor: all 22 required modules built successfully; MetricsCdiConfigurationTest passed.

Coverage includes:

  • empty direct flushes
  • oversized and rollover batches
  • all three Apache HTTP timeout values
  • a local server that responds after the configured limit
  • a later successful batch after a timeout
  • Spring binding for omitted, millisecond-suffixed, and bare-second values
  • capable, unsupported, omitted, and invalid timeout configuration branches
  • CDI millisecond configuration and preservation of the existing producer overload

Fixes #3732
Fixes #3753

@arthurianresolve arthurianresolve changed the title Prevent empty NSQ metric batches Prevent empty NSQ batches and bound HTTP waits Jul 24, 2026
@arthurianresolve
arthurianresolve marked this pull request as draft July 24, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug (NSQMetricsReporter) - HTTP requests have no timeout Bug (NSQMetricsReporter) - sends empty batches to NSQ

1 participant