Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUPPORT] Dynamic configuration updates cause metrics reporting to stop #12282

Open
mstahlswrx opened this issue Nov 18, 2024 · 0 comments
Open

Comments

@mstahlswrx
Copy link

Describe the problem you faced

When running the Hudi streamer with the https://hudi.apache.org/releases/release-0.14.0#dynamic-configuration-updates feature configured, the streamer restarted after a configuration change as expected; however, metrics reporting stopped.

I believe the sequence of events are as follows:

Hudi calls streamSync.close() which ends up shutting down metrics reporting. Since the metrics instances are all cached in a static field, when a new StreamSync instance is created, it creates a new HoodieMetrics instance which reuses the stopped Metrics instance.

#10132 is a fix for a similar issue, but not related to the streamer. (That patch is already in hudi 0.14.)

To Reproduce

Steps to reproduce the behavior:

  1. Create an implementation of the ConfigurationHotUpdateStrategy class
  2. Start the hudi streamer with the --config-hot-update-strategy-class option configure with your hot update strategy class
  3. Verify metrics reporting is working
  4. Change a configuration property so that your ConfigurationHotUpdateStrategy.updateProperties method returns updated properties
  5. Wait for the streamer to re-initialize with the updated properties
  6. Observe that metrics reporting no longer works

Expected behavior

After the streamer re-initializes metrics reporting continues to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant