Add Prometheus metrics instrumentation #38
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Foundational work to enable all loader tests to inherit common test patterns
Adds generalized streaming test infrastructure and migrates Redis and Snowflake loader tests to use the shared base classes.
Migrates the final three loader test suites to use the shared base test infrastructure
Iceberg loader was using an outdated reorg deletion method that used a _meta_block_ranges column instead of using the modern state_store + _amp_batch_id approach. Changes: 1. _handle_reorg now uses state_store.invalidate_from_block() to get affected batch IDs, matching PostgreSQL/Snowflake/DeltaLake approach 2. _perform_reorg_deletion now filters rows by _amp_batch_id instead of trying to parse non-existent _meta_block_ranges JSON column 3. Efficient filtering using set membership checks on batch IDs
Changed test_append_mode to append data with different IDs (6-10) instead of reusing the same IDs (1-5) to avoid duplicate key conflicts in key-value stores like LMDB and Redis.
- Redis stream storage: Corrected test to use f'{table_name}:stream' key
format to match how Redis loader stores stream data
- LMDB overwrite mode: Fixed _clear_data() to properly delete named
databases when overwriting data
- LMDB streaming: Added tx_hash column to test data for compatibility
with key pattern requirements
- Base streaming tests: Updated column references from transaction_hash
to tx_hash for consistency across all loaders
**Iceberg Loader:** - Added snapshot_id to metadata for test compatibility - Modified base loader to pass table_name in kwargs to metadata methods - Skipped partition_spec test (requires PartitionSpec object implementation) **PostgreSQL Loader:** - Fixed _clear_table() to check table existence before TRUNCATE - Prevents "relation does not exist" errors in overwrite mode **DeltaLake Loader:** - Added partition_by property for convenient access - Added delta_version and files_added metadata aliases - Fixed test fixture to use unique table paths per test - Prevents data accumulation across tests **Test Infrastructure:** - Updated delta_basic_config fixture to generate unique paths per test - Prevents cross-test contamination in DeltaLake tests
Key fixes: - Changed loader.conn to loader.connection (Snowflake uses different attribute name) - Set supports_overwrite = False (Snowflake doesn't support OVERWRITE mode) - Set requires_existing_table = False (Snowflake auto-creates tables) - Added cleanup_tables fixture for Snowflake-specific test cleanup
fordN
force-pushed
the
ford/metrics-instrumentation
branch
from
60b9016 to
67c07a0
Compare
- use new dependency `prometheus-client` -
fordN
force-pushed
the
ford/metrics-instrumentation
branch
from
67c07a0 to
5ee72d0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for prometheus metrics and creates a few initial metrics.
Summary
AmpMetricssingleton with Prometheus metrics for loadersDepends on PR #32 (test generalization)
Resolves #7