Skip to content

Add support for new test topic and improve error logging in writers#49

Merged
oto-macenauer-absa merged 2 commits intomasterfrom
feature/tests
Sep 5, 2025
Merged

Add support for new test topic and improve error logging in writers#49
oto-macenauer-absa merged 2 commits intomasterfrom
feature/tests

Conversation

@oto-macenauer-absa
Copy link
Collaborator

@oto-macenauer-absa oto-macenauer-absa commented Sep 5, 2025

This pull request introduces support for a new public.cps.za.test topic across the configuration, schema, and writers, and adds a comprehensive configuration validation test suite. The changes ensure that the new topic is handled end-to-end, while also improving error logging in writer modules and enforcing config/schema correctness with new tests.

Support for new topic public.cps.za.test:

  • Added public.cps.za.test topic with associated user access in conf/access.json.
  • Introduced a new schema for the test topic in conf/topic_test.json.
  • Updated src/event_gate_lambda.py to load the new topic schema and switched config loading to config.dev.json.

Writer enhancements for the new topic:

  • Added postgres_test_write function in src/writer_postgres.py to handle inserts for the new topic, and updated the main write logic to support it [1] [2].

Improved error logging:

  • Enhanced error messages in src/writer_eventbridge.py, src/writer_kafka.py, and src/writer_postgres.py to provide more context on unknown exceptions [1] [2] [3].

Configuration and schema validation tests:

  • Added tests/test_conf_validation.py to verify that config files contain required keys, access files have correct structure, and topic schemas are well-formed.

Release notes
Release notes:

  • Added unit test validation topic schemas
  • Added schema for test topic (will be used to e2e test)
  • Improved logging - added info witch writer failed

Related work
Closed #48

@oto-macenauer-absa oto-macenauer-absa merged commit c5d1c81 into master Sep 5, 2025
1 check passed
@oto-macenauer-absa oto-macenauer-absa deleted the feature/tests branch September 5, 2025 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add unit test validating JSON schemas

2 participants