Conversation
yruslan
left a comment
yruslan
left a comment
There was a problem hiding this comment.
Looks good. Just one thing to check when we start using it. It should be allowed to send a run event with the empty jobs array to the run topic... i think.
conf/topics.json
Outdated
| "status" | ||
| ], | ||
| "type": "object" | ||
| "required": ["event_id", "job_ref", "tenant_id", "source_app", "source_app_version", "environment", "timestamp_start", "timestamp_end", "jobs", "jobs.catalog_id", "jobs.status", " jobs.timestamp_start", "jobs.timestamp_end"] |
There was a problem hiding this comment.
Not 100% sure how it works when jobs is a mandatory field. It is an array, and it is possible to have an empty array when the pipeline has failed and was not able to run any jobs. For instance, when EMR is not available 😆
There was a problem hiding this comment.
actually.... we need to remove
jobs.* required fields... as jsonschema validate complains as that prescribes them to be present in underlying object, not in the element of a array
those are
jobs.catalog_id
jobs.status
jobs.timestamp_start
jobs.timestamp_end
as none of those would be present at that xpath
as for jobs..... schema just requires field to be there and to be array.... so empty array is fine.... null is not fine though
Co-authored-by: Ruslan Yushchenko <yruslan@gmail.com>
2 participants
Updates schema to ADR 009 as of 2024-10-17
closes #22