Bug 1672448 Persist Glean retention_days setting to generated schema #166
Conversation
This translates the `retention_days` setting from Glean repositories.yaml to `mozPipelineMetadata.expiration_policy.delete_after_days` in all generated JSON schemas for the application. We will need a further step in the ops logic to act on this schema-level metadata, setting the appropriate table-level retention policy in BQ.
|
See diff of generated-schemas which demonstrates that only pings for |
acmiyaguchi
left a comment
acmiyaguchi
left a comment
There was a problem hiding this comment.
I'm ambivalent about breaking the API, but I don't think there are use-cases of the package being used outside of bigquery-etl. It does look like the right way to do things though.
@robhudson you don't happen to still be using the msg package?
| "glean_client_info", | ||
| } | ||
|
|
||
| def __init__(self, repo, app_id, **kwargs): # TODO: Make env-url optional |
There was a problem hiding this comment.
Can you bump the major version on the package?
There's a utility function in the bigquery-etl glam package that assumes the repo argument is a string, and that the output format of the get_repos is a list of tuples.
There was a problem hiding this comment.
Bumped from 0.2.0 to 0.3.0
| @pytest.fixture | ||
| def glean(): | ||
| return glean_ping.GleanPing("glean", "org-mozilla-glean") | ||
| return glean_ping.GleanPing({"name": "glean", "app_id": "org-mozilla-glean"}) |
There was a problem hiding this comment.
There should be a test that includes the retention_days field from the repo metadata.
…166) This translates the `retention_days` setting from Glean repositories.yaml to `mozPipelineMetadata.expiration_policy.delete_after_days` in all generated JSON schemas for the application. We will need a further step in the ops logic to act on this schema-level metadata, setting the appropriate table-level retention policy in BQ.
This translates the
retention_dayssetting from Glean repositories.yamlto
mozPipelineMetadata.expiration_policy.delete_after_daysin all generatedJSON schemas for the application.
We will need a further step in the ops logic to act on this schema-level
metadata, setting the appropriate table-level retention policy in BQ.