Auto delete job history logs after specific period

[venkateshkelevo]

Topic

Auto delete job history logs after specific period

Relevant information

We have used helm charts(Kubernetes environment) to deploy airbyte platform. Used the flag TEMPORAL_HISTORY_RETENTION_IN_DAYS="7" to auto delete the job history logs after 7 days, but the cleanup is not happening and logs are still available. Can you provide a way how can we achieve this?

helm command:
helm install -n foresight-infra airbyte airbyte/airbyte --set global.env_vars.TEMPORAL_HISTORY_RETENTION_IN_DAYS="7"

Thanks.

[marcosmarxm]

@venkateshkelevo this flag will clean temporal database logs. Is there any other logs you want to clean besides those?

[venkateshkelevo]

Our basic problem is airbyte-db-0 pod pvc, used space is getting increased slowly. I assume the job history logs and metadata getting stored in db because of that used space is getting increased, we would like to cleanup the db space automatically.

We configured a connector to pull the data from rest source using cron job which runs every two minutes, attached screenshot for logs which I am referring to.

Please suggest a way to auto clean up the db pvc space.

Thanks.

[gingeard]

We have a similar situation:
after a week of data processing with only one connection set up, Airbyte's Postgres DB consumed 1GB already:

1. db-airbyte db:

2. temporal db:

If I understand correctly, setting the TEMPORAL_HISTORY_RETENTION_IN_DAYS parameter can help clean up the temporal db, but not the db-airbyte one.

[iulianapostol-ozone]

Would someone suggest me a temporary solution for this ? I know in case of minio space issue we can kubectl into minio (in case of k8s) and delete manually the log files. But what about the psql DB? Can someone suggest me a way to clean up the jobs database ?