Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing job_id type from Int to BigInt #40946

Open
2 tasks done
gyli opened this issue Jul 22, 2024 · 3 comments
Open
2 tasks done

Changing job_id type from Int to BigInt #40946

gyli opened this issue Jul 22, 2024 · 3 comments
Assignees
Labels
airflow3.0:candidate Potential candidates for Airflow 3.0 area:db-migrations PRs with DB migration kind:feature Feature Requests

Comments

@gyli
Copy link
Contributor

gyli commented Jul 22, 2024

Description

Changing job_id type from Int to BigInt to support more task executions.

Use case/motivation

Current type of job_id in taskinstance is int, which has a reachable limit 2147483647 for users who have many DAG runs or many tenants. Once it's reached, task won't start new execution due to error from db.
For better potential multi-tenancy support, I believe it would be much safer if the job_id type can be changed to BigInt.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@gyli gyli added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Jul 22, 2024
Copy link

boring-cyborg bot commented Jul 22, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@dosubot dosubot bot added the area:db-migrations PRs with DB migration label Jul 22, 2024
@vatsrahul1001 vatsrahul1001 removed the needs-triage label for new issues that we didn't triage yet label Jul 23, 2024
@uranusjr
Copy link
Member

uranusjr commented Aug 1, 2024

Since this is ultimately a pk in Job, even BigInt sounds reachable considering it’s ever incremental. A UUID or another solution may be more desirable.

@uranusjr uranusjr added the airflow3.0:candidate Potential candidates for Airflow 3.0 label Aug 1, 2024
@uranusjr
Copy link
Member

uranusjr commented Aug 1, 2024

Adding this as a possible candidate since it’s arguably a breaking change. Although we could justify this later on, it’d be easiest to bundle it into 3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
airflow3.0:candidate Potential candidates for Airflow 3.0 area:db-migrations PRs with DB migration kind:feature Feature Requests
Projects
None yet
Development

No branches or pull requests

3 participants