Skip to content

BITMAKER-1932: API control max number of simultaneous jobs #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mgonnav wants to merge 9 commits into
base: main
Choose a base branch
from
Open

BITMAKER-1932: API control max number of simultaneous jobs #63

mgonnav wants to merge 9 commits into from

Conversation

@mgonnav
Copy link
Contributor

@mgonnav mgonnav commented Aug 17, 2022

Description

Control from the API how many jobs are running simultaneously and scale this number according to the current scale of our job manager engine (for Kubernetes, the number of available nodes).

Issue

Checklist before requesting a review

  • I have performed a self-review of my code.
  • My code follows the style guidelines of this project.
  • I have made corresponding changes to the documentation.
  • New and existing tests pass locally with my changes.
  • If this change is a core feature, I have added thorough tests.
  • If this change affects or depends on the behavior of other estela repositories, I have created pull requests with the relevant changes in the affected repositories. Please, refer to our official documentation.
  • I understand that my pull request may be closed if it becomes obvious or I did not perform all of the steps above.

mgonnav and others added 9 commits October 14, 2022 15:34
@mgonnav @Adriana618
BITMAKER-1732: Processing and Storage Cost Associated to a Project (#30)
3916028 
* Add lifespan & total_response_bytes fields to SpiderJob.
* Add UsageRecord model.
* Add endpoint to get UsageRecords in a date range and current project usage.
@mgonnav @Adriana618
BITMAKER-1809: Usage module improvements (#37)
94a0738 
* Use DatabaseInterface
* Create usage records per project based on events (completed job, delete data)
* Do not delete the project but use a flag deleted to keep usage records
* Add item_count and request_count to SpiderJob
@Adriana618
Add logs data size calculation
24367bd
@Adriana618
Add migrations file
8e4f90b
@Adriana618
Formatted code
33ea53b
@Adriana618
Update Swagger API docs
0104520
@Adriana618
Update migrations file
ec514d0
@Adriana618
Add functions to get database and collection sizes. Refactor code
53d78aa
@Adriana618
API only allow MAX_CONCURRENT_JOB*NODE_COUNT concurrent jobs
a372c2e
@mgonnav
Copy link
Contributor Author

mgonnav commented Oct 24, 2024

@joaquingx This is an extra feature to control the number of jobs to launch in Kubernetes based on the current cluster size. The idea is that if we can keep the number of jobs close to the limit, it will help Kubernetes autoscale better when it detects this situation. This PR needs further testing since the method get_scale_size() may not work on all platforms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mgonnav