Skip to content

Support running pipelines in scheduled task queue #1871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 51 commits into from
May 28, 2025
Merged

Support running pipelines in scheduled task queue #1871

merged 51 commits into from
May 28, 2025

Conversation

keshav-space
Copy link
Member

@keshav-space keshav-space commented May 6, 2025 •
edited
Loading

End-to-end support for running pipelines and legacy importer/improver periodically in task queue.

  • PipelineSchedule model to track and manage periodic runs of pipelines
  • PipelineRun model to store state of pipeline execution
  • Scheduler and worker to automatically schedule and run enabled pipelines at regular intervals
  • API endpoint /api/v2/schedule/ to retrieve, update and create pipeline schedules
  • UI to browse pipeline schedules and view the detailed execution state of a pipeline
  • Populate tag and commit in .VERSION on pulling archive
  • Add captcha challenge to admin login page

Resolves #1783

keshav-space added 14 commits May 5, 2025 23:54
@keshav-space
Add PipelineSchedule and PipelineRun models
380cd6f 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add pipeline execution task
0b7f47c 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add scheduler for pipeline tasks
6e48ddc 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add management command to init schedule
21d9d5a 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Configure RQ settings
7bc98b3 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add utility to get latest commit hash
efd8d2c 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Update logs in PipelineRun instance
656ef3c 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add API endpoint for Pipeline schedule
7a79948 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add docker service for pipeline schedule
202db9d 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add list view for pipeline schedules
dd8b252 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add css and js for log highlighting
baae4a8 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add pipeline run list view
ba3879f 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add detail view for pipeline run
e8b1cda 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Show default values for empty fields
d616ce6 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Enable datetime localization for client
0ba0549 
- Detect and store client timezone in cookies
- Add UserTimezoneMiddleware to activate localization based on cookies

Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space keshav-space changed the title Support runing pipelines in scheduled task queue Support running pipelines in scheduled task queue May 7, 2025
pombredanne
pombredanne reviewed May 8, 2025
View reviewed changes
keshav-space added 12 commits May 9, 2025 23:03
@keshav-space
Allow temporary copy of .git to extract commit hash
34dcd72 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Populate tag and commit on pulling git archive
1936d9f 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Extract commit hash from git archive and local docker deployment
354c605 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use uuid to track pipeline job id
ae1a260 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use scheduler to explicitly queue pipeline execution jobs
a96f775 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Handle the stats for queued pipeline
279bd06 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add pipeline_url property to construct pipeline URL
472edc2 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Show execution_time for running jobs
0df4d48 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Do not humanize execution time in api response
e7c8cba 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Return only the latest pipeline run in API
54a1911 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Truncate log to 5000 characters in API response
deda888 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Restrict modifications to admin users
edb6b61 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Pass redis hostname to docker image
5c4e7c1 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Defer unused fields to optimize pipeline queries
c1d6375 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Render stopped and stale pipeline statuses
2bfbc46 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Dequeue job awaiting execution when stop job is requested
84efc89 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Skip linkcheck for unresponsive URL
7879fb5 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Update failing view tests
fd28465 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Use fa arrow icon for back buttons
2ffbb24 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add tests for schedule and run model
a34f485 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Ensure run fields are reset before job requeue
9bdeba6 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
TG1999
TG1999 reviewed May 21, 2025
View reviewed changes
@keshav-space
Add method to reset and requeue pipeline
263f503 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Track execution timeout in schedule model
6130597 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Show timezone in logs and stats
fefe611 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add clientside pagination for bigger log snippets
587328e 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Handle boundary condition in snippet navigation
312674f 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add setting to toggle live logging for pipelines
c67cf66 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add copy button for pipeline and job ids
217a466 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Restrict modifications to staff users authed via session
54ec053 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add tests for /api/v2/schedule endpoint
9d8d22e 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Display pipeline description in UI
5f2eb94 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Reset throttling to properly test api rate limits
0db4727 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Add captcha challenge to staff login page
3c7def2 
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
TG1999
TG1999 approved these changes May 28, 2025
View reviewed changes
Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@keshav-space keshav-space merged commit e316059 into main May 28, 2025
10 checks passed
@keshav-space keshav-space mentioned this pull request Jun 16, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TG1999 TG1999 TG1999 approved these changes

@pombredanne pombredanne Awaiting requested review from pombredanne

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Run pipelines parallelly in Task Queue
3 participants
@keshav-space @pombredanne @TG1999