Skip to content
This repository was archived by the owner on Sep 23, 2024. It is now read-only.

AP-1424: limit config to mitigate long running incremental queries #211

Merged
merged 6 commits into from
Mar 30, 2023

Conversation

Samira-El
Copy link
Contributor

@Samira-El Samira-El commented Mar 29, 2023

Problem

When running incremental sync on busy tables, the query will run for hours or days which causes issues on the database side, as it interferes with things like vacuuming.

Proposed changes

To avoid making any sweeping changes on the incremental logic, I suggest introducing an optional limit config which would be added to every single incremental query to limit how many rows the query returns thus limit its runtime too.

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • Description above provides context of the change
  • I have added tests that prove my fix is effective or that my feature works
  • Unit tests for changes (not needed for documentation changes)
  • CI checks pass with my changes
  • Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
  • Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions

@Samira-El Samira-El changed the title AP-1424 Mitigate long running incremental queries AP-1424: limit config to mitigate long running incremental queries Mar 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants