Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSA: Cleanup silk_requests before updating to 1.1.0 #261

Closed
hvdklauw opened this issue Jan 16, 2018 · 0 comments
Closed

PSA: Cleanup silk_requests before updating to 1.1.0 #261

hvdklauw opened this issue Jan 16, 2018 · 0 comments

Comments

@hvdklauw
Copy link
Contributor

Just a warning to other people, we just had sudden timeouts on some of our requests, turns out that having run silk for a couple of months gives you a lot of requests, so now the garbage collection had to catch up, the result was that the list of id's became SO big that the whole process crashed and the cleanup never happened, so every ~10% of requests would try again.

Also, maybe the garbage collection needs to use the start_time to cleanup old requests instead of a list of id's:

SELECT start_time FROM silk_requests ORDER BY start_time ASC LIMIT 1 OFFSET {prune_count}
and then just do a
DELETE FROM silk_requests WHERE start_time <= {start_time}

Because using the Request.objects.filter(id__in=...).delete() does a select first, which is highly inefficient. (Also as mysql doesn't support limit's in subqueries so that's not an option here)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant