You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Just a warning to other people, we just had sudden timeouts on some of our requests, turns out that having run silk for a couple of months gives you a lot of requests, so now the garbage collection had to catch up, the result was that the list of id's became SO big that the whole process crashed and the cleanup never happened, so every ~10% of requests would try again.
Also, maybe the garbage collection needs to use the start_time to cleanup old requests instead of a list of id's:
SELECT start_time FROM silk_requests ORDER BY start_time ASC LIMIT 1 OFFSET {prune_count}
and then just do a DELETE FROM silk_requests WHERE start_time <= {start_time}
Because using the Request.objects.filter(id__in=...).delete() does a select first, which is highly inefficient. (Also as mysql doesn't support limit's in subqueries so that's not an option here)
The text was updated successfully, but these errors were encountered:
Just a warning to other people, we just had sudden timeouts on some of our requests, turns out that having run silk for a couple of months gives you a lot of requests, so now the garbage collection had to catch up, the result was that the list of id's became SO big that the whole process crashed and the cleanup never happened, so every ~10% of requests would try again.
Also, maybe the garbage collection needs to use the start_time to cleanup old requests instead of a list of id's:
SELECT start_time FROM silk_requests ORDER BY start_time ASC LIMIT 1 OFFSET {prune_count}
and then just do a
DELETE FROM silk_requests WHERE start_time <= {start_time}
Because using the Request.objects.filter(id__in=...).delete() does a select first, which is highly inefficient. (Also as mysql doesn't support limit's in subqueries so that's not an option here)
The text was updated successfully, but these errors were encountered: