Description
We are using v0.3.2 via Docker and noticed that Docker's own CPU limitation is having a hard time constraining the crawler.
Our worker used to be on dedicated HW (ran via docker) and that was fine but for external reasons, we installed another service on the same hardware (still via docker).
What we experienced then was that although we instructed Docker to set a quota on the container (say 0.5 CPU), the container was always oscillating (bursting?) to 100% which caused the other service to lack CPU resources and fail.
We had to set the cpusets on the container to explicitly use specific cores/cpu for the two services to coexist on the same HW.
Although I understand there's probably nothing to do about it, I wanted to share this information. Might not be present in newer version, might be a docker issue, or a kernel issue, or just not an issue but a limitation of how cgroups works.
Might be interesting to keep it in mind when offering k8s manifests…
Maybe #103 will help.