Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aggregate expired pid db files, control the number of files and improve scrape effectiveness #443

Open
cai-personal opened this issue Jul 26, 2019 · 1 comment

Comments

@cai-personal
Copy link

cai-personal commented Jul 26, 2019

history related issues: #441
#430

Can we aggregate all the db files from a period of time ago and non-current pid into a total db file, to control the number of pid files ?

I have realized this idea with golang,here are some details:

Project deploy info:
gunicorn django
128 workers
gunicorn max_requests:10000(create a new pid file almost every minute)

  1. I can't solve the problem that the pid file has been growing, and it can reach 6,000 in four days;
  2. Try to delete the expired pid regularly in the code, but it will cause the figure to drop with grafana;
  3. The time to request metric is getting longer as the program runs.

Improve scrape efficiency:
I used golang to rewrite the logic of python aggregate metrics(generate metric still using python). After rewriting, each scrape time is less than 1 second.

Solve the growing pid files:
Aggregate all the db files from a period of time ago and non-current pid into a total db file. Then delete these files. When calculating metric, history total db + curent pid = current pid db. (i do it every hour)

Now, num of pid files is <200 in my project. if we can do change this, it would be a big strengthen. Just like prometheus will also aggregate historical data

@cai-personal
Copy link
Author

I just raised this idea to see if it is necessary. and sorry,i used Golang to rewrite aggregate metric, just because I was trying to improve the scrape time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant