Feature: user defined pidprovider

[AMekss]

Similarly as in #430 we are hitting problem when monitoring RQ workers. By design RQ worker forks new process for every new job, while it always execute one job at a time. The churn of process pids is insane and our worker pods are running out of memory in a few days of running because of this.

Would be cool to have a possibility to override PID provider for such edge cases. This PR is an attempt to do so in less obtrusive way I managed to find. I'm not seasoned Python developer nor very familiar with all the ins and outs of this project, so feel free to give me hints on how proposed solution can be improved.

This PR makes possible the following:

import uwsgi
from prometheus_client.pidprovider import Pidprovider
Pidprovider.source = uwsgi.worker_id

# or in case of RQ
Pidprovider.source = os.getppid

[AMekss]

Looking to fix broken build..

[brian-brazil]

I don't see how this works, any code like this would need to be called before anything that might instantiate a metric is.

[AMekss]

Correct, I think the best place for it would be before other imports while bootstrapping workers. While I also think that it should work alright even if changed later as it will be handled as PID change. Please correct me if I'm wrong

[brian-brazil]

That'd still have files for all the original pids though, so it'd make the number of files higher

…

On Thu 18 Jul 2019, 20:05 Arturs Mekss, ***@***.***> wrote: Correct, I think the best place for it would be before other imports while bootstrapping workers. While I also think that it should work alright even if changed later as it will be handled as PID change. Please correct me if I'm wrong — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#441?email_source=notifications&email_token=ABWJG5SVM2NDJKZGVBZUX6LQACWIPA5CNFSM4IEZ6OD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2JJ5DY#issuecomment-512925327>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWJG5TSFHQGR7ZXSCS4PCTQACWIPANCNFSM4IEZ6ODQ> .

[AMekss]

Sure, the correct way of setting this up is before importing any application logic and metrication code. I'm trying to figure out the worst case scenario.. where even if that would mistakenly happen afterwards it would not mess the metrics and be handled by pid change logic.

[AMekss]

After playing around with python I figured out a way of how to achieve the same within a current implementation, leaving it here for anyone else tackling the same issue. Before importing any application code this snippet can customise PID provider function:

import os
if os.getenv('prometheus_multiproc_dir'):
    from prometheus_client import values
    values.ValueClass = values.MultiProcessValue(os.getppid)

It works for us while it feels a bit of a hack to me so would be cool have some official and better way of dealing with this edge case. Anyways closing this PR for now as it doesn't bring any value.

Update After almost a week of running it I can confirm it fixed our memory problems. I slightly updated code snippet

[cai-personal]

After playing around with python I figured out a way of how to achieve the same within a current implementation, leaving it here for anyone else tackling the same issue. Before importing any application code this snippet can customise PID provider function:

import os
from prometheus_client import values
values.ValueClass = values.MultiProcessValue(os.getppid)

It works for us while it feels a bit of a hack to me so would be cool have some official and better way of dealing with this edge case. Anyways closing this PR for now as it doesn't bring any value.

can we aggregate the metrics of non-current pid files into a specific file(count_0.db/histogram_0.db), so that the total number of pid files can be controlled without increasing.

[cai-personal]

After playing around with python I figured out a way of how to achieve the same within a current implementation, leaving it here for anyone else tackling the same issue. Before importing any application code this snippet can customise PID provider function:

import os
from prometheus_client import values
values.ValueClass = values.MultiProcessValue(os.getppid)

It works for us while it feels a bit of a hack to me so would be cool have some official and better way of dealing with this edge case. Anyways closing this PR for now as it doesn't bring any value.

The purpose of this is to control the number of pid files and not let it continue to grow. So, I think it's a viable way to aggregate all the db files from a period of time ago into a specific db file.

[AMekss]

can we aggregate the metrics of non-current pid files into a specific file(count_0.db/histogram_0.db), so that the total number of pid files can be controlled without increasing.

This could be the way, however I have no idea how efficient would be to merge files once pid got changed