feat: ensure only one instance of codecarbon is run per machine #562
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Imagine I open 2 clis and run
codecarbon monitor
in each of them. This will result on both of them outputing very similar data to the same output. Now imagine I have multiple instances of codecarbon (e.g. a python script, some cli, some script somewhere that I forgot it is running...). This could become complex rapidly.Goal of this PR
By default it will allow only one instance per machine to measure at the same time.
This is regulated via the parameter
allow_multiple_runs
.Default is
false
.How to test this
Option 1: using 2 consoles
Open 2 consoles and run
hatch run python examples/logging_to_file.py
on both of them.Option 2: multiprocess
I have added an example of exclusive run using multiprocess.
Run it via
hatch run python examples/logging_to_file_exclusive_run.py
Example of output:
~/codecarbon$ hatch run python examples/logging_to_file_exclusive_run.py [codecarbon INFO @ 09:43:43] offline tracker init [codecarbon INFO @ 09:43:43] offline tracker init [codecarbon INFO @ 09:43:43] offline tracker init [codecarbon INFO @ 09:43:43] offline tracker init [codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting. [codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting. [codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting. [codecarbon INFO @ 09:43:43] offline tracker init [codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting. [codecarbon INFO @ 09:43:43] [setup] RAM Tracking... [codecarbon INFO @ 09:43:43] [setup] GPU Tracking... ...
We see that 5 instances of
codecarbon
are started and 4 end up in error and exit.