Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ensure only one instance of codecarbon is run per machine #562

Merged
merged 5 commits into from
Sep 8, 2024

Conversation

inimaz
Copy link
Contributor

@inimaz inimaz commented May 28, 2024

Imagine I open 2 clis and run codecarbon monitor in each of them. This will result on both of them outputing very similar data to the same output. Now imagine I have multiple instances of codecarbon (e.g. a python script, some cli, some script somewhere that I forgot it is running...). This could become complex rapidly.

Goal of this PR

By default it will allow only one instance per machine to measure at the same time.
This is regulated via the parameter allow_multiple_runs.

  • If true --> multiple runs of codecarbon can be done in the same machine.
  • If false --> only one at the same time.

Default is false.

How to test this

Option 1: using 2 consoles

Open 2 consoles and run hatch run python examples/logging_to_file.py on both of them.

  • Output of Console 1: Normal behavior
  • Output of Console 2:
[codecarbon ERROR @ 18:17:11] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.

Option 2: multiprocess

I have added an example of exclusive run using multiprocess.
Run it via hatch run python examples/logging_to_file_exclusive_run.py
Example of output:

~/codecarbon$ hatch run python examples/logging_to_file_exclusive_run.py 
[codecarbon INFO @ 09:43:43] offline tracker init
[codecarbon INFO @ 09:43:43] offline tracker init
[codecarbon INFO @ 09:43:43] offline tracker init
[codecarbon INFO @ 09:43:43] offline tracker init
[codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.
[codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.
[codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.
[codecarbon INFO @ 09:43:43] offline tracker init
[codecarbon ERROR @ 09:43:43] Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.
[codecarbon INFO @ 09:43:43] [setup] RAM Tracking...
[codecarbon INFO @ 09:43:43] [setup] GPU Tracking...
...

We see that 5 instances of codecarbon are started and 4 end up in error and exit.

@inimaz inimaz added the enhancement New feature or request label May 28, 2024
@inimaz inimaz marked this pull request as ready for review May 29, 2024 18:10
codecarbon/lock.py Outdated Show resolved Hide resolved
@benoit-cty benoit-cty merged commit 4c1ec98 into master Sep 8, 2024
10 checks passed
@benoit-cty benoit-cty deleted the feat/lock branch September 8, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants