feat: reinit="create_new" setting for multiple simultaneous runs in one process

[timoffex]

• WB-23108
• Docs: Document new reinit='allow' setting docs#1145

Adds a create_new option to the reinit setting which causes wandb.init() to create a new run even if other runs exist. This should eventually become the default, and the other options ("return_previous" and "finish_previous") should be deprecated.

Although it's possible to pass reinit as a wandb.init() argument, the best way to use this is with wandb.setup().

Here is an example that uses the create_new setting to (serially) run multiple sub-experiments while logging their results to another run:

wandb.setup(wandb.Settings(reinit="create_new"))

with wandb.init() as experiment_results_run:
    for ...:
        with wandb.init() as run:
            # The do_experiment() function logs fine-grained metrics
            # to the given run and then returns result metrics that
            # we'd like to track separately.
            experiment_results = do_experiment(run)

            # After each experiment, we log its results to a parent
            # run. Each point in the parent run's charts corresponds
            # to one experiment's results.
            experiment_results_run.log(experiment_results)

---

This PR also removes util._is_offline() and replaces it with a heuristic that uses the newly added most_recent_active_run. This way, _is_offline continues to work as expected in most cases.

[timoffex]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: reinit="create_new" setting for multiple simultaneous runs in one process #9562 👈 (View in Graphite)
• refactor: avoid global run in wandb ipython magic #9710
• refactor: remove the workspace option to wandb ipython magic #9709
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

Attention: Patch coverage is 93.51852% with 7 lines in your changes missing coverage. Please review.

Project coverage is 79.71%. Comparing base (a4587d3) to head (867ad2f).

Files with missing lines	Patch %	Lines
wandb/beta/workflows.py	20.00%	4 Missing ⚠️
wandb/jupyter.py	33.33%	2 Missing ⚠️
wandb/sdk/data_types/base_types/wb_value.py	92.30%	1 Missing ⚠️

Additional details and impacted files

@@                      Coverage Diff                      @@
##           04-10-wandb_magic_cleanup    #9562      +/-   ##
=============================================================
- Coverage                      80.40%   79.71%   -0.70%     
=============================================================
  Files                            781      782       +1     
  Lines                          81118    81172      +54     
=============================================================
- Hits                           65225    64707     -518     
- Misses                         15164    15738     +574     
+ Partials                         729      727       -2     

Flag	Coverage Δ
func	46.11% <52.94%> (+0.01%)	⬆️
system	64.78% <93.20%> (-0.04%)	⬇️
unit	63.81% <28.76%> (-1.86%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tests/conftest.py	98.35% <100.00%> (-0.41%)	⬇️
tests/system_tests/test_core/test_wandb_init.py	100.00% <ø> (ø)
...s/system_tests/test_core/test_wandb_init_reinit.py	100.00% <100.00%> (ø)
tests/system_tests/test_core/test_wandb_setup.py	100.00% <100.00%> (ø)
wandb/sdk/artifacts/artifact.py	88.28% <100.00%> (+0.03%)	⬆️
...db/sdk/artifacts/storage_handlers/azure_handler.py	67.05% <ø> (ø)
wandb/sdk/data_types/table.py	89.90% <100.00%> (ø)
wandb/sdk/data_types/trace_tree.py	80.89% <ø> (ø)
wandb/sdk/internal/profiler.py	75.00% <ø> (ø)
wandb/sdk/wandb_init.py	84.49% <100.00%> (+0.16%)	⬆️
... and 7 more

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kptkin]

i think this behavior of global run is going to cause some confusion with users, as it will require to carefully track when they create and finish each run, maybe a better solution is to disallow global run when we have multiple active runs?

[kptkin]

or have multiple global runs? and they need to use id or something, not sure if this approach is better, but an alternative to fully disabling global run

[timoffex]

This behavior is for integrations; end users should avoid using wandb.run.

An alternative I considered is to make accesses to wandb.run raise an error after reinit="allow" is used. But we check wandb.run often in our own code (for example, to get its settings), so without fixing all those instances, it would break the SDK.

So wandb.run needs to either be None or a run. If starting a reinit="allow" run were to change it to None or the new run, then integrations relying on wandb.run that were initialized before the run would break or misbehave.

The rule in this PR is that wandb.run can only change when you call wandb.init() and refers to the same run until that run finishes.

[kptkin]

hmmm i see... i guess i was thinking of integrations in third party repo, where i don't think they change these settings, but i guess there is a case for integrations in our code base.

i guess it's fine, we probably should document it and discourage certain usage patterns especially with integrations?

[timoffex]

We don't mention using wandb.run in our guide for integrations, and I recently updated those docs and some other top-level docs to not use the global run.

[timoffex]

I updated reinit="create_new" to not touch wandb.run as discussed!