Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new "context" option to specify the multiprocessing context internally used by handlers #851

Closed
Delgan opened this issue Apr 22, 2023 · 0 comments · Fixed by #852
Closed
Labels
feature Request for adding a new feature

Comments

@Delgan
Copy link
Owner

Delgan commented Apr 22, 2023

A better fix should be implemented for #746, because changing the default context globally using set_start_method() is not always suitable.

Invalid use case example on Linux (will hang forever because child processes crash):

import multiprocessing
import sys

from loguru import logger


def set_logger(logger_):
    global logger
    logger = logger_


def log_test(num):
    print("before log")
    logger.info(f"hi {num}!")
    print("do we reach this?")
    logger.complete()


def main():
    logger.remove()
    logger.add(sys.stderr, enqueue=True)

    with multiprocessing.get_context("spawn").Pool(
        4, initializer=set_logger, initargs=(logger,)
    ) as pool:
        pool.map(log_test, list(range(10)))


if __name__ == "__main__":
    main()

Proposed solution using a new context attribute:

import multiprocessing
import sys

from loguru import logger


def set_logger(logger_):
    global logger
    logger = logger_


def log_test(num):
    print("before log")
    logger.info(f"hi {num}!")
    print("do we reach this?")
    logger.complete()


def main():
    context = multiprocessing.get_context("spawn")
    logger.remove()
    logger.add(sys.stderr, enqueue=True, context=context)

    with context.Pool(4, initializer=set_logger, initargs=(logger,)) as pool:
        pool.map(log_test, list(range(10)))


if __name__ == "__main__":
    main()
@Delgan Delgan added the feature Request for adding a new feature label Apr 22, 2023
jmctune referenced this issue in dqx-translation-project/dqxclarity Oct 9, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [loguru](https://togithub.com/Delgan/loguru)
([changelog](https://togithub.com/Delgan/loguru/blob/master/CHANGELOG.rst))
| `==0.7.0` -> `==0.7.2` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/loguru/0.7.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/loguru/0.7.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/loguru/0.7.0/0.7.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/loguru/0.7.0/0.7.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>Delgan/loguru (loguru)</summary>

###
[`v0.7.2`](https://togithub.com/Delgan/loguru/blob/HEAD/CHANGELOG.rst#072-2023-09-11)

[Compare
Source](https://togithub.com/Delgan/loguru/compare/0.7.1...0.7.2)

\=====================

- Add support for formatting of `ExceptionGroup` errors (`#&#8203;805
<https://github.com/Delgan/loguru/issues/805>`\_).
- Fix possible `RuntimeError` when using
`multiprocessing.set_start_method()` after importing the `logger`
(`#&#8203;974 <https://github.com/Delgan/loguru/issues/974>`\_)
- Fix formatting of possible `__notes__` attached to an `Exception`
(`#&#8203;980 <https://github.com/Delgan/loguru/issues/980>`\_).

###
[`v0.7.1`](https://togithub.com/Delgan/loguru/blob/HEAD/CHANGELOG.rst#071-2023-09-04)

[Compare
Source](https://togithub.com/Delgan/loguru/compare/0.7.0...0.7.1)

\=====================

- Add a new `context` optional argument to `logger.add()` specifying
`multiprocessing` context (like `"spawn"` or `"fork"`) to be used
internally instead of the default one (`#&#8203;851
<https://github.com/Delgan/loguru/issues/851>`\_).
- Add support for true colors on Windows using ANSI/VT console when
available (`#&#8203;934 <https://github.com/Delgan/loguru/issues/934>`*,
thanks `@tunaflsh <https://github.com/tunaflsh>`*).
- Fix possible deadlock when calling `logger.complete()` with concurrent
logging of an asynchronous sink (`#&#8203;906
<https://github.com/Delgan/loguru/issues/906>`\_).
- Fix file possibly rotating too early or too late when re-starting an
application around midnight (`#&#8203;894
<https://github.com/Delgan/loguru/issues/894>`\_).
- Fix inverted `"<hide>"` and `"<strike>"` color tags (`#&#8203;943
<https://github.com/Delgan/loguru/pull/943>`*, thanks `@tunaflsh
<https://github.com/tunaflsh>`*).
- Fix possible untraceable errors raised when logging non-unpicklable
`Exception` instances while using `enqueue=True` (`#&#8203;329
<https://github.com/Delgan/loguru/issues/329>`\_).
- Fix possible errors raised when logging non-picklable `Exception`
instances while using `enqueue=True` (`#&#8203;342
<https://github.com/Delgan/loguru/issues/342>`*, thanks `@ncoudene
<https://github.com/ncoudene>`*).
- Fix missing seconds and microseconds when formatting timezone offset
that requires such accuracy (`#&#8203;961
<https://github.com/Delgan/loguru/issues/961>`\_).
- Raise `ValueError` if an attempt to use nanosecond precision for time
formatting is detected (`#&#8203;855
<https://github.com/Delgan/loguru/issues/855>`\_).

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/dqx-translation-project/dqxclarity).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4wLjMiLCJ1cGRhdGVkSW5WZXIiOiIzNy44LjEiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Request for adding a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant