Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use PipeChannels or FdStreams to make worker processes natively async #1783

Closed
wants to merge 64 commits into from
Closed

Use PipeChannels or FdStreams to make worker processes natively async #1783

wants to merge 64 commits into from

Commits on Dec 15, 2020

  1. Create _worker_processes.py 

    Make a process cache with similar semantics to thread cache

provide interface to submit sync functions to the process cache
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    2fcf89d View commit details
    Browse the repository at this point in the history

  2. Create test_worker_process.py and fix revealed bugs

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    2c33843 View commit details
    Browse the repository at this point in the history

  3. improve tests, reduce race condition

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    62a28a5 View commit details
    Browse the repository at this point in the history

  4. Fixups from test failures

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    041370d View commit details
    Browse the repository at this point in the history

  5. Better way to raise BrokenWorkerError

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    0740307 View commit details
    Browse the repository at this point in the history

  6. Py3.6 compat for kill method

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    440153a View commit details
    Browse the repository at this point in the history

  7. prevent excessive lock releases

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    7117c3e View commit details
    Browse the repository at this point in the history

  8. revamp cache pruning test

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    b9f5076 View commit details
    Browse the repository at this point in the history

  9. don't cover test helpers that happen to be in the global namespace

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    8e39143 View commit details
    Browse the repository at this point in the history

  10. attempt to fix Alpine CI timeout and get a real traceback

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    52ca536 View commit details
    Browse the repository at this point in the history

  11. give segfault test a little longer timeout for CI

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    558b58c View commit details
    Browse the repository at this point in the history

  12. mark tests slow

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    7369314 View commit details
    Browse the repository at this point in the history

  13. the cache is now a deque

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    46fc931 View commit details
    Browse the repository at this point in the history

  14. remove unnecessary import time error possibility

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    3247502 View commit details
    Browse the repository at this point in the history

  15. match worker and thread names

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    a4a209a View commit details
    Browse the repository at this point in the history

  16. comment for sleeping during exception handler

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    2b41da4 View commit details
    Browse the repository at this point in the history

  17. check correct worker is pruned

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    a8291a9 View commit details
    Browse the repository at this point in the history

  18. clean up imports

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    473bf6f View commit details
    Browse the repository at this point in the history

  19. add alternative segfault method

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    3d2d3ed View commit details
    Browse the repository at this point in the history

  20. reformat and go back to using CI timeout

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    8bcbf78 View commit details
    Browse the repository at this point in the history

  21. avoid multierror, but can't race segfault types

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    2e57398 View commit details
    Browse the repository at this point in the history

  22. sometimes stack overflow is a memory error instead of a segfault

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    eea2acb View commit details
    Browse the repository at this point in the history

  23. cleanups

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    0c597b8 View commit details
    Browse the repository at this point in the history

  24. Don't join on kill 

    It's not actually necessary, I was using it to confirm that the processes died as expected and should have removed it earlier.
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    742f412 View commit details
    Browse the repository at this point in the history

  25. Don't double-check life of cached worker process 

    No longer necessary after introduction of _prune_expired_procs
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    92a6a80 View commit details
    Browse the repository at this point in the history

  26. eliminate worker timeout race 

    there was always a chance of having the worker timeout while going from _prune_expired_procs() to release().

Barrier provides the additional extra shared state to fully synchronize a process release
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    4c2b770 View commit details
    Browse the repository at this point in the history

  27. fix flakiness due to slow CI

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    a72cdd8 View commit details
    Browse the repository at this point in the history

  28. fix flakiness due to slow CI, continued

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    f074ef8 View commit details
    Browse the repository at this point in the history

  29. remove redundant test 

    was same as third case of test_run_in_worker_process_cancellation, but flakier
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    0f59d9d View commit details
    Browse the repository at this point in the history

  30. rename prune and PROC_CACHE

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    69fe295 View commit details
    Browse the repository at this point in the history

  31. relative imports

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    33a4c4d View commit details
    Browse the repository at this point in the history

  32. child monitor task is apparently unnecessary 

    also avoids extra thread per process on windows
    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    3dc7e06 View commit details
    Browse the repository at this point in the history

  33. optimize barrier use in is_alive

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    6b29e15 View commit details
    Browse the repository at this point in the history

  34. see if any platforms are persistently hanging

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    e560eca View commit details
    Browse the repository at this point in the history

  35. see if the monitor helps on pypy

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    8abac5b View commit details
    Browse the repository at this point in the history

  36. avoid multierror on child death

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    d42b9a5 View commit details
    Browse the repository at this point in the history

  37. xfail on flaky segfault behavior

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    e9040d6 View commit details
    Browse the repository at this point in the history

  38. don't check _work coverage because it only happens in the subprocess

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    076cdf8 View commit details
    Browse the repository at this point in the history

  39. boost coverage

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    e0685f6 View commit details
    Browse the repository at this point in the history

  40. boost coverage and xfail on TooSlowError

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    e9b204d View commit details
    Browse the repository at this point in the history

  41. more coverage

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    29e6221 View commit details
    Browse the repository at this point in the history

  42. ensure death on pop

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    9a2f094 View commit details
    Browse the repository at this point in the history

  43. expose API

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    1397e9a View commit details
    Browse the repository at this point in the history

  44. Docs and newsfragment

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    07c20be View commit details
    Browse the repository at this point in the history

  45. Expose new error

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    bed71fe View commit details
    Browse the repository at this point in the history

  46. fix docstring warning

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    bb464ff View commit details
    Browse the repository at this point in the history

  47. fix and test rare cancellation cleanup

    @richardsheridan
    richardsheridan committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    9260def View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2020

  1. flake-free test to raise BrokenWorkerError

    @richardsheridan
    richardsheridan committed Dec 16, 2020
    Configuration menu
    Copy the full SHA
    4253762 View commit details
    Browse the repository at this point in the history

  2. try just very long single segfault timeout

    @richardsheridan
    richardsheridan committed Dec 16, 2020
    Configuration menu
    Copy the full SHA
    e9bd34b View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2020

  1. prevent another obscure race

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    f7f2eb7 View commit details
    Browse the repository at this point in the history

  2. move comment

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    f6b9187 View commit details
    Browse the repository at this point in the history

  3. base synchronization purely on pipe behavior 

    TODO: close race condition between timeout and is_alive()==False
    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    91280a3 View commit details
    Browse the repository at this point in the history

  4. close race condition between timeout and is_alive()==False

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    1195190 View commit details
    Browse the repository at this point in the history

  5. move join to method, only really needed for tests

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    8cf7610 View commit details
    Browse the repository at this point in the history

  6. Close another race condition

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    e4801ba View commit details
    Browse the repository at this point in the history

  7. Use Windows message-oriented pipes to create PipeSendChannels

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    a139b1a View commit details
    Browse the repository at this point in the history

  8. improve coverage

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    f9fc4e1 View commit details
    Browse the repository at this point in the history

  9. more coverage

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    41e3965 View commit details
    Browse the repository at this point in the history

  10. style and comments

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    90ae32f View commit details
    Browse the repository at this point in the history

  11. Optimize large messages by avoiding copies

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    a839bca View commit details
    Browse the repository at this point in the history

  12. cover small message branch

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    5e7c3ff View commit details
    Browse the repository at this point in the history

  13. resolve disagreement between black versions :(

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    fd4cd8c View commit details
    Browse the repository at this point in the history

  14. detect conflict earlier, prevent message races

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    e303c23 View commit details
    Browse the repository at this point in the history

  15. Convert workers to use PipeSendChannels or FdStreams

    @richardsheridan
    richardsheridan committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    4ba5c30 View commit details
    Browse the repository at this point in the history