Make sure parallel mode can be used with sphinx-automodapi #38
Conversation
… catch cases where it is used inside a code block as an example
…ult indent was incorrect.
Fix encoding error with automodapi_writereprocessed and py2
…luding reference output files that can be compared with the input files.
…ue to the :no-main-docstr: option), the autosummary directive did not recognize that it was in the correct module.
Fix behavior of :inherit-members: when multiple automodapi directives are present
Use generic version of setup_conda rather than OS dependent
Adding python3.6 to appveyor testing
…s when locale is C (restricting the charset to ASCII)
Add a test configuration with a default ASCII locale
|
@eteq - since you have done a lot with the sphinx extensions in the past, could you review this? |
| @@ -110,6 +117,8 @@ def test_run_full_case(tmpdir, case_dir): | |||
| path_relative = os.path.relpath(path_reference, output_dir) | |||
| path_actual = os.path.join(docs_dir, path_relative) | |||
| assert os.path.exists(path_actual) | |||
| actual = io.open(path_actual, encoding='utf8').read() | |||
| reference = io.open(path_reference, encoding='utf8').read() | |||
| with io.open(path_actual, encoding='utf8') as f: | |||
There was a problem hiding this comment.
I could be wrong here, but I think this might cause trouble in locales where UTF8 doesn't come out as a default? Might it be sufficient to just do a Byte-wise comparison to sidestep the encoding problem?
sphinx_automodapi/smart_resolver.py
Outdated
| from docutils.nodes import literal, reference | ||
|
|
||
| # NOTE: This extension originally used a dictionary stored in app.env to keep |
There was a problem hiding this comment.
This whole approach seems a bit problematic, as it depends on file systems and sort of implementation details. Is there a reason you didn't try using theevent-env-merge-info event to do the parallel gathering (http://www.sphinx-doc.org/en/stable/extdev/appapi.html#event-env-merge-info)? I thought it was meant to solve exactly this problem? (Although I've not used it myself...)
|
This was closed as the master branch was updated to include history. I'll need to do a new implementation based on @eteq's comments anyway. |
Currently sphinx can't run in parallel mode when using sphinx-automodapi because extensions have to declare they are parallel-safe. However, I discovered that
smart_resolveris not parallel-safe since it relies on populating a global dictionary which doesn't get communicated back to the main process. Instead I've implemented a file-based approach for the dictionaries, with one file per process.For the astropy docs, using parallel mode
-j4on a 4-core mac laptop I get a speedup of 40%:compared to the serial mode:
I need to investigate why the speedup isn't more significant.