Split docutils only functionality into separate package #347
Comments
I think this would be great. The thing that's missing right now for convenient integration with docutils, I think (besides separating dependencies), is a version of from docutils.core import publish_string
from myst_parser.sphinx_parser import MystParser
print(publish_string(source="{math}`e^{i\pi} = -1`", parser=MystParser(), writer_name='html5'))(and with that integration into a variety of other docutils-based pipelines; my use case is integrating with Alectryon for Coq proofs.) As currently written, this code crashes: (in fact I see now that it's almost exactly the same example as in https://sourceforge.net/p/docutils/mailman/docutils-users/thread/rkv4nb%24139g%241%40ciao.gmane.io/#msg37118232) This crash is due to def parse(self, inputstring: str, document: nodes.document, renderer: str = "sphinx") -> None:
if renderer == "sphinx":
config = document.settings.env.myst_config # Here, since docutils will call this function without specifying the renderer
else:
config = MdParserConfig()It works fine if I use a subclass of from docutils.core import publish_string
from myst_parser.sphinx_parser import MystParser
class NonSphinxMystParser(MystParser):
def parse(self, inputstring, document) -> None:
return super().parse(inputstring, document, r"¯\_(ツ)_/¯")
print(publish_string(source="{math}`e^{i\pi} = -1`", parser=NonSphinxMystParser(), writer_name='html5').decode('utf-8'))… but it's no clear whether that's the right approach (?) So separating things out in a way that makes this works would be very very nice, especially setting MyST options works from the usual Also nice would be some canonical way to register MyST's directives and roles: some function that clients can call to make the appropriate calls to In #342 @astrojuanlu wrote:
and @choldgraf wrote
But am I correct to think that this requires extra work to also parse MyST's config, recreate its custom roles/directives (?) and make a In any case it looks like with very small changes (and independently of integrating with the official docutils package) applications like Alectryon that already depend on docutils but not necessarily sphinx could add support for MyST, which would be great! |
The global approach in 395 isn't really sustainable: it requires all-ways cooperation between all projects that want to customize MathJax. Additionally, when processing a MyST document without Sphinx, the MathJax configuration changes are not performed (part of executablebooks#347). And, of course, this approach of overriding the MathJax object causes issues down the line for projects that need to customize MathJax (the setting in Sphinx isn't sufficient, see sphinx-doc/sphinx#9450) The following two approaches would not cause these issues: 1. Add a custom script instead of touching the mathjax3_config variable; something like this, essentially: ```js app.add_js_file(None, priority=0, body=""" var MathJax = window.MathJax || MathJax; MathJax.options = MathJax.options || {}; MathJax.options.processHtmlClass = (MathJax.options.processHtmlClass || "") + "|math"; """) ``` - Don't touch MathJax_config at all; instead, add an explicit `mathjax_process` class on all math nodes, either by changing `docutils_renderer` (this PR) or by adding a Docutils transform to processes all math nodes: ```python class ActivateMathJaxTransform(Transform): default_priority = 800 @staticmethod def is_math(node): return isinstance(node, (math, math_block)) def apply(self, **kwargs): for node in self.document.traverse(self.is_math): node.attributes.setdefault("classes", []).append("mathjax_process") ``` This PR isn't ready for merging; it's just to start a discussion.
|
Given the end-of-life for recommonmark, is there a chance for a Docutils-only version of the MyST parser that can be utilised by Docutils? |
|
Heya, @cpitclaudel has kindly made a PR to allow for controlling the configuration via docutils: #426, that I'm just trying to find time to circle round to and finalise the tests. The only sticking point perhaps is myst-parser's pinned dependency on sphinx, creating a cylic dependency. I specifically added this, because changes in docutils/sphinx kept breaking myst-parser for users, so would have to think how this could be achieved |
|
myst-parser v0.16.0 now introduces docutils-only functionality (https://myst-parser.readthedocs.io/en/latest/docutils.html) and https://pypi.org/project/myst-docutils/ release pipeline, which inlcudes no install dependencies on sphinx/docutils 😄 @gmilde, do you want me to open an issue on docutils, to move from recommonmark to myst-docutils? |
|
Am 10.12.21 schrieb Chris Sewell:
myst-parser v0.16.0 now introduces docutils-only functionality
This is good news. Thank you.
From [MyST with Docutils](https://myst-parser.readthedocs.io/en/latest/docutils.html#myst-with-docutils)
I see that 5 front-end tools shall be installed as well (but in the
package download I didn't find the corresponding files).
IMV, separate front end tools are not necessary when we can get this working
with `docutils-cli.py --parser=myst_parser.docutils_`.
@gmilde, do you want me to open an issue on docutils, to move from
recommonmark to myst-docutils?
If the above works, Docutils can add `'myst': 'myst_parser.docutils_',`
to `docutils.parsers._parser_aliases` to provide a shorter parser name.
In future, the default "markdown" and "commonmark" parsers may switch to
"myst", but this is an API change which requires planning, advance warnings,
and tests.
Feel free to open an enhancement ticket or start a thread on
***@***.*** to discuss arising issues, a test
strategy and possible closer integration.
|
They don't need separate files, they use python entry points: Lines 50 to 57 in 11fb239 I feel this is the more modern approach for including CLI tools in python package distributions.
I felt this was easier for end users, and also inline with the separate
Yep will do then 👍 |
|
Also, just to note, there are now separate test jobs for basic testing of myst-docutils against docutils 0.16, 0.17, and 0.18 (on top of the full test suite): https://github.com/executablebooks/MyST-Parser/runs/4546203130?check_suite_focus=true |
|
Am 16.12.21 schrieb Chris Sewell:
IMV, separate front end tools are not necessary
I felt this was easier for end users, and also inline with the separate
rst2xxx.py front end tools that docutils already ships with
Unfortunately, the approach of a separate front-end tool for every
reader-parser-writer combination doesn't scale well. This is why Docutils
0.17 introduced the generic "docutils-cli" front-end together with Markdown
support.
From the description at [MyST with Docutils](https://myst-parser--426.org.readthedocs.build/en/426/docutils.html),
I suppose that `docutils-cli.py --parser=`myst_parser.docutils_`
should do the trick.
(It should also enable the user to get output from writers not in the
myst2... set, e.g., ODT or LaTeX for processing with LuaTeX or XeTeX
engines).
The generic front-end has also command-line support for all
[Docutils configuration settings](https://docutils.sourceforge.io/docs/user/config.html).
For end users consistent configuration framework would help a lot.
If `myst_parser.docutils_.Parser.settings_spec` could
provide an interface to the relevant myst configuration options, this would
allow configuring reader, parser, and writer from the command line
or a common config file. A user would easily get help for all available
options with `docutils-cli --parser=myst_parser.docutils_ --help`.
|
Oh indeed, but also having to write the full parser path, etc, every time is not ideal.
Yes the parser is just a standard docutils parser.
Yep this is already what happens, |
|
Am 20.12.21 schrieb Chris Sewell:
Unfortunately, the approach of a separate front-end tool for every
reader-parser-writer combination doesn't scale well.
Oh indeed, but also having to write the full parser path, etc, every time is
not ideal.
This would be mitigated by a "myst" parser alias (implemented in
[r8916]). Now, `docutils-cli.py --parser=myst` should work just like
`myst-docutils-html5`. Different default components can be set in
[docutils.conf files](https://docutils.sourceforge.io/docs/user/config.html):
system-wide, user-wide and per directory.
Another option would be a common `myst-docutils` front-end (with a
`--writer` option) instead of a set of 5 (out of 12) front-ends. Again,
docutils.conf files would allow to set the preferred writer and reader.
I suppose that docutils-cli.py --parser=myst_parser.docutils_` should do
the trick.
Yes the parser is just a standard docutils parser.
I realized that docutils-cli.py restricted the `--parser` values to
"rst", "recommonmark" and "markdown". This is fixed in [r8915].
Again here, I would suggest you use entry points to load parsers,
rather than module paths.
I have been experimenting with a `docutils.parsers.myst_wrapper` module
(analogue to `docutils.parsers.recommonmark_wrapper`).
Currently, the only difference to the parser name alias is a more helpful
error message in case the upstream parser is missing.
A wrapper module may become necessary, if the import name changes
or to implement a "commonmark" parser that selects whichever of "myst" or
"recommonmark" is available.
If myst_parser.docutils_.Parser.settings_spec could provide an
interface to the relevant myst configuration options, [...]
Yep this already what happens, [...]
This is good news. From the documentation it was not clear to me, that this
means the `myst-…` options can be set also from a `docutils.conf` file,
nor that the `myst-doctuils-…` front-ends also feature the config settings
of the relevant writers.
...
reStructuredText Parser Options
-------------------------------
BTW: Why do the `myst-doctuils-…` front-ends include rst parser settings?
Maybe some settings should move to a generic "parser options" section (and
"smart-quotes" to the generic options). But settings may also be defined in
several components (see, e.g. the "stylesheet…" settings in HTML and LaTeX).
|
yep that sounds good, although I would note, I do like the "inherent" tab-completion you get in the terminal with the $ myst-docutils-<tab>
myst-docutils-html myst-docutils-html5 myst-docutils-latex myst-docutils-pseudoxml myst-docutils-xmlmakes it very easy to use.
myst-parser hooks in to the RST directive/role parsing mechanisms, I seem to recall some of these settings were required, and it was breaking without them being present.
Anyhow, I have now opened https://sourceforge.net/p/docutils/feature-requests/86/ and created #487 for any parallel discussion here, so any more conversation can move to them cheers |
extracted from #342:
ah I completely missed the response on the email thread.
Ah well that's a different proposition: I literally have myst_parser/docutils_renderer.py and myst_parser/sphinx_renderer.py as a sub-class, so that it is certainly possible to use myst-parser with "docutils only" functionality.
If I had known previously about such an intention to ship Markdown in docutils, I would certainly have considered splitting the "docutils only" aspects in to a separate "myst-docutils" package (and have that as a dependency here), which I think would probably be a better solution than just moving dependencies to extras.
Indeed, if there was an agreement in principle with the docutils guys to include something like myst-docutils in docutils I would look into this.
The text was updated successfully, but these errors were encountered: