feature request: option to have html parsed inside markdown

[amueller]

It would be really really great if there was an option to parse html inside markdown.
I (and many other people) are using jupyter notebook to write complex documents. These are written in markdown, rendered in html and converted to other formats using pandoc.

However, when converting tables or image tags or any other html included in the markdown to, say latex, all the formatting is lost.
That makes it really hard to create complex documents, as markdown is not a very powerful layout language (there is not standard way to specify image sizes, tables can't have colspan or rowspan etc).

It would be great to be able to leverage html for that, in particular as pandoc already has html support.

Thanks!

[jgm]

+++ Andreas Mueller [Feb 15 16 11:50 ]:

It would be really really great if there was an option to parse html
inside markdown.
I (and many other people) are using jupyter notebook to write complex
documents. These are written in markdown, rendered in html and
converted to other formats using pandoc.

However, when converting tables or image tags or any other html
included in the markdown to, say latex, all the formatting is lost.
That makes it really hard to create complex documents, as markdown is
not a very powerful layout language (there is not standard way to
specify image sizes, tables can't have colspan or rowspan etc).

The difficulty is that in many of these cases, the Pandoc
data structure (the intermediate format that pandoc uses in
all its conversions) shares the expressive limitations of
Markdown. (After all, it started as a way to represent
Markdown.) It doesn't currently allow colspans or rowspans.

Note that there IS now a standard way in pandoc's Markdown
to specify image sizes. (As of 1.16.x)

I'm having a hard time thinking of things that can be
expressed in HTML and can be represented in the Pandoc
AST, but can't be expressed in pandoc's Markdown.

[amueller]

but pandoc can convert html to latex, right? I was operating under the assumption that that would "work", i.e. rowspan and colspan would be preserved. But it looks like the table formatting is actually discarded. Is that right?

[amueller]

I saw the addition of the image size. That's great, though the trouble with that is that it is pandoc specific, so using any other way to render the markdown will likely not interpret that correctly.

[jgm]

All conversions go:

source format -> Pandoc structure -> target format

This is how we can do hundreds of conversions without
writing hundreds of converters -- we just need to write
a parser for each format to a Pandoc structure,
and a writer from a Pandoc to each format.

So all conversions are limited by the expressive limitations
of the Pandoc structure. (See the beginning of the README
where this is made explicit.)

+++ Andreas Mueller [Feb 15 16 12:10 ]:

but pandoc can convert html to latex, right? I was operating under the
assumption that that would "work", i.e. rowspan and colspan would be
preserved. But it looks like the table formatting is actually
discarded. Is that right?

—
Reply to this email directly or [1]view it on GitHub.

References

1. feature request: option to have html parsed inside markdown #2716 (comment)

[amueller]

Yeah that makes total sense. Are there any plans to enrich the pandoc internal format?

[jgm]

+++ Andreas Mueller [Feb 15 16 13:47 ]:

Yeah that makes total sense. Are there any plans to enrich the pandoc
internal format?

Yes, but every change is a lot of work, because all the
readers and writers need to be updated (as well as just
about every other part of the project).

With just a couple exceptions, we've limited the AST to
what is representable in Pandoc's Markdown. Since we don't
have a good way of doing colspans and rowspans in tables
in Markdown, we haven't added that. But it might be worth
doing so anyway.

[jeffmcneill]

Is this open somewhere else, or closed due to lack of interest and/or level of difficulty?

[minrk]

Is the general issue of parsing inline HTML not addressable? I've taken a stab at just passing RawInline blocks through pandoc --to json in a filter, and it seems to work for image tags, though I've no idea where it will fall down.

[minrk]

My super-simplistic filter:

#!/usr/bin/env python
"""
pandoc filter for handling inline HTML in markdown

Seems to work for simple image tags, at least
"""
import json
import sys
from pandocfilters import toJSONFilter
from nbconvert.utils.pandoc import pandoc

def inline_html(key, value, format, meta):
    if key != 'RawInline':
        return

    raw_format, raw_html = value
    new_json = pandoc(raw_value, raw_format, 'json')
    new_data = json.loads(new_json)
    return new_data['blocks']

if __name__ == '__main__':
    toJSONFilter(inline_html)

This replaces any RawInline block with the result of passing it through pandoc with the associated reader. It seems to do what I want for simple cases (image tags), but the RawInline blocks are segmented in such a way that nontrivial bits of HTML (i.e. more than a single tag) will probably not be handled correctly.

[jgm]

The problem isn't that this can't be done, technically. It's more a question whether this would be desirable. After all, in Markdown you're supposed to be able to pass through raw HTML. But the result of (writeHtml . readHtml) in pandoc might be different from the original. Pandoc can throw away information, e.g. attributes. So authors might find that they wrote one thing as raw HTML but got something else in the HTML output. This is why it would be problematic to parse raw HTML inside the Markdown reader. (Note also that the reader doesn't know what the output format will be, and so it can't implement the rule "leave it alone if we're going to HTML.") I suppose one could implement this as a filter run in pandoc.hs (for non-HTML output) or even in individual writers. However, some people take advantage of the present behavior (where raw HTML simply won't appear, e.g., in LaTeX output) by putting raw HTML and raw LaTeX side by side, knowing that only one will appear in the html or latex output. +++ Min RK [Jan 25 17 07:14 ]:

…

Is the general issue of parsing inline HTML not addressable? I've taken a stab at just passing RawInline blocks through pandoc --to json in a filter, and it seems to work for image tags, though I've no idea where it will fall down. — You are receiving this because you modified the open/close state. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread. References 1. #2716 (comment) 2. https://github.com/notifications/unsubscribe-auth/AAAL5P-u8TCdhAJf1ombD16vw8nQGLhWks5rV2bqgaJpZM4HanH6

[amueller]

I find just throwing away content of a supported input type pretty odd behavior.
While it makes sense for html output, as you pointed out, the reader doesn't know the output and it doesn't make a lot of sense for any other output format.

Using the implicit dropping of some content seems like a pretty bad way to support different mime-types. I'm not sure if there's something in markdown that would enable declaring raw types like in ReST.

[jgm]

I really don't like this particular feature of Markdown, and I find RST better designed in this respect. However, it is part of the Markdown syntax description that you can insert raw HTML, and it's passed through unchanged to the (HTML) target. So we have to respect that, I think. +++ Andreas Mueller [Jan 25 17 10:06 ]:

…

I find just throwing away content of a supported input type pretty odd behavior. While it makes sense for html output, as you pointed out, the reader doesn't know the output and it doesn't make a lot of sense for any other output format. Using the implicit dropping of some content seems like a pretty bad way to support different mime-types. I'm not sure if there's something in markdown that would enable declaring raw types like in ReST. — You are receiving this because you modified the open/close state. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread. References 1. #2716 (comment) 2. https://github.com/notifications/unsubscribe-auth/AAAL5F2vJ6n-CNn6p9JizdfG6xeDSEnvks5rV48mgaJpZM4HanH6