Make `Format` an enumerated type

[anton-k]

Format is a synonym for String. User have to look at the source code to find out right values for this type. (It can be "html" or "Html" or "latex" or "LaTeX" or "tex"). It's not clear wich from the docs alone. Maybe it's better to define a new data type:

data Format = FormatHtml | FormatTex | ...

[mpickering]

+1 to this

I couldn't find any actual type synonym definition for Format.
I used ag -s type | ag -s Format and the results were useless.

Is the type synonym referred to a philosophical one? Or has it been fixed already?

[timtylin]

Format is defined in pandoc-types, specifically in Text.Pandoc.Definition.

Note that it already has a "it just works" instance of Eq that ignores case:

newtype Format = Format String
               deriving (Read, Show, Typeable, Data, Generic)

instance IsString Format where
  fromString f = Format $ map toLower f

instance Eq Format where
  Format x == Format y = map toLower x == map toLower y

I seem to recall that the reason Format is a String is mainly due to the way extensions are specified, where they are just concatenated onto Format using the + and - char. This is not just the way the CLI works, but also how the module itself exposes getReader and getWriter (i.e., the formats get passed through Text.Pandoc.parseFormatSpec).

I mean it's arguably a hack, but changing it to an actual sum type (in fact a product type when you consider the set of extensions) will definitely break backward compatibility, so this is most likely going to be a 2.0 thing.

Oh, so it's a different package. Thanks for the info.
I'll drop it for the time being.

[jgm]

+++ Tim T.Y. Lin [Mar 04 15 00:45 ]:

I seem to recall that the reason Format is a String is mainly due to the way extensions are specified, where they are just concatenated onto Format using the + and - char. This is not just the way the CLI works, but also how the module itself exposes getReader and getWriter (i.e., the formats get passed through Text.Pandoc.parseFormatSpec).

No, the extensions are not part of the string on Format.

I mean it's arguably a hack, but changing it to an actual sum type (in fact a product type when you consider the set of extensions) will definitely break backward compatibility, so this is most likely going to be a 2.0 thing.

Right. In principle, a sum type would be better. However, it's a big
change and would break lots of existing filters, so it's not clear it's
worth it.

[timtylin]

No, the extensions are not part of the string on Format.

Right, turns out getReader/getWriter takes a String and not a Format. Well that just makes it even more inconsequential then. Does anything directly use Format when interfacing with Pandoc, other than filters written in Haskell?

[tarleb]

One question is whether it should be possible to pass a custom Format, or whether Format can only contain known formats. I.e., should we use

data Format = Markdown | Docx | ReStructuredText | …

or rather

data KnownFormat = Markdown | Docx | ReStructuredText | …

data Format = Format KnownFormat
            | CustomFormat String

I can see arguments for both variants; most arguments in favor of a finite sum type are given above. On the other hand, we'd limit users in their ability to pass format information to filters, custom writers, and programs built on top of pandoc's library.

Personally, I lean towards a finite sum type, as I feel the advantages out-weight the slight loss in flexibility. The only real problem I see is how to handle unknown formats specifications during parsing: Should those be turned into a default format, or maybe a code block?

[jgm]

I'm not sure about the finite vs extensible question, but like you I lean towards finite.
The obvious approach would be to just omit raw content with an unknown format, with a log warning.

If we're thinking about this question, I think we might want to address a bigger issue about raw blocks. This has come up with ipynb. Jupyter notebook code cells will often generate output in multiple formats: for example, a table might be produced in text/latex and text/plain. The plain version is a fallback, so if you're converting to HTML, the HTML version will be used; if to LaTeX, the fallback would be to include the plain text version in a verbatim environment.

It's tough to handle this properly in pandoc. Given that the behavior of the reader is supposed to be independent of the writer, we can either (a) include both the HTML version as a raw block and the plain text version as a code block, with the result that you'll see TWO versions of the table when it's converted to HTML or (b) just include the HTML version, with the result that there will be no fallback when it's converted to LaTeX or other formats. A bad choice, which makes it impossible to fully emulate nbconvert.

One thing that would help here would be an AST element that includes content conditionally on the format. Something like this:

[ IfFormat HTML [RawBlock "<table>..."]
, IfFormat LaTeX [CodeBlock "..."]
]

With this kind of structure one could remove the Format specifier from the RawBlock itself.

But thinking about the fallback part of this, one sees a need for format specifications that encompass multiple formats, like HTML OR Markdown or NOT(HTML OR Markdown). (Format could perhaps be a Boolean algebra, https://hackage.haskell.org/package/cond-0.4.1/candidate/docs/Data-Algebra-Boolean.html)