ChunkedDoc lua APIs

[multimeric]

I've had a read of #6122, and I think there's still a need for a general chunking mechanism accessible to lua. Currently the only (?) lua API for working with a ChunkedDoc is pandoc.structure.split_into_chunks, which returns one, but it can't be used any further.

Describe your proposed improvement and the problem it solves.
I think there are two necessary tools to make this fully configurable for users in lua:

• Ideally, a constructor for ChunkedDoc so we don't have to, for example, include headers in each chunk and then split the document. I imagine the constructor would just take a list of Chunk objects (which don't have a constructor either).
• Some mechanism for returning a ChunkedDoc from a lua filter, or otherwise wrap a ChunkedDoc inside a Pandoc so it can be returned from lua.

Describe alternatives you've considered.
In theory a user could write a filter that converts the document into a series of divs with a header, and ask the user to --to chunkedhtml --split-level 2, but this pushes too much onto the user when ideally it could all be automated.

[tarleb]

I'm open to extending the API for chunks, but I think we need to clarify the intended use-cases, and how those could be extended, because I don't think I fully understand the goal yet.

One reason why the current API looks the way it does is that ChunkedDoc objects are intended to either

• generate a TOC.
• or write chunks into different files.

Only the first point makes sense in the context of a filter, and even then, there's a specialized function. The primary use case is for custom writers to generate and handle those chunks, including writing them to files.

Converting a ChunkedDoc back into a Pandoc type would be very lossy and opinionated, so I'm not sure a predefined function would be helpful for most users.

There aren't' any ChunkedDoc/Chunk constructors yet because I didn't anticipate that those could be useful. Could you provide some short pseudo-code that might allow me to understand how those would be used?

[arcmode]

I think this issue applies to my need as well. I am writing a Lua reader that collects documents from a directory structure and creates a document structure that I am currently passing to the chunkedhtml writer. My goal is to create groups from the collected documents (each doc can appear in multiple groups) and produce multiple index files (similar to TOC); one for all the groups, linking each group by name to the index (TOC) for the respective group of documents. My current approach involving the encapsulation of pandoc.read results into pandoc.Div preceded by pandoc.Heading is cumbersome and would prefer to control the structure via ChunkedDoc and Chunk. This way I could could create a structure that is not exactly the same as the default chunkedhtml process creates. then if I need I would be adding a Lua writer for my custom use case.

[iandol]

This may seem a bit like a dog chasing its own tail -- but say I have a single markdown file and I want to split it into several markdown files, with footnotes and links etc fixed so I can pass it to Quarto's book / website renderer (which requires each chapter in its own separate .md file); would Lua with a ChunkedDoc API be able to accomplish this??

[multimeric]

say I have a single markdown file and I want to split it into several markdown files, with footnotes and links etc fixed so I can pass it to Quarto's book / website renderer (which requires each chapter in its own separate .md file); would Lua with a ChunkedDoc API be able to accomplish this??

I'm also coming from Quarto. From @tarleb's answer, it sounds like the ChunkedDoc isn't designed for use in filters, but instead is designed for custom writers. I don't believe Quarto supports custom writers, meaning that, to use this in Quarto we would need a custom ChunkedDoc constructor, and support for custom writers in Quarto.

Could you provide some short pseudo-code that might allow me to understand how those would be used?

The context for this question was mostly this repo. In trying to split out figures into separate documents, I'm currently using
file:write(pandoc.write(...)) inside a filter to write multiple outputs, but I would rather simply return a ChunkedDocument. The lua API I would prefer might be something like: chunked_doc = ChunkedDoc(table_of_divs). Of course I would need Quarto to know how to handle this in some way for this API to be helpful to me.