idea: include files (and csv tables)

[anton-k]

As far as I understand pandoc can process several files
in one way only. You have to list them in the command line. There is
a solution to simulate include files with scripting. It's indicated
in the pandoc's official guide.

Markdown is a tiny language. We should keep it small. So here is an idea
of how to simulate latex's input command without extending Markdown syntax.
We can overload include image construction. If a file has an image extension,
than it's treated as an image, but if it's .txt, it can be treated as Markdown:

![Show me if there is no such a file](subfile.txt)

I've come to this idea while thinking about long tables.
Imagine that someone is writing a research report. There are long
tables produced by an algorithm. Tables are saved in some
standard format for tables, for example CSV. And then user can write

![So it goes](table.csv)

[daroczig]

Not sure if this would fit in Pandoc's goals as being a "universal document converter", but you can do this easily with some wrapper around Pandoc. This would of course require some technical skills from you, but you would gain much more then the above suggested features.

There are a bunch of programming tools in Pandoc extra, from which I know (and develop) pander. You could easily write a simple brew file which would compile a list of all images in a directory and create a link for those and also reading csv files and printing would be not problematic too (e.g. putting a simple read.table(foo) in a brew chunk between <%=...%> tags.

I hope you would find this useful.

[michelk]

anton-k, I like the idea; had something similar in mind, when writing
a technical report recently. File-extension-dependent inclusion would be a nice
pandoc-extension for markdown.

Another idea related to that:

It would be great to have more general support for literate programming.

Currently I use the R-knitr-package
for mixing programming languages in technical reports; as an example see
https://github.com/yihui/knitr/blob/master/inst/examples/knitr-lang.Rmd.

Using pandoc directly with a file format, say lmd -- for literate markdown,
would facilitate the workflow considerably.

In that sense knitr works pretty well: You could include different languages
eg

``` {r test-r, engine='R'}
set.seed(123)
rnorm(5)
```

Unfortunately haskell is currently not included like eg

``` {engine='ghc'}
[x^2|x <- [1..10], x > 3 ]
```

With that in mind writing tutorials with REPLs like ghci, irb, R would be
more pleasant.

[jgm]

This could be done easily using the techniques described in the scripting documentation.

[michelk]

I opened a seperate issure #656 for it.

[jcangas]

Hi. I'm also looking for this feature :)
I found Marked.app has a nice extension:

  <<[Code title](folder/filename)

Same syntax is also supported by Leanpub system. I think some "include feature" is a must if you write large text in markdown.
Now I'm using Marked.app with Custom Markdown processor configured for pandoc, so I can include files that include files and so on. Very useful if you are writing a little book with code source samples :). But is a bit tedious need printing to PDF from the Marked.app. Having this feature in pandoc will allow for command line automation :)

[thewatts]

@jcangas -> it looks like ThoughtBot has done this before, based on looking at the raw markdown files from their Backbone on Rails book.

@jasonm was the person who worked on the project.

[jcangas]

@thewatts, Thanks for the clue. It is very easy to follow the "do your self" way, of course: I have a bit of Ruby that does the magic. But I see value in it as a standard feature with a standard syntax a no need for externals tools...

[thewatts]

Found what they use - they have a rakefile that will take and parse the <<[Code title](folder/filename) code, and then add it into the main file.

[dloureiro]

There is gpp in Pandoc Extras mentionned by @daroczig that can be used to include file directly (gpp is a gcc-like preprocessor) and much more. It provides a syntax to preprocess files and execute commands and the file inclusion could be achieved through #include <include ..> or even \include directives depending on the mode you select.
I'm currently working on a python wrapper aiming at using gpp to preprocess special commands in a markdown file before providing it to pandoc (things like file inclusion, code inclusion, color, underline, etc).
I will soon put it on github and if people are interested in such a wrapper I will add some more info about it.

[jcangas]

@thewatts I also have a rake file doing the same thing :). Well, mine is recursive also. I copy here so it can help others

# yields every line. Assume root_dir & file are Pathname objects
def merge_mdown_includes(root_dir, file, &block)
  file.each_line do |line|
    if line =~/(.*)<<\[(.*)\]$/
      incl_file = root_dir + $2
      yield $1 if block
      merge_mdown_includes(root_dir, incl_file, &block)
    else
      yield line if block
    end
  end
end

# hin about use previous routine:
merge_mdown_includes(root_dir, file) do |line|
   output_file.puts line
end

[nichtich]

Instead of adding another preprocessing syntax on top of Pandoc Markdown I use the following syntax to include files:

`filename.md`{.include}

one could also extend this to:

~~~ {.include}
filename.md
~~~

This way the inclusion syntax can act on the abstract syntax tree (AST) of a Pandoc document - one can get the same result from HTML like this (HTML -> Markdown -> Markdown with inclusions -> Target format):

<code class="include">filename</code>

Here is a small hack in form of a Perl script that I use by now.

while(<>) {
    if (/^`([^`]+)`\{\.include\}\s*$/) {
        if (-e $1 && open my $fh, '<', $1) {
            local $/;
            print <$fh>;
            close $fh;
        } else {
            print STDERR "failed to include file $1\n";
        }
    } else {
        print $_;
    }
}

The final implementation should work on the AST as well to allow inclusion inside other elements, for instance:

* `longlistitem.md`{.include}

[mdengler]

@nichtich Nice idea; converted to python and combined with Makefile:

# Makefile fragment

%.pdf : %.md
    cat $^ | ./include.py | pandoc -o $@

#!/usr/bin/env python

import re
import sys                                                                                                     
include = re.compile("`([^`]+)`\{.include}")
for line in sys.stdin:
    if include.search(line):
        input_file = include.search(line).groups()[0]
        file_contents = open(input_file, "rb").read()
        line = include.sub(line, file_contents)
    sys.stdout.write(line)