escape-shell not working for minted

[gandalfsaxe]

Hi,

I'm having trouble using the --shell-escape flag for converting markdown to PDF having source code syntaxed by minted using a pandoc-minted filter.

Minimal example:

1. Input file minted-test.md:

---
header-includes: \usepackage{minted}
---

```python
import numpy as np 
```

2. Install pandoc-minted with pip install pandoc-minted

3. Run command:
   pandoc --filter pandoc-minted --pdf-engine-opt=--shell-escape -o minted-test.pdf minted-test.md

I get error:

Error: cannot read infile: [Errno 2] No such file or directory: 'input.pyg'
system returned with code 256
Error: cannot read infile: [Errno 2] No such file or directory: 'input.pyg'
system returned with code 256
Error: cannot read infile: [Errno 2] No such file or directory: 'input.pyg'
system returned with code 256
Error producing PDF.
! Package minted Error: Missing Pygments output; \inputminted was
probably given a file that does not exist--otherwise, you may need
the outputdir package option, or may be using an incompatible build tool,
or may be using frozencache with a missing file.

See the minted package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.62 \end{minted}

with the --verbose option I get the full error output: https://gist.github.com/gandalfsaxe/e4f0bb280c8e441385bf46cd5be443fe

May relate to nick-ulle/pandoc-minted#2 or gpoore/minted#93 - however I can't find out if the fault is with minted or pandoc? Any suggestions?

I did test doing intermediate latex conversion, then converting the .tex file to pdf via pdflatex. It works if I do it in two steps:

1. pandoc: .md -> .tex (using filter pandoc-minted)
2. pdflatex .tex -> .pdf (using --shell-escape)

Even if it cannot be solved immediately / easily, it would be nice to gain some clarity into what is causing the issue.

[mb21]

Most definitely not a fault in plain pandoc. Sounds like minted didn’t install or cannot find all its files, please ask on their support channels or maybe better stackoverflow... or possibly there are minted users on pandoc-discuss.

[mb21]

btw. from the manual:

To debug the PDF creation, it can be useful to look at the intermediate representation: instead of -o test.pdf, use for example -s -o test.tex to output the generated LaTeX. You can then test it with pdflatex test.tex.

[gandalfsaxe]

I forgot to mention, I did test that. It works if I do it in two steps:

1. pandoc: .md -> .tex (using filter pandoc-minted)
2. pdflatex .tex -> .pdf (using --shell-escape)

Therefore I think the problem could be pandoc.

[davidcarlisle]

your log shows the problem, it is "-output-directory "-output-directory" "./tex2pdf.55141 which means that the intermediate files get written somewhere that tex can not find. If pandoc has an option not to do that, it would probably work or you could put ./tex2pdf.55141// at the front of the TEXINPUTS environment variable if that directory name is known in advance so any files in there may be input.

[davidcarlisle]

@mb21 are you sure this should be closed here? The issue is in the way pandoc is calling pdftex not anything really that can be handled in minted (or latex itself)

[gandalfsaxe]

@davidcarlisle Ah, that makes me very interesting in whether pandoc has such an option. If anyone knows it's probably @jgm.

As for a temporary fix, are you suggesting putting export TEXINPUTS=./tex2pdf.55141// in my bash profile or elsewhere?

[davidcarlisle]

@gandalfsaxe I was suggesting something along those lines but it doesn't work and if I'm reading the haskell at

https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/PDF.hs#L297

correct;y pandoc basically does that already.

trouble is minted doesn't just require tex to find the files it needs the shell (pygmentize in particular) to know where they got written, tex's --output-directory option is such a pain it complicates the entire pipeline:-)

However minted has an option to specify the directory so if you can change header-includes: \usepackage{minted} to header-includes: \usepackage[outputdir=...]{minted} then it should all work again but I don't know the timing of how those includes get constructed.

[mb21]

I can reopen this, but I don’t see a simple solition implementable on pandoc... maybe you’re better off just doing the two-steps.

[davidcarlisle]

@mb21 doing it in two steps only works as the user isn't using --output-directory when calling pdflatex.
Could pandoc have an option not to use --output-directory or failing that an option to specify a fixed temp directory. If pandoc had something like --tempdir=foobar which made it use --output-directory=foobar on the pdflatex call then the OP could call minted via \usepackage[outputdir=foobar]{minted} and pygments would know where the files were. Calling pdflatex --output-directory with a more or less randomly generated directory name makes it pretty hard to configure any subsidiary programs to know where generated files are.

[gandalfsaxe]

@davidcarlisle Out of curiosity, would \usepackage[outputdir=foobar]{minted} still work universally? The ideal solution would be one that worked for both pandoc: md -> pdf, and two-step pandoc: md ->tex + pdflatex: tex -> pdf.

[davidcarlisle]

@gandalfsaxe it would work if you used pdflatex --output-directory=foobar in your second stage command. A simpler solution for the user of course is not to use --output-directory and have a way to tell pandoc not to use it either, but if pandoc is already written assuming --output-directory is being used just having a way to fix that directory may be a less intrusive change in the coding there.

[gandalfsaxe]

Ok. Well I guess the best bet for a solution is what you write in the other thread:

It shouldn't be hopeless: pandoc knows the number (it is probably the the $$ shell process number) as it uses it in its commandline call to pdftex so it could in principle use it while generating the tex file that it is using with that commandline.

[gandalfsaxe]

If pandoc had something like --tempdir=foobar which made it use --output-directory=foobar on the pdflatex call then the OP could call minted via \usepackage[outputdir=foobar]{minted} and pygments would know where the files were. Calling pdflatex --output-directory with a more or less randomly generated directory name makes it pretty hard to configure any subsidiary programs to know where generated files are.

I've noticed that pandoc already generates a temporary folder in the same path as the output files. It's deleted after the pandoc command is finished running. Seems like a good candidate for a temporary folder for minted, and should be very easy to implement, @jgm ?

[ayatkyo]

i found a solution by setting -aux-directory to same directory with outputdir in minted options.

as described in minted manual :

The -output-directory and -aux-directory (MiKTeX) command-line options for LATEX cause problems for minted, because the minted temporary files are saved in , but minted still looks for them in the document root directory. There is no way to access the value of the command-line option so that minted can automatically look in the right place. But it is possible to allow the output directory to be specified manually as a package option.

so change \usepackage{minted} to \usepackage[outputdir=temp]{minted} and --pdf-engine-opt=-aux-directory=temp then run command :

pandoc --filter pandoc-minted --pdf-engine-opt=--shell-escape --pdf-engine-opt=-aux-directory=temp -o minted-test.pdf minted-test.md

[jgm]

Another approach: Try with --pdf-engine=latexmk --pdf-engine-opt=-outdir=temp --pdf-engine-opt=--shell-escape. That's what the changelog suggests for a persistent temp directory.