Output to PDF in IEEE format

[cleong110]

Following on from #25 , let's try and output to a PDF using IEEE journal format.

https://template-selector.ieee.org/secure/templateSelector/publicationType, I picked Journal/IEEE Transactions on Emerging Topics in Computational Intelligence

[cleong110]

IEEE-Transactions-LaTeX2e-templates-and-instructions.zip

Here's the official IEEE instructions/templates

[cleong110]

And here's the Makefile in the project:

markdown: dst dst/index.html dst/style.css

server: dst dst/style.css dst/index.md dst/sitemap.xml

dst/index.html: dst/index.md src/references.bib src/template/index.html dst/style.css
	pandoc dst/index.md --template src/template/index.html -s --table-of-contents --bibliography=src/references.bib --citeproc --columns 1000 -H src/header.html -V lang=en -o $@

dst/index_shortcode.md: dst/index.md
	node addons/emoji-to-shortcode/main.js dst/index.md > $@

dst/index.pdf: dst/index_shortcode.md src/references.bib
	cd dst && pandoc -f markdown+emoji -L../addons/latex-emoji.lua index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=../src/references.bib --citeproc -o index.pdf

dst/thesis.pdf: dst/index_shortcode.md src/references.bib
	#pandoc -f markdown+emoji -L addons/latex-emoji.lua src/thesis/main.tex -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=src/references.bib --citeproc -o index.pdf
	cd src/thesis && pandoc main.tex -s -N --natbib --pdf-engine=xelatex -o index.pdf


dst/index.md: src/index.md src/markdown_fix.sh src/formats.md dst tmp/datasets.md dst/assets
	cat src/index.md > $@
	bash src/markdown_fix.sh $@

dst/style.css: dst src/styles/custom.css
	cat src/styles/custom.css > $@

dst/sitemap.xml: dst src/sitemap.js
	node src/sitemap.js > $@

# TODO make this depend on all asset files
dst/assets: src/assets/tasks/tasks.svg
	mkdir -p $@
	cp -r src/assets/* $@

dst:
	mkdir $@


# Temporary files
tmp:
	mkdir $@

# TODO make this depend on all dataset json files
tmp/datasets.md: src/datasets.js tmp
	node src/datasets.js > $@


dst/sections: dst/index.tex
	python src/split_sections.py


overleaf: dst/sections tmp
	rm -rf tmp/overleaf
	git clone https://git.overleaf.com/611a535f64617c334d122e31 tmp/overleaf
	mkdir -p tmp/overleaf/parts/background
	cp -r dst/sections tmp/overleaf/parts/background
	rm -f tmp/overleaf/parts/background/sections/.DS_Store
	cp -r dst/assets tmp/overleaf/parts/background
	cp src/references.bib tmp/overleaf/parts/background
	rm -f tmp/overleaf/parts/background/assets/.DS_Store
	cd tmp/overleaf && \
		git add -A && \
		git commit -am "autoamtic sections upload" && \
		git push

#
#latex: tex/main.tex
#
#tex:
#	mkdir $@
#
dst/index_emoji.tex: dst dst/index_shortcode.md src/references.bib
	pandoc -f markdown+emoji -L addons/latex-emoji.lua dst/index_shortcode.md --shift-heading-level-by=-1 -s -N --natbib -o $@

dst/index.tex: dst/index_emoji.tex src/replace_gifs.py
	python src/replace_gifs.py dst/index_emoji.tex $@

#
#
#tex/references.bib: src/references.bib tex
#	cp src/references.bib $@

[cleong110]

Not really sure which to use, but maybe base it off of the overleaf one?

[cleong110]

OK, now how to run this? Somehow I need Python, I need make, and I need it all to run on Windows. Alternatively give up and do this tomorrow on my Linux workstation. But for now...

# manually replace all `cat` with `type`... no wait get-content? https://superuser.com/questions/434870/what-is-the-windows-equivalent-of-the-unix-command-cat
# run in Anaconda Powershell prompt, so I have both make and Python
conda create -n slp_to_pdf 
conda install Python
conda install -c conda-forge imageio
conda install pip
python -m pip install regex

And it fails with an error.

[cleong110]

Here's my ieee target

ieee: dst/sections tmp
	rm -rf tmp/overleaf
	git clone https://git.overleaf.com/611a535f64617c334d122e31 tmp/overleaf
	mkdir -p tmp/overleaf/parts/background
	cp -r dst/sections tmp/overleaf/parts/background
	rm -f tmp/overleaf/parts/background/sections/.DS_Store
	cp -r dst/assets tmp/overleaf/parts/background
	cp src/references.bib tmp/overleaf/parts/background
	rm -f tmp/overleaf/parts/background/assets/.DS_Store

[cleong110]

Tried a number of things, keep running into various errors. I think I will need to run it on Linux.

[cleong110]

Giving Colab a try. Had to manually install Pandoc from the .deb in order to get around an error with "unrecognized option" for
--citeproc.

Then ran into an issue where it didn't have lualatex. Giving apt install texlive-full a go. Problem is, it takes forever

[cleong110]

It finally worked! I got an index.pdf!

So make dst/index.pdf is what I want to build off of then

[cleong110]

https://pandoc.org/MANUAL.html#citation-rendering-1 is relevant

[cleong110]

https://github.com/stsewd/ieee-pandoc-template could give some insights into how this can work. Also https://github.com/stsewd/ieee-pandoc-template/blob/master/template.latex in particular might be useful.

[cleong110]

I note with interest that https://github.com/sign-language-processing/sign-language-processing.github.io/blob/master/.github/workflows/deploy.yml uses a docker container.

https://hub.docker.com/r/pandoc/latex is a possibility that might work.

[cleong110]

https://github.com/pandoc/dockerfiles

[cleong110]

Today I am pursuing two avenues:

1. Install texlive-full, pandoc, etc on my Linux Laptop.
2. pandock, aka docker container for pandoc/extra which should have everything needed. Fonts, etc.

[cleong110]

BUILDING NATIVELY WITH TEXLIVE-FULL, PANDOC ON UBUNTU

Option 1, installing texlive-full, is taking a VERY long time, it has been on this step for hours:

Meanwhile I'm pursuing pandock (see below)

After several hours, I finally Googled it, finding this:
https://askubuntu.com/questions/956006/pregenerating-context-markiv-format-this-may-take-some-time-takes-forever

Turns out you need to press enter???

Yup that fixed it!

What's up with this?

[WARNING] [makePDF] LaTeX Warning: Citation `yin-etal-2021-including' on page 5
  undefined on input line 460.

[cleong110]

BUILDING WITH PANDOCK (stalled)

OK, so we're starting with this as a basis

docker run --rm \
       --volume "$(pwd):/data" \
       --user $(id -u):$(id -g) \
       pandoc/extra README.md -o outfile.epub

And trying to combine that with

dst/index.pdf: dst/index_shortcode.md src/references.bib
	cd dst && pandoc -f markdown+emoji -L../addons/latex-emoji.lua index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=../src/references.bib --citeproc -o index.pdf

Let's just try replacing pandoc with docker run --rm --volume "$(pwd):/data" --user $(id -u):$(id -g) pandoc/extra first, replicating the original result

dst/index_pandock.pdf: dst/index_shortcode.md src/references.bib
	cd dst && docker run --rm --volume "$(pwd):/data" --user $(id -u):$(id -g) pandoc/extra -f markdown+emoji -L../addons/latex-emoji.lua index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=../src/references.bib --citeproc -o index.pdf

OK, somehow "$(pwd)" is blank. I believe that's because Make is interpreting this as trying to get a variable, it's not running the shell.

Using $(shell pwd) instead...

dst/index_pandock.pdf: dst/index_shortcode.md src/references.bib
	cd dst && docker run --rm --volume "$(shell pwd):/data" --user $(shell id -u):$(shell id -g) pandoc/extra -f markdown+emoji -L../addons/latex-emoji.lua index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=../src/references.bib --citeproc -o index.pdf

OK, getting some kind of error about index_shortcode.md not existing?

pandoc: index_shortcode.md: withBinaryFile: does not exist (No such file or directory)

Let's log into the container:

# in dst folder
# docker flags: remove afterwards, interactive, mount volume, user, entrypoint (no bash installed)
$ docker run --rm -it --volume "$(pwd):/data" --user $(id -u):$(id -g) --entrypoint /bin/sh pandoc/extra
/data $ ls
assets              index.html          index.md            index.pdf           index_shortcode.md  style.css

Edited the paths to use absolute paths inside the Docker

dst/index_pandock.pdf: dst/index_shortcode.md src/references.bib
	docker run --rm --volume "$(shell pwd):/data" --user $(shell id -u):$(shell id -g) pandoc/extra -f markdown+emoji -L/data/addons/latex-emoji.lua /data/dst/index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=/data/src/references.bib --citeproc -o index.pdf

That made progress, now we're seeing this: ! LaTeX Error: File 'pdflscape.sty' not found.

It seems it's not installed!

Making a Dockerfile?
pandoc/dockerfiles#135 says to use tlmgr

OK so it exists.

Dockerfile

FROM pandoc/extra

RUN tlmgr install pdflscape

And then edit the Makefile

# set up custom docker image based on https://hub.docker.com/r/pandoc/extra
docker:
	cd docker && docker build . -t pandoc/extra/slp

dst/index_pandock.pdf: dst/index_shortcode.md src/references.bib docker
	docker run --rm --volume "$(shell pwd):/data" --user $(shell id -u):$(shell id -g) pandoc/extra/slp -f markdown+emoji -L/data/addons/latex-emoji.lua /data/dst/index_shortcode.md -s -N --pdf-engine=lualatex --shift-heading-level-by=-1 --bibliography=/data/src/references.bib --citeproc -o index.pdf

gives us this

Perhaps referring to this? https://ctan.org/tex-archive/fonts/twemoji-colr

in which case we can add a tlmgr command to install it in the Dockerfile

Yup that worked. Now we get

! Paragraph ended before \Gin@iii was complete.

Which, according to https://tex.stackexchange.com/questions/37650/paragraph-ended-before-giniii-was-complete-while-inserting-image-with-inclu means we need to add

\usepackage{graphicx}

Added that to the top of index.md, now we get

! Package luatex.def Error: File `assets/representation/continuous.pdf' not fou
nd: using draft setting.

Which I fixed by ctrl+r in index.md, from "assets" to "src/assets", but then the normal make dst/index.pdf doesn't work.

Let's try and get in?
https://stackoverflow.com/questions/45356985/how-to-run-an-existing-stopped-container-and-get-inside-the-bash