Use processes to run literate examples #309
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…Processes/KernelFunctions.jl into tgf/use_literate_processes
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
No idea why but it seems to work. |
docs/make.jl
Outdated
| Literate.markdown("$(filepath)", "$(EXAMPLES_OUT)"; name="$(example)", documenter=true, execute=true) | ||
| """ | ||
| run( | ||
| addenv( |
There was a problem hiding this comment.
Maybe withenv would be a bit more idiomatic?
| @@ -17,44 +17,35 @@ const BLACKLIST = ["deep-kernel-learning", "support-vector-machine"] | |||
| ispath(EXAMPLES_OUT) && rm(EXAMPLES_OUT; recursive=true) | |||
| mkpath(EXAMPLES_OUT) | |||
|
|
|||
| # preprocessor for Literate example scripts: | |||
| # - add Documenter @setup snippet that activates each example's own project environment | |||
| function preprocess(content) | |||
There was a problem hiding this comment.
I guess it would still be useful to preprocess the examples in a similar way as done in https://github.com/devmotion/CalibrationErrors.jl/blob/c73b2abe19b349586ca1448b55c7e697b814c03c/docs/make.jl#L19-L34 and https://github.com/JuliaGaussianProcesses/AbstractGPs.jl/blob/9b14c286e5885c2e94edf5bc3502347126b23276/docs/make.jl#L16-L23, such that the documentation links to the notebook and vice versa.
There was a problem hiding this comment.
This can eventually be added later. I think the main point right now is to have examples running
docs/make.jl
Outdated
| using Literate | ||
| Literate.markdown("$(filepath)", "$(EXAMPLES_OUT)"; name="$(example)", documenter=true, execute=true) | ||
| """ | ||
| run( |
There was a problem hiding this comment.
Can we parallelize this such that examples are executed in parallel?
There was a problem hiding this comment.
I guess we could! Does GH Actions allows this? I could not find anything in the docs
There was a problem hiding this comment.
I thought we could just use Julia's parallelization capabilities by e.g. spawning multiple processes in parallel. It seems a bit tricky to use GH actions for it since we would have to collect all outputs and only update the gh-pages branch once after everything was run and the documentation was built.
| end | ||
|
|
||
| cmd = Base.julia_cmd() | ||
| docs_env = @__DIR__ |
There was a problem hiding this comment.
Do we actually need to inherit anything? Maybe it would be cleaner to just install Literate manually (or even better, add it to the dependencies of the example's Project.toml such that it is pinned and tracked as well) and not inherit anything.
Then one could even include the notebook run in the external process as well.
There was a problem hiding this comment.
To be honest I don't fully understand what this does... I just copied the example from https://discourse.julialang.org/t/multiple-environments-in-documenter/63436/4
There was a problem hiding this comment.
It builds a stacked environment and adds both the packages in docs/ and examples/current_example/ to the load path (the latter are prioritized). However, I don't think we have to inherit anything if we add the Literate dependency to examples/current_example/. Then we could just run the external process, instantiate the example environment (without any additional special stuff), and generate both the Markdown and the notebook.
docs/make.jl
Outdated
| Dict("JULIA_LOAD_PATH" => load_path), | ||
| ), | ||
| ) | ||
| Literate.notebook(filepath, EXAMPLES_OUT; name=example, documenter=true, execute=false) |
There was a problem hiding this comment.
Maybe just include this in the external process as well? (see https://github.com/JuliaGaussianProcesses/KernelFunctions.jl/pull/309/files#r658620202).
| withenv("JULIA_LOAD_PATH" => load_path) do | ||
| run(`$(cmd) --project=$(exampledir) -e $(code)`) | ||
| end |
There was a problem hiding this comment.
Without stacked environments this is just
| withenv("JULIA_LOAD_PATH" => load_path) do | |
| run(`$(cmd) --project=$(exampledir) -e $(code)`) | |
| end | |
| run(`$(cmd) --project=$(exampledir) -e $(code)`) |
There was a problem hiding this comment.
Please feel free to apply the correct edits, I am moving in the dark here 😅
|
This has been superseded by #314 - thanks for getting it started! |
This PR is aiming as using difference processes for each markdown file in #303
I created a new branch in case everything goes to 💩...