Adds `rmarkdown` module #944

averissimo · 2025-11-05T17:42:45Z

Pull Request

Fixes https://github.com/insightsengineering/coredev-tasks/issues/675

Thanks to @StefanThoma's for working on the first prototype

Changes description

Adds rmarkdown module
Renders the output to module UI and to Reporter
Source code is not verified as it adds a comment on how to render it
Adds download button to get the original Rmd file
- Option in module to enable or disable this functionality
- downloaded Rmd has module's data included (data pre-processing + filter)
Uses extra_transform parameter to add "generic decorators" to the top of the UI
- name is temporary
Ensure figures are displayed properly in reporter (PDF output)
Avoid encoding images to base64 (mayb keep it in raw or equivalent)

To discuss

Should we render the .R to get all the R code from the report and add it to the code of the qenv?
[limitation] Use of markdown when rendering doesn't print title nor author
- Rendering it on the page breaks teal
- Wrapping in iframe doesn't extend full height and then creates strange reports
Should we use transformers to add extra input / data manipulation
- Or keep a parameters that allows to add transforms (for example extra inputs) to the module by the app developer

(topics below added from conversation with @gogonzo )

should teal_report contain markdown elements?
should teal_data evaluate rmarkdown::render() and include this to @code
should markdown be evaluated with rmarkdown::render(envir = <qenv>) instead of params = list()?
- visibility concern here on developing the Rmd

App

pkgload::load_all("../teal.modules.general")

data <- teal_data()
data <- within(data, {
  CO2 <- CO2
  CO2[["primary_key"]] <- seq_len(nrow(CO2))
})
join_keys(data) <- join_keys(join_key("CO2", "CO2", "primary_key"))

static_decorator <- teal_transform_module(
  label = "N Cols selector",
  ui = function(id) {
    ns <- NS(id)
    tags$div(
      numericInput(ns("n_rows"), "Show n rows", value = 5, min = 0, max = 200, step = 5)
    )
  },
  server = function(id, data) {
    moduleServer(id, function(input, output, session) {
      reactive({
        req(data())
        within(data(), {
          n_rows <- n_rows_value
        }, n_rows_value = input$n_rows)
      })
    })
  }
)

app <- init(
  data = data,
  modules = modules(
    tm_rmarkdown(
      label = "RMarkdown Module",
      rmd_file = system.file(file.path("sample_files", "test.Rmd"), package = "teal.modules.general"),
      allow_download = FALSE,
      extra_transform = list(static_decorator)
    )
  )
) |> shiny::runApp()

averissimo · 2025-11-06T13:02:14Z

Marked as ready to review to start the discussion

github-actions · 2025-11-06T13:16:22Z

Code Coverage Summary

Filename                      Stmts    Miss  Cover    Missing
--------------------------  -------  ------  -------  -------------------------------------------
R/tm_a_pca.R                    864     864  0.00%    141-1132
R/tm_a_regression.R             751     751  0.00%    180-1027
R/tm_data_table.R               201     201  0.00%    100-348
R/tm_file_viewer.R              172     172  0.00%    47-254
R/tm_front_page.R               143     132  7.69%    77-246
R/tm_g_association.R            320     320  0.00%    161-547
R/tm_g_bivariate.R              664     406  38.86%   332-786, 827, 932, 949, 967, 978-1000
R/tm_g_distribution.R          1106    1106  0.00%    156-1404
R/tm_g_response.R               345     345  0.00%    179-594
R/tm_g_scatterplot.R            709     709  0.00%    261-1065
R/tm_g_scatterplotmatrix.R      272     253  6.99%    200-501, 562, 576
R/tm_missing_data.R            1080    1080  0.00%    124-1380
R/tm_outliers.R                1029    1029  0.00%    162-1341
R/tm_rmarkdown.R                171     171  0.00%    111-332
R/tm_t_crosstable.R             263     263  0.00%    177-482
R/tm_variable_browser.R         887     881  0.68%    89-1113, 1164-1347
R/utils.R                       185     120  35.14%   87-249, 278-304, 316-325, 330, 344-363, 452
R/zzz.R                           2       2  0.00%    2-3
TOTAL                          9164    8805  3.92%

Diff against main

Filename            Stmts    Miss  Cover
----------------  -------  ------  --------
R/tm_rmarkdown.R     +171    +171  +100.00%
TOTAL                +171    +171  -0.07%

Results for commit: d05971a

Minimum allowed coverage is 80%

♻️ This comment has been updated with latest results

gogonzo

Shouldn't both documets be reproducible?

document downloaded from tm_rmarkdown
document downloaded via reporter

gogonzo · 2025-11-06T12:47:23Z

R/tm_rmarkdown.R

+          params = datasets[["rmd_data"]],
+          envir = new.env(parent = globalenv()),


Passing data through params breaks reproducibility. In Rmd one has to use params$<dataset name>. I think data should be passed via envir.

Suggested change

params = datasets[["rmd_data"]],

envir = new.env(parent = globalenv()),

envir = q_r(),

I'm fine with this, it's a bit more custom that using params, but it would be cleaner.

The only reason to use params would be to use existing features of R Markdown, instead of custom workflow

That is, if data is passed via envir, what would be the ideal workflow for the statistician to develop the original R Markdown file?

1. Using envir = data() in rmarkdown::render()

Statistician will have a code chunk that generates whatever data they require

Removes chunk:

Manually before sending it or use chunk opts to hide it {r eval = FALSE, include = FALSE}

We provide a flag in envir?

{r eval = !(exists(".teal_takes_care_data") && isTRUE(.teal_takes_care_data)), include = !(exists(".teal_takes_care_data") && isTRUE(.teal_takes_care_data))}

Send to app developer

App developer uses it module

2. Using params = as.list(data()) in rmarkdown::render()

As it is right now as it has an override mechanism

--- # ... params: ADSL: !r teal.data::rADSL # ... ---

Statistician develops Rmd with params set to whatever data they use

Send to app developer

App developer uses it on module

3. Other?

?

That is, if data is passed via envir, what would be the ideal workflow for the statistician to develop the original R Markdown file?

Maybe sort of a debug-mode where data() is dumped into environment and app-developer can just execute code-chunks one by one.

R/tm_rmarkdown.R

gogonzo · 2025-11-06T13:19:14Z

R/tm_rmarkdown.R

+  moduleServer(id, function(input, output, session) {
+    if (allow_download) {
+      output$download_rmd <- downloadHandler(
+        filename = function() basename(rmd_file),


Shouldn't downloadable rmarkdown contain also "reproducible" qenv code? IMO tm_rmarkdown(rmd_file) is just a template and returning a template doesn't make sense if it lacks data to actually execute templete-file.

Edit:

I think app user should be able to download rendered document?

Shouldn't downloadable rmarkdown contain also "reproducible" qenv code? IMO tm_rmarkdown(rmd_file) is just a template and returning a template doesn't make sense if it lacks data to actually execute templete-file.

Good point. I didn't want to do that before the rendering as it mean that the pre-processing would be run again.

But this can be added in post-rendering to the Download document and in the reporter markdown version.

This does imply we need to parse the Rmd and resulting markdown:

Read the md

Find line after yml header -- fallback to start of document if no header present

Add new R code chunk with contents of qenv

(we could even inject this in the module UI as well to keep consistency)

I think app user should be able to download rendered document?

They can, when downloading the reporter. I don't think we should overlap functionality here.

See how to convert .Rmd to .R and back using knitr (thats' in teal.reporter dependency)

@llrs-roche parsing and adding the section in .R or in .Rmd would have similar challenges?

From my initial findings knitr::purl() is promising, but it does have some minor limitations with chunk options (it doesn't play well when evaluating params: {r, eval = !is.null(params$CO2)})

R/tm_rmarkdown.R

gogonzo · 2025-11-06T14:37:52Z

R/tm_rmarkdown.R

+
+      teal.reporter::teal_card(out_data) <- c(
+        teal.reporter::teal_card(out_data),
+        rendered_html_r()


markdown fits here better. We can also split it into list() suitable with teal_card()

I'm not sure what would be better as I think of this as a monolith element. At least until we can actually break down the executable code.

We could keep it as a single markdown object (i.e. character vector) and only render as needed. note: the current approach was meant to reduce duplicate rendering (1. Module UI; 2. Reporter previewer; 3. Reporter download).

Splitting might be difficult to achieve as there are multiline grammar and context in markdown (for instance code chunks, or just newlines without a new paragraph in text)

toHTML(c("# ada", "", "bla.", "ca.", "```r", "1+1", "3*3", "```", "some text")) # works # vs. lapply(c("# ada", "", "bla.", "ca.", "```r", "1+1", "3*3", "```", "some text"), toHTML) # doesn't work

We could keep it as a single markdown object

I think its good to have a single markdown string because anyway at the end (before download) we will concatenate everything to the single file. Only editor will show one combined text-field for this element (i think it is not a big issue so far)

the current approach was meant to reduce duplicate rendering

This is a fair reason. I feel (not a strong opinion) that if you render markdown using rmarkdown::render and includeMarkdown the speed should be similar to rmarkdown::render to html. rmarkdown renders in two stages to html, first by converting Rmd -> md and then pandoc renders from md -> html.

The biggest issue with including html is that html can't be converted to pdf, otf etc.

@gogonzo see new commits.

Uses qenv environment instead of params

Downloaded Rmd from main module adds data processing in module

Markdown is rendered once

HTML is rendered only once on top of markdown

Uses custom to_rmd and toHTML to achieve goals

Let me know if you have comments :-)

gogonzo · 2025-11-06T14:51:16Z

Key questions:

What app-user should be able to download in tm_markdown

Should this document be reproducible and contain pre-processing, filter-panel and transformator code? Should this document contain outputs?
I don't think that app-user should receive the same file as app-developer provides do the tm_markdown. app-developer uses rmd file just as a template to handle data, produce outputs (plots and tables) and produce final document by specifying document: html_document/md_document/etc.. App-user thought receives document which content is controlled by app-developer - this could be markdown file containing/not containing code-chunks, html-file, pdf file etc.

What should be added to the teal_card (reporter)

html/md content as single teal_card element?
md content converted to teal_card elements (multiple elements)?
rmarkdown::render() call (this is a joke - markdown::render inside of markdown file would be crazy)
nothing

StefanThoma · 2025-11-07T07:41:49Z

Would it be possible to use the reporter function to download a rendered document? Ideally the user would have the choice between html and pdf. What do you think?

…

On Fri, Nov 7, 2025 at 8:38 AM Dawid Kałędkowski ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In R/tm_rmarkdown.R <#944 (comment)> : > + + if (allow_download) { + out_data <- eval_code( + q_r(), + paste( + sep = "\n", + sprintf("## R Markdown contents are generated from file, please download it from the module UI."), + sprintf("# rmarkdown::render(%s, params = params)", shQuote(basename(rmd_file), type = "cmd")) + ) + ) + ***@***.*** <- FALSE # manual change verified status as code is being injected + } + + teal.reporter::teal_card(out_data) <- c( + teal.reporter::teal_card(out_data), + rendered_html_r() We could keep it as a single markdown object I think its good to have a single markdown string because anyway at the end (before download) we will concatenate everything to the single file. Only editor will show one combined text-field for this element (i think it is not a big issue so far) the current approach was meant to reduce duplicate rendering This is a fair reason. I feel (not a strong opinion) that if you render markdown using rmarkdown::render and includeMarkdown the speed should be similar to rmarkdown::render to html. rmarkdown renders in two stages to html, first by converting Rmd -> md and then pandoc renders from md -> html. The biggest issue with including html is that html can't be converted to pdf, otf etc. — Reply to this email directly, view it on GitHub <#944 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJUWWERCKCNM7NA66YG6VVT33REA3AVCNFSM6AAAAACLHVPL3CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTIMZSGA3TSNRYGU> . You are receiving this because you were mentioned.Message ID: <insightsengineering/teal.modules.general/pull/944/review/3432079685@ github.com>

averissimo · 2025-11-07T12:00:50Z

@StefanThoma that's one of our goals with this PR. It's now possible for HTML output, the other formats are a WIP, but I believe we'll make it happen

averissimo · 2025-11-07T14:34:05Z

R/tm_rmarkdown.R

+        out_data@verified <- FALSE # manual change verified status as code is being injected
+      }
+
+      teal.reporter::teal_card(out_data) <- c(


Issue with keeping it in markdown format: Images are rendered to file that needs to be "kept" until report rendering

Possible workarounds:

Modify generated md file to convert image from paths to base64

Problem: there's an issue in generating pdf: Base64 image not appearing in pdf_document rstudio/rmarkdown#2604

Fix idea (without rmarkdown release): convert the markdown image tag to a code chunk that includes graphic

Keep the images until the report generation

Re-render during report generation

This has been solved in c563c71 by creating a custom reporter object for this module.

It keeps the rendered HTML and uses it as caching (only renders once)

It also keeps the contents of images in memory to be dumped and used later on.

Why memory?

In case files on disk change (multiple reports being added)

Prevent having to track and keep all images being generated

There's still room for improvement here, but we have a working solution.

Improvement:

rmarkdown fixes issue raised above (and we manipulate image tag to use base64)

Avoid encoding to base64 keep it in raw format for example.

averissimo added 5 commits November 5, 2025 17:35

feat: adds tm_rmarkdown module

d082c16

docs: update roxygen

6d064e1

docs: update roxygen

2c6b90e

chore: fix some comments

b326567

fix: minor typo

2e690ff

averissimo added the core label Nov 5, 2025

chore: enforce srv protection against allow_download = FALSE

2b17cde

gogonzo self-assigned this Nov 6, 2025

averissimo added 2 commits November 6, 2025 12:56

feat: add extra inputs

157c194

fix: minor fix in names

a279ec0

averissimo marked this pull request as ready for review November 6, 2025 12:59

[skip style] [skip vbump] Restyle files

611acf5

averissimo added 2 commits November 6, 2025 13:03

chore: remove extra parameter

1fa0d8b

docs: update documentation

309c5ac

gogonzo reviewed Nov 6, 2025

View reviewed changes

R/tm_rmarkdown.R Outdated Show resolved Hide resolved

gogonzo reviewed Nov 6, 2025

View reviewed changes

R/tm_rmarkdown.R Outdated Show resolved Hide resolved

averissimo added 2 commits November 6, 2025 13:37

pr: copy rmd and setup tempdir only once

3640c41

chore: use the output of render (file path) instead of explicit name

3b0c5d4

gogonzo reviewed Nov 6, 2025

View reviewed changes

R/tm_rmarkdown.R Show resolved Hide resolved

pr: rename rmd_data to params for consistency

14cdf42

gogonzo reviewed Nov 6, 2025

View reviewed changes

averissimo commented Nov 7, 2025

View reviewed changes

averissimo and others added 2 commits November 7, 2025 18:06

pr: address main points of review from @gogonzo

c563c71

[skip roxygen] [skip vbump] Roxygen Man Pages Auto Update

8c1acaa

fix: adds baseenc for now in description

d05971a

		params = datasets[["rmd_data"]],
		envir = new.env(parent = globalenv()),

	params = datasets[["rmd_data"]],
	envir = new.env(parent = globalenv()),
	envir = q_r(),

Uh oh!

Adds rmarkdown module #944

Are you sure you want to change the base?

Adds rmarkdown module #944

Uh oh!

Conversation

averissimo commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Changes description

To discuss

App

Uh oh!

averissimo commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Summary

Diff against main

Uh oh!

gogonzo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1. Using envir = data() in rmarkdown::render()

2. Using params = as.list(data()) in rmarkdown::render()

3. Other?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gogonzo Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

averissimo Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

averissimo Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gogonzo commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanThoma commented Nov 7, 2025 via email

Uh oh!

averissimo commented Nov 7, 2025

Uh oh!

averissimo Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

averissimo Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Adds `rmarkdown` module #944

Adds `rmarkdown` module #944

averissimo commented Nov 5, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading

1. Using `envir = data()` in `rmarkdown::render()`

2. Using `params = as.list(data())` in `rmarkdown::render()`

gogonzo Nov 6, 2025 •

edited

Loading

averissimo Nov 7, 2025 •

edited

Loading

averissimo Nov 7, 2025 •

edited

Loading

gogonzo commented Nov 6, 2025 •

edited

Loading

averissimo Nov 7, 2025 •

edited

Loading

averissimo Nov 7, 2025 •

edited

Loading