????: Orchestration infrastructure for fully automated dashboards

[Bisaloo]

Description

There is high appetite among policy makers and the general public for regularly-updated dashboards allowing them to follow the latest trends in an ongoing epidemic.

While the underlying epidemiological analysis can be done with existing tools (e.g., incidence2 for epicurves, EpiNow2 for Rt estimation and short-term forecasts, etc.), the actual setup and automation of the dashboard remains challenging and often result in ad-hoc solutions, which rely on a specific team member or key partner.

Indeed, epidemiologists and even R developers are rarely trained to do DevOps work:

• Orchestration via GitHub Actions or any other scheduling mechanism
• Communication with a cloud computing backend for computationally expensive steps
• Use of efficient and performant storage solutions (in terms of actual format but also location. E.g., use of an S3 bucket)
• Continuous deployment

Additionally, and as is often the case for DevOps work, it is possible that some of the existing infrastructure used by certain groups is not open-source and therefore impossible for others to replicate.

This package would use existing pipelines, possibly from the episoap package, to create the dashboard, and handle the orchestration behind the scenes by creating necessary config files, secrets, etc. based on a single configuration file.

Typical end-users

Epidemiologists or modellers who want to produce a continuously updated dashboard of their analysis

Potential contributors

??

Key collaborators

@Bisaloo, @sbfnk

Inputs

NA

Outputs

NA

Imports

• episoap

However, if we want this package to be useful outside the Epiverse-TRACE core team, we shouldn't make it completely linked to episoap. Users should be able to provide their own custom pipelines.

Used by

??

Related projects

• The European Covid-19 Forecast Hub website. Code:

  • pipelines
  • orchestration: 1, 2, 3

• The Covid-19 Rt estimate dashboard. Code:

  • orchestration

Usage

The source repo of the dashboard would contain 3 branches

flowchart LR
    `main` -- automatically generated by this package --> `infra`
    `infra` -- automatically generated by this package --> `gh-pages`

Loading

Additional comments

No response

[Bisaloo]

If we were to pursue this, we should probably llok at https://whipson.github.io/maestro/.