🔬 Bioinformatics Project Manager (BPM)

Bioinformatics research is diverse: every dataset has quirks, and labs juggle a mix of scripts, pipelines, and frameworks. This flexibility is powerful, but it also creates problems:

• Projects are hard to reproduce.
• Scripts get copied, modified, and lost.
• Sharing across people or facilities means reinventing the same work.

BPM (Bioinformatics Project Manager) was created to solve this gap.

✅ What BPM Is

BPM is a lightweight, Python-based command-line tool that provides a management layer for bioinformatics projects. It brings order and reusability without forcing you into a single framework.

At its core:

• BPM = the engine (stable CLI for project and template management).
• BRS = Bioinformatics Resource Store (repositories of templates, workflows, hooks, and resolvers customized for your facility or personal work).

🚫 What BPM Is Not

• Not a workflow engine (Nextflow, Snakemake, Cromwell). It doesn’t replace them — it wraps and organizes them.
• Not a LIMS. It doesn’t manage samples, machines, or lab metadata — it focuses on analysis.
• Not a cloud service or central registry. State lives in plain files (project.yaml, stores.yaml) under your control.

Instead, BPM complements these tools:

• Organize projects with BPM, then run Nextflow/Snakemake inside BPM templates.
• Keep facility-specific environments, scripts, and settings in a BRS for reuse.
• Still log into a LIMS or DB if you want — BPM keeps the analysis side reproducible.

Why It’s Useful

• Reusability: Share and rerun templates across datasets with one command.
• Consistency: Naming policy and status tracking simplify archiving and collaboration.
• Flexibility: Each group can maintain its own BRS — no central server required.
• Transparency: Everything in YAML; works with version control by default.
• Lifecycle management: Track project and template states automatically.
• Hooks & resolvers: Automate environment-specific paths and post-processing.
• Ad‑hoc mode: Run templates outside BPM projects when you just need scripts.

In short: BPM doesn’t replace your workflow engine, pipelines, or LIMS. It sits one layer above them, helping you organize, reuse, and share your bioinformatics projects in a clean and reproducible way.

Quickstart

pixi install
pixi run test   # run all tests
pixi run lint   # ruff
pixi run fmt    # black

CLI overview

• bpm resource …: manage BRS stores (add/activate/remove/list/info)
• bpm project …: init/info/status for a project directory
• bpm template …: render/run/publish templates from the active BRS
• bpm workflow …: run workflows from the active BRS

---

Concepts

Bioinformatics Resource Store (BRS)

A BRS is a folder containing the reusable building blocks for your org or personal work:

• config/ — authors, hosts, and settings (e.g., project name policy)
• templates/ — reusable analysis blueprints
• workflows/ — higher-level wrappers similar to templates
• hooks/ — Python hook functions (pre/post render/run)
• resolvers/ — publish resolvers to compute structured outputs

See the minimal example in tests/data/brs_min/.

Templates

Each template lives in templates/<id>/ and is described by template.config.yaml.

Key fields:

• id: Template id (must match folder name).
• params: Map of parameters (type, required, default, optional cli alias).
• render.into: Where to render (supports ${ctx.*} placeholders).
• render.files: List of mappings (e.g., a.j2 -> a) — *.j2 renders with Jinja2, others copied.
• run.entry: Optional script to execute (e.g., run.sh); BPM marks it executable.
• required_templates: Dependencies that must already exist in the project.
• publish: Resolvers to compute structured values after run.
• hooks: post_render, pre_run, post_run lists of dotted hook functions.

Parameter precedence when rendering:

1. descriptor defaults < 2) project‑stored values < 3) CLI --param overrides

Jinja has access to a rich context via ctx (see Context System below).

Workflows

Workflows live under workflows/<id>/ with workflow_config.yaml. They execute entry scripts from the workflow folder and can use ${ctx…} placeholders in args/env. When a project is provided, BPM can record workflow run history in project.yaml. They are useful for one‑off utilities and glue tasks.

---

Modes: Project vs Ad‑hoc

Project Mode (with Context System)

Create a project directory and a project.yaml with:

• name, created, project_path (host‑aware string like nextgen:/projects/NAME)
• authors (expanded from config/authors.yaml)
• status (initiated → active, etc.)
• templates (list of rendered templates, params, and statuses)

The context object ctx passed to templates/hooks/resolvers contains:

• ctx.project: { name, project_path }
• ctx.template: { id, published }
• ctx.params: final resolved params
• ctx.brs: { repo, authors, hosts, settings }
• ctx.cwd: Path used as the base for rendering/running
• Helpers: ctx.hostname(), ctx.materialize(hostpath), ctx.now()

Project‑mode rendering updates project.yaml and sets:

• template entry: status → active, params → final
• project status: active

Running a template marks it completed. Publish persists resolver outputs.

CLI parameters (project mode):

# Create a project (policy enforced by active BRS settings)
bpm project init <project_name> \
  --project-path <host:path> \
  [--author <id1,id2>] \
  [--cwd <dir>]

# Inspect
bpm project info   --dir <project_dir>
bpm project status --dir <project_dir>

# Render / run / publish
bpm template render  <template_id> --dir <project_dir> [--param KEY=VALUE] [--dry]
bpm template run     <template_id> --dir <project_dir>
bpm template publish <template_id> --dir <project_dir>

Ad‑hoc Mode (no project.yaml)

Render a template directly to an output folder without changing a project:

• Skips dependency checks and hooks
• Does not read or write project.yaml
• Overrides render.into to . so files materialize under the output folder
• Writes bpm.meta.yaml with source metadata and final params

CLI parameters (ad‑hoc):

bpm template render <template_id> \
  --out <output_dir> \
  [--param KEY=VALUE] \
  [--dry]

Shell completion

Enable shell completion for the bpm command.

• Zsh: echo 'eval "$( _BPM_COMPLETE=zsh_source bpm )"' >> ~/.zshrc && exec zsh
• Bash: echo 'eval "$( _BPM_COMPLETE=bash_source bpm )"' >> ~/.bashrc && exec bash
• Fish: echo 'eval (env _BPM_COMPLETE=fish_source bpm)' >> ~/.config/fish/config.fish && exec fish

Or use Typer helper once completion is enabled: bpm --install-completion.

---

Installation

Pick one of pip, pixi, or conda.

• Pip (user install):

  python -m pip install bpm-cli
  # or in editable mode if working on sources
  python -m pip install -e .

• Pixi (development and testing):

  pixi install        # creates an environment with runtime deps
  pixi run python -m pip install bpm-cli
  pixi run test       # run tests
  pixi run lint       # ruff
  pixi run fmt        # black

• Conda/Mamba (environment for running bpm):

  mamba create -n bpm -c conda-forge python>=3.10 typer jinja2 pyyaml rich
  mamba activate bpm
  python -m pip install bpm-cli  # or: python -m pip install -e .

Notes

• Python 3.10+ is required.
• Rich is included for nicer table output; if missing, BPM falls back to plain text.

Editable install with pip:

Releasing to PyPI (maintainers)

• The GitHub Actions workflow at .github/workflows/publish.yml builds sdist/wheels and publishes on GitHub Releases.
• Pre-releases (marked as pre-release) are published to TestPyPI using the TEST_PYPI_API_TOKEN secret.
• Final releases are published to PyPI using the PYPI_API_TOKEN secret.
• Tag format must be vX.Y.Z and match the version in bpm/_version.py.
• Steps to release:
  1. Update bpm/_version.py to the new version.
  2. Create a GitHub Release with tag vX.Y.Z (check or uncheck “This is a pre-release” accordingly).
  3. The workflow builds and publishes automatically.

python -m pip install -e .[dev]
pytest -q

---

Links

• Roadmap: roadmaps.md
• Structure & design notes: structure.md

Resource

# add a local BRS and activate it
bpm resource add /path/to/brs --activate

# list stores (active marked with *)
bpm resource list

# switch active
bpm resource activate <id>

Project

# create a new project (policy enforced by active BRS)
bpm project init 250901_Demo_UKA \
  --outdir /tmp \
  [--author ckuo,lgan] \
  [--host nextgen]

# inspect
bpm project info --dir /tmp/250901_Demo_UKA
bpm project status --dir /tmp/250901_Demo_UKA

Template

# render into project with params
bpm template render hello --dir /tmp/250901_Demo_UKA --param name=Alice

# run (executes run.sh in the rendered folder, with hooks)
bpm template run hello --dir /tmp/250901_Demo_UKA

# publish (runs resolvers and persists to project.yaml)
bpm template publish hello --dir /tmp/250901_Demo_UKA

# ad‑hoc render (no project.yaml changes); writes bpm.meta.yaml in output
bpm template render hello --out /tmp/adhoc_out --param name=Alice
# or derive the ad-hoc output via a template resolver with --adhoc
bpm template render demux_bclconvert --adhoc --param bcl_dir=/data/runs/FC001

Workflow

Workflows execute entry scripts from the BRS and optionally record run history in project.yaml.

# run a workflow (optional project context)
bpm workflow run clean --project /tmp/250901_Demo_UKA/project.yaml --name Alice

See example skeleton under tests/data/brs_min and tests/cli/test_workflow_run.py.