LLNL · bgunnar5 · Apr 25, 2025 · Oct 4, 2024 · Oct 17, 2024 · Oct 17, 2024
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -3,7 +3,7 @@ version: 2
 build:
   os: "ubuntu-20.04"
   tools:
-    python: "3.8"
+    python: "3.13"
 
 python:
   install:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+- API documentation for Merlin's core codebase
 - Added support for Python 3.12 and 3.13
 - Added additional tests for the `merlin run` and `merlin purge` commands
 - Aliased types to represent different types of pytest fixtures

diff --git a/docs/README.md b/docs/README.md
@@ -41,4 +41,84 @@ MkDocs relies on an `mkdocs.yml` file for almost everything to do with configura
 
 ## How Do API Docs Work?
 
-Coming soon...
+The API documentation in this project is automatically generated using a combination of MkDocs plugins and a custom Python script. This ensures that the documentation stays up-to-date with your codebase and provides a structured reference for all Python modules, classes, and functions in the merlin directory.
+
+This section will discuss:
+
+- [Code Reference Generation](#code-reference-generation)
+    - [How the Script Works](#how-the-script-works)
+- [Viewing the API Docs](#viewing-the-api-docs)
+- [Keeping API Docs Up-to-Date](#keeping-api-docs-up-to-date)
+- [Example Docstring](#example-docstring)
+- [Plugins Involved](#plugins-involved)
+
+### Code Reference Generation
+
+The script `docs/gen_ref_pages.py` is responsible for generating the API reference pages. It scans the `merlin` directory for Python files and creates Markdown files for each module. These Markdown files are then included in the `api_reference` section of the documentation.
+
+#### How the Script Works
+
+1. File Scanning:
+
+    The script recursively scans all Python files in the `merlin` directory using `Path.rglob("*.py")`.
+
+2. Ignore Patterns:
+
+    Certain files and directories are excluded from the API docs based on the `IGNORE_PATTERNS` list. For example:
+
+    - `merlin/examples/workflows`
+    - `merlin/examples/dev_workflows`
+    - `merlin/data`
+    - Files like `ascii_art.py`
+
+    The `should_ignore()` function checks each file against these patterns and skips them if they match.
+
+3. Markdown File Creation:
+
+    For each valid Python file:
+
+    - The script determines the module path (e.g., merlin.module_name) and the corresponding Markdown file path.
+    - Special cases like `__init__.py` are handled by renaming the generated file to index.md for better navigation.
+    - Files like `__main__.py` are ignored entirely.
+
+    The script then writes the mkdocstrings syntax (::: module_name) into the Markdown file, which tells the mkdocstrings plugin to generate the documentation for that module.
+
+4. Navigation File:
+
+    The script builds a navigation structure using the `mkdocs_gen_files.Nav` class. This structure is saved into a `SUMMARY.md` file, which is used by the `literate-nav` plugin to define the navigation for the API reference section.
+
+### Viewing the API Docs
+
+Once the script generates the Markdown files, they are included in the documentation site under the `api_reference` section. You can explore the API docs in the navigation bar under `Reference Guide` -> `API Reference`, with the navigation organized based on the module hierarchy.
+
+### Keeping API Docs Up-to-Date
+
+To ensure the API documentation remains accurate:
+
+Update the docstrings in Merlin's code whenever changes are made to functions, classes, or modules. The `docs/gen_ref_pages.py` file will run automatically when docs are created (i.e. when you run `mkdocs serve`).
+
+### Example Docstring
+
+The API documentation relies on properly formatted docstrings. Here’s an example using the Google-style docstring format:
+
+```python
+def add_numbers(a: int, b: int) -> int:
+    """
+    Adds two numbers.
+
+    Args:
+        a (int): The first number.
+        b (int): The second number.
+
+    Returns:
+        The sum of the two numbers.
+    """
+```
+
+### Plugins Involved
+
+Several MkDocs plugins work together to generate and display the API documentation:
+
+- `mkdocstrings`: Parses Python docstrings and generates the actual API content.
+- `mkdocs-gen-files`: Handles the creation of Markdown files and navigation structure.
+- `literate-nav`: Uses the `SUMMARY.md` file to organize the API reference section in the documentation sidebar.
diff --git a/docs/api_reference/index.md b/docs/api_reference/index.md
@@ -1,5 +1,3 @@
 # Merlin API Reference
 
-Coming soon!
-
-<!-- Welcome to the Application Program Interface (API) Reference Guide for Merlin! This comprehensive guide is designed to provide developers with a detailed understanding of the various modules, classes, and functions available within the Merlin API. -->
+Welcome to the Application Program Interface (API) Reference Guide for Merlin! This comprehensive guide is designed to provide developers with a detailed understanding of the various modules, classes, and functions available within the Merlin API.
diff --git a/docs/faq.md b/docs/faq.md
@@ -245,7 +245,7 @@ See the docs on all [Merlin Commands](./user_guide/command_line.md) that are ava
 
 ### How do I set up a workspace without executing step scripts?
 
-Use [Merlin's Dry Run](./user_guide/command_line.md#dry-run) capability:
+Use [Merlin's Dry Run](./user_guide/running_studies.md#dry-runs) capability:
 
 === "Locally"
 
@@ -438,7 +438,7 @@ run:
     procs: 3
 ```
 
-See [The `LAUNCHER` and `VLAUNCHER` Variables](./user_guide/variables.md#the-launcher-and-vlauncher-variables) and the [Scheduler Specific Properties](./user_guide/specification.md#scheduler-specific-properties) sections for more information.
+See [The `LAUNCHER` and `VLAUNCHER` Variables](./user_guide/variables.md#the-launcher-and-vlauncher-variables) and [The Run Property](./user_guide/specification.md#the-run-property) sections for more information.
 
 ### What is `level_max_dirs`?
 

diff --git a/docs/gen_ref_pages.py b/docs/gen_ref_pages.py
@@ -6,10 +6,27 @@
 
 nav = mkdocs_gen_files.Nav()
 
-# print(sorted(Path("merlin").rglob("*.py")))
+IGNORE_PATTERNS = [
+    Path("merlin/examples/workflows"),
+    Path("merlin/examples/dev_workflows"),
+    Path("merlin/data"),
+    "*/ascii_art.py",
+]
+
+
+def should_ignore(path):
+    """Check if the given path matches any ignore patterns."""
+    for pattern in IGNORE_PATTERNS:
+        # if Path(pattern).is_relative_to(path):
+        if path.is_relative_to(Path(pattern)):
+            return True
+        if path.match(pattern):
+            return True
+    return False
+
 
 for path in sorted(Path("merlin").rglob("*.py")):
-    if "merlin/examples" in str(path):
+    if should_ignore(path):
         continue
     module_path = path.relative_to("merlin").with_suffix("")
     doc_path = path.relative_to("merlin").with_suffix(".md")

diff --git a/docs/index.md b/docs/index.md
@@ -71,7 +71,7 @@ First, let's create a folder to store our server files and our examples. We'll a
 mkdir merlin_examples ; cd merlin_examples/
 ```
 
-Now let's set up a [containerized server](./user_guide/configuration/merlin_server.md) that Merlin can connect to.
+Now let's set up a [containerized server](./user_guide/configuration/containerized_server.md) that Merlin can connect to.
 
 1. Initialize the server files:
 

diff --git a/docs/user_guide/command_line.md b/docs/user_guide/command_line.md
@@ -95,7 +95,7 @@ Create a local containerized server for Merlin to connect to. Merlin server crea
 
 Merlin server has a list of commands for interacting with the broker and results server. These commands allow the user to manage and monitor the exisiting server and create instances of servers if needed.
 
-More information on configuring with Merlin server can be found at the [Merlin Server Configuration](./configuration/merlin_server.md) page.
+More information on configuring with Merlin server can be found at the [Containerized Server Configuration](./configuration/containerized_server.md) page.
 
 **Usage:**
 

diff --git a/docs/user_guide/configuration/containerized_server.md b/docs/user_guide/configuration/containerized_server.md
@@ -616,6 +616,6 @@ You can check that everything ran properly with:
 merlin status hello_samples.yaml
 ```
 
-Or, if you're using a version of Merlin prior to v1.12.0, you can ensure that the `hello_samples_<timestamp>/` output workspace was created. More info on the expected output can be found in [the Hello World Examples page](../../examples/hello.md#expected-output-1).
+Or, if you're using a version of Merlin prior to v1.12.0, you can ensure that the `hello_samples_<timestamp>/` output workspace was created. More info on the expected output can be found in [the Hello World Examples page](../../examples/hello.md#expected-output_1).
 
 Congratulations, you just ran a cross-node workflow with a containerized server!
diff --git a/docs/user_guide/configuration/merlin_server.md b/docs/user_guide/configuration/merlin_server.md
diff --git a/docs/user_guide/installation.md b/docs/user_guide/installation.md
@@ -1,6 +1,6 @@
 # Installation
 
-The Merlin library can be installed by using [virtual environments and pip](#installing-with-pip) or [spack](#installing-with-spack).
+The Merlin library can be installed by using [virtual environments and pip](#installing-with-virtual-environments-pip) or [spack](#installing-with-spack).
 
 Contributors to Merlin should follow the [Developer Setup](#developer-setup) below.
 

diff --git a/docs/user_guide/interpreting_output.md b/docs/user_guide/interpreting_output.md
@@ -677,7 +677,7 @@ A visual representation of the `merlin_info/` subdirectory with sample generatio
         f.write(result)
     ```
 
-    Since this script uses some third party libraries ([`names`](https://pypi.org/project/names/) and [`numpy`](https://numpy.org/)), you'll need to install them to your current environment in order to run this example. If you're using a [virtual environment](./installation.md#installing-with-virtual-environments--pip), these can be installed with:
+    Since this script uses some third party libraries ([`names`](https://pypi.org/project/names/) and [`numpy`](https://numpy.org/)), you'll need to install them to your current environment in order to run this example. If you're using a [virtual environment](./installation.md#installing-with-virtual-environments-pip), these can be installed with:
 
     ```bash
     pip install names numpy

diff --git a/docs/user_guide/running_studies.md b/docs/user_guide/running_studies.md
@@ -152,7 +152,7 @@ merlin stop-workers --spec my_specification.yaml
 
 !!! note
 
-    If you wish to execute a workflow after dry-running it, simply use [`merlin restart`](#restart-merlin-restart) (to understand why this works, see the section below on [Restarting Workflows](#restarting-workflows)).
+    If you wish to execute a workflow after dry-running it, simply use [`merlin restart`](./command_line.md#restart-merlin-restart) (to understand why this works, see the section below on [Restarting Workflows](#restarting-workflows)).
 
 'Dry run' means telling workers to create a study's workspace and all of its necessary subdirectories and scripts (with variables expanded) without actually executing the scripts.
 

diff --git a/docs/user_guide/specification.md b/docs/user_guide/specification.md
@@ -82,7 +82,7 @@ The `batch` block is an optional block that enables specification of HPC schedul
 | Property Name | Required? | Type | Description |
 | ------------- | --------- | ---- | ----------- |
 | `bank` | Yes | str | Account to charge computing time to |
-| `dry_run` | No | bool | Execute a [dry run](./command_line.md#dry-run) of the study |
+| `dry_run` | No | bool | Execute a [dry run](./running_studies.md#dry-runs) of the study |
 | `launch_args` | No | str | Extra arguments for the parallel launch command |
 | `launch_pre` | No | str | Any configuration needed before the scheduler launch command (`srun`, `jsrun`, etc.) |
 | `nodes` | No | int | The number of nodes to use for all workers. This can be overridden in [the `resources` property of the `merlin` block](#resources). If this is unset the number of nodes will be queried from the environment, failing that, the number of nodes will be set to 1. |