Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ notebooks/.ipynb_checkpoints/
notebooks/beginner/.ipynb_checkpoints/
notebooks/intermediate/.ipynb_checkpoints/
notebooks/advanced/.ipynb_checkpoints/
.scalable/

# -----------------------------
# Python bytecode / caches
Expand Down Expand Up @@ -110,4 +111,7 @@ communicator/src/communicator
# -----------------------------
# Existing repo-specific generated files
# -----------------------------
scalable/custom_types.py
scalable/custom_types.py
# Demeter tutorial artifacts (see docs/tutorials/demeter_setup.rst)
/demeter_data/
/outputs/
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Documentation recommends beginner path**: `README.md`, `docs/getting_started.rst`,
and `docs/tutorials/index.rst` now recommend starting with beginner tutorials for
users unfamiliar with distributed computing concepts.
- **New optional dependency extra**: added `[project.optional-dependencies] all`
to install test, dev, ai, cloud, kubernetes, and ml dependency groups in one
command (`pip install scalable[all]`).

### Changed

- Bumped package version to `2.0.0b2`.
- Updated installation examples in `README.md` and `docs/getting_started.rst`
to use `pip install scalable[all]` for all optional dependencies.

---

Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ cff-version: 1.2.0
message: "If you use this software, please cite it as below."
type: software
title: "Scalable"
version: "2.0.0-beta.1"
version: "2.0.0-beta.6"
doi: "10.5281/zenodo.20295640"
url: "https://github.com/JGCRI/scalable"
repository-code: "https://github.com/JGCRI/scalable"
Expand Down
36 changes: 2 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ Scalable is a Python framework for orchestrating containerized, distributed work
- [Telemetry and Reports](#telemetry-and-reports)
- [Resource Advising](#resource-advising)
- [ML Optimization](#ml-optimization)
- [Model Emulation](#model-emulation)
- [AI Assistants](#ai-assistants)
- [Cloud and Kubernetes](#cloud-and-kubernetes)
- [Artifact Storage](#artifact-storage)
Expand Down Expand Up @@ -120,11 +119,11 @@ pip install scalable[cloud]
# Kubernetes provider (Dask Kubernetes Operator)
pip install scalable[kubernetes]

# ML optimization and emulation (LearnedAdvisor, AdaptiveScaler, emulators)
# ML optimization (LearnedAdvisor, AdaptiveScaler)
pip install scalable[ml]

# All optional dependencies
pip install scalable[ai,cloud,kubernetes,ml]
pip install scalable[all]
```

If your shell cannot find installed scripts (for example, `scalable_bootstrap`), add the relevant scripts directory to `PATH`.
Expand Down Expand Up @@ -372,34 +371,6 @@ CLI access:
scalable advise --task run_gcam --model-type gradient_boosting --format json
```

### Model Emulation

The emulation subsystem (`scalable[ml]`) provides uncertainty-aware surrogate
model dispatch for expensive scientific functions:

```python
from scalable import emulatable, EmulatorRegistry, EmulatorDispatch

@emulatable(
inputs=["temperature", "precipitation"],
outputs=["yield"],
domain_bounds={"temperature": (250, 350), "precipitation": (0, 5000)},
confidence_threshold=0.9,
)
def run_crop_model(temperature, precipitation):
# Expensive model execution
...

# Register and manage trained emulators
registry = EmulatorRegistry(".scalable/emulators")
dispatch = EmulatorDispatch(registry, confidence_threshold=0.9)

# Confidence-gated routing: uses emulator when confident, falls back to full model
result = dispatch.predict("run_crop_model", inputs={"temperature": 300, "precipitation": 1200})
print(result.source) # "emulator" or "full_model"
print(result.confidence)
```

### AI Assistants

AI assistants help with onboarding, diagnostics, workflow generation, and
Expand Down Expand Up @@ -630,9 +601,6 @@ Override `AI_API_KEY` for individual providers when using multiple services:
| `SCALABLE_DEFAULT_STORAGE` | *(unset)* | Default artifact storage URI |
| `SCALABLE_ML` | `1` | Enable ML features |
| `SCALABLE_ML_CACHE_DIR` | `.scalable/models` | ML model cache directory |
| `SCALABLE_EMULATION` | `0` | Enable model emulation |
| `SCALABLE_EMULATOR_DIR` | `.scalable/emulators` | Emulator registry directory |
| `SCALABLE_EMULATION_CONFIDENCE` | `0.9` | Emulation confidence threshold |

### Advanced AI Overrides

Expand Down
4 changes: 2 additions & 2 deletions docs/advising.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Quick start

advisor = ResourceAdvisor.from_history("./.scalable/runs")
recommendation = advisor.recommend(
task="run_gcam",
task="run_demeter_scenario",
target="local",
confidence=0.95,
)
Expand Down Expand Up @@ -42,7 +42,7 @@ CLI access

.. code-block:: bash

scalable advise --task run_gcam --target local --confidence 0.95
scalable advise --task run_demeter_scenario --target local --confidence 0.95

The CLI ``advise`` command first attempts ML-backed recommendations (if
``scalable[ml]`` is installed) and falls back to the heuristic advisor when
Expand Down
13 changes: 5 additions & 8 deletions docs/ai_assistants.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
AI Assistants
=============

Scalable v2.0.0 includes AI-assisted features (Phase 4) that help users
onboard models, compose workflows, diagnose failures, explain plans, and
migrate manifests.
Scalable v2.0.0 includes AI-assisted features that help users onboard
models, compose workflows, diagnose failures, explain plans, and migrate
manifests.

All features work **without** an LLM backend via deterministic heuristic
fallbacks. LLM enhancement is opt-in via the ``SCALABLE_AI_BACKEND``
Expand Down Expand Up @@ -162,17 +162,15 @@ scalable compose

Generate a workflow from a natural-language description::

scalable compose "Run GCAM reference scenario then Stitches for daily weather"
scalable compose "Run Hector model" --output-dir ./generated
scalable compose "Run the GCAM reference scenario and downscale land allocation using Demeter" --output-dir ./generated
scalable compose "Run the GCAM reference scenario" --output-dir ./generated

Options:

* ``--output-dir`` — Directory for generated files
* ``--format`` — Output format (``text`` or ``json``)
* ``--no-ai`` — Use heuristics only

Known model patterns: GCAM, Stitches, Demeter, Tethys, Xanthos, Hector.

scalable migrate
~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -264,5 +262,4 @@ See Also

- :doc:`manifest` — Manifest schema and session API
- :doc:`ml` — ML-backed resource optimization
- :doc:`emulation` — Model emulation with surrogate dispatch
- :doc:`telemetry` — Run telemetry that powers diagnosis
6 changes: 3 additions & 3 deletions docs/artifacts.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Artifact Store
==============

The :mod:`scalable.artifacts` module (Phase 3) provides a protocol-based
abstraction for storing and retrieving workflow artifacts across local and
remote backends.
The :mod:`scalable.artifacts` module provides a protocol-based abstraction
for storing and retrieving workflow artifacts across local and remote
backends.

Overview
--------
Expand Down
4 changes: 2 additions & 2 deletions docs/cloud.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ GCP Provider (Scaffold)
-----------------------

The :class:`~scalable.providers.cloud.gcp.GCPProvider` is a validation-only
scaffold in Phase 3. It validates manifest options but raises
``NotImplementedError`` on ``build_cluster()``.
scaffold. It validates manifest options but raises ``NotImplementedError``
on ``build_cluster()``.

Target options:

Expand Down
183 changes: 0 additions & 183 deletions docs/emulation.rst

This file was deleted.

Loading
Loading