feat: Gleam binding enhancements, dev mode launcher, and VFS improvements#36
Merged
DamianReeves merged 67 commits intomainfrom Jan 24, 2026
Merged
feat: Gleam binding enhancements, dev mode launcher, and VFS improvements#36DamianReeves merged 67 commits intomainfrom
DamianReeves merged 67 commits intomainfrom
Conversation
Update the Getting Started installation section with working commands: - mise: use github:finos/morphir-rust backend instead of registry - cargo-binstall: add --git flag since not published to crates.io - Add manual download option linking to GitHub Releases - List all supported platforms
Add a launcher script that acts as both installer and version manager for morphir, similar to Mill's approach. Features: - Auto-download: Downloads morphir binary on first run - Version pinning: Supports .morphir-version and morphir.toml files - Version override: Use +0.1.0 syntax for explicit version - Multiple backends: mise > cargo-binstall > GitHub releases > cargo - Self commands: upgrade, list, which, install, prune, update Scripts: - scripts/morphir.sh - Unix (bash) launcher - scripts/morphir.ps1 - Windows (PowerShell) launcher - scripts/install.sh - Unix bootstrap installer - scripts/install.ps1 - Windows bootstrap installer Also includes: - Jekyll setup with Poole theme for documentation site - mise tasks for local Jekyll development (docs:serve) - Updated README with new installation instructions
The jekyll-remote-theme plugin only copies layouts, includes, and sass partials - not static assets. Add: - docs/styles.scss to generate CSS from theme partials - docs/public/ with favicon and apple touch icon from Poole v2.1.0
Add entries for: - OS files (Thumbs.db) - IDE/Editor files (.idea, .vscode, swap files) - Environment files (.env, .envrc) - Root node_modules/ - Morphir launcher local cache
Binary/CLI projects should commit Cargo.lock to ensure: - Reproducible builds for contributors - Consistent CI builds - Reproducible `cargo install --path` results
- Switch from Poole to just-the-docs theme for sidebar navigation - Add dark/light theme toggle support - Split documentation into multiple pages: - index.md: Landing page - getting-started.md: Getting started guide - install.md: Detailed installation guide - cli/: Auto-generated CLI reference with nesting - ir-migrate.md: IR migration guide - Update docs generator to add just-the-docs front matter - CLI commands now nest properly in sidebar navigation
- Add logo, favicon, and icon assets from morphir repo - Create custom color scheme matching morphir.finos.org: - Light mode: #2e8555 (green) - Dark mode: #25c2a0 (teal) - Add hero banner to landing page with morphir logo - Add link to morphir.finos.org in header - Update footer with FINOS and Morphir links
… output - Add dark/light theme toggle to docs site with localStorage persistence - Add release notes page (docs/releases.md) auto-generated from CHANGELOG - Create file-based mise tasks for docs generation (docs:releases, docs:generate) - Change ir migrate input from -i flag to positional argument - Add --json flag to ir migrate for scripting/CI use cases - Add termimad for console markdown rendering - Update AGENTS.md with CLI docs generation workflow - Update release-manager skill with docs:generate task
- Add ratatui-based JSON pager with syntax highlighting, line numbers, scrolling - Add structured logging module (tracing) with workspace-aware log directories - Fix --json mode to emit migrated IR to stdout when no output file specified - Update AGENTS.md with TUI stack (ratatui, crossterm, tui-realm) and logging standards - Progress messages now go to stderr (stdout reserved for actual output) - Add universal --help flag support for all commands
- Fix Rust formatting in pager.rs - Add #[allow(dead_code)] to scaffolded but unused logging and help functions - Fix shell script formatting in releases task - Fix pre-push task to reference correct lint task names - Regenerate documentation
The IR migration from Classic to V4 format was still using Classic-style
array naming (e.g., ["word", "parts"]) instead of V4 canonical naming
(kebab-case strings like "word-parts").
Changes:
- Add V4Name newtype wrapper that serializes as kebab-case string
- Update V4 structs (ConstructorDefinition, ConstructorArg, TypeDefinition)
to use V4Name instead of Name for proper serialization
- Add comprehensive type/value expression converters:
- convert_type_expr_to_v4() for Classic array -> V4 object wrapper format
- convert_value_expr_to_v4() for value expressions with canonical FQNames
- convert_pattern_to_v4() and convert_literal_to_v4() for patterns/literals
- extract_fqname_from_classic() for canonical "pkg/path:mod#name" format
- Update extract_type_params() to return Vec<V4Name>
- Update convert_constructor_to_v4() to use V4Name for names
The V4 output now correctly shows:
- Constructor names: "close-deal" instead of ["close", "deal"]
- Type params: ["comparable-node"] strings instead of [[...]] arrays
- FQNames: "morphir/sdk:set#set" canonical format
- Type expressions: {"Variable": {"name": "a"}} object wrapper format
Rename the V4 string-serialized name type from V4Name to Name for a cleaner API: - V4 types now use `v4::Name` (serializes as kebab-case string) - Classic types use `naming::Name` (aliased as ClassicName in v4.rs) This makes V4 code more natural since `Name` is the expected type name, while making it clear that Classic format uses a different representation.
Restructure naming so that V4 format (kebab-case string) is the default:
- `naming::Name` now serializes as kebab-case string ("my-type-name")
- `classic::ClassicName` wraps Name and serializes as array (["my", "type", "name"])
- `v4::Name` is now just a re-export of `naming::Name`
- Both formats support bidirectional conversion via From/Into
This makes V4 the default going forward, with ClassicName available
for backward compatibility when working with Classic IR.
Deserialization remains flexible - both formats accept either
string or array input for seamless migration.
Update the Format Differences section to document the compact V4 format:
Type expressions:
- Variables: bare name string ("a")
- References (no args): bare FQName string
- References (with args): {"Reference": {"fqname": ..., "args": [...]}}
- Records: {"Record": {fields}} without wrapper
Value expressions:
- Variables: {"Variable": "name"}
- References: {"Reference": "fqname"}
- Records: {"Record": {fields}}
Also update the LCR example to show compact format output.
Changed Reference with type arguments from object format:
{"Reference": {"fqname": "pkg:mod#name", "args": [...]}}
To array format:
{"Reference": ["pkg:mod#name", arg1, arg2, ...]}
The first element is the FQName, followed by type arguments. This is
more compact and easier to read/write.
Adds --expanded flag to `morphir ir migrate` command that outputs
the expanded (non-compact) V4 format:
Compact (default):
- Variable: "a" (bare string)
- Reference (no args): "pkg:mod#name" (bare FQName)
- Reference (with args): {"Reference": ["pkg:mod#name", arg1, ...]}
Expanded (--expanded):
- Variable: {"Variable": "a"}
- Reference: {"Reference": {"fqname": "pkg:mod#name", "args": [...]}}
This is useful for debugging, tooling compatibility, and when a more
explicit format is needed.
Change expanded Variable format from {"Variable": "a"} to
{"Variable": {"name": "a"}} for consistency with other expanded
type expressions.
Added comprehensive tests for both compact and expanded formats
for Variable and Reference type expressions.
…umentation - Add builtin extension support to extension registry and CLI list command - Implement explicit config merging (workspace > project > CLI args) - Extend Vfs trait with remove, copy, and metadata methods - Implement NotebookVfs for Jupyter notebook-based testing - Add comprehensive CLI commands (compile, generate, gleam subcommands) - Implement JSON and JSON Lines output support - Add rich error handling with miette diagnostics - Create user documentation (5 tutorials) - Create contributor documentation (5 design docs) - Add CLI E2E testing with BDD scenarios - Add unit and integration tests for CLI commands - Update .morphir/ folder structure to Mill-inspired layout
- Fix nbformat API usage in NotebookVfs (use v4 module correctly) - Fix PathBuf comparison issues in VFS implementations - Fix config.rs type mismatch for project_root unwrap_or - Add uuid dependency for NotebookVfs - Fix unused variable warnings - Apply cargo fmt formatting across all files - Fix import organization and code style issues
- Update workspace edition from 2021 to 2024 in Cargo.toml - Add rust-version = "1.85" to workspace package - Update rustfmt.toml edition to 2024 - Run cargo fix --edition to automatically migrate all crates - Update dependency version constraints to latest compatible versions: - starbase: 0.10 - ratatui: 0.30 - crossterm: 0.29 - chumsky: 1.0.0-alpha.8 (for API compatibility) - Various other dependencies updated - Fix error conversions (anyhow::Error -> CliError) for Rust 2024 compatibility - Add missing load_ir function to morphir-common::loader - Fix SourceLocation field access in error conversion - Remove Logos #[error] attribute (no longer needed in 0.14+) - Add priority to Logos regex patterns to resolve conflicts - Add Deserialize to Diagnostic struct - Fix unused import warnings - All core crates compile successfully - All tests pass (105 tests in core crates) Core crates verified: - morphir-common ✅ - morphir-design ✅ - morphir-ir ✅ - morphir-daemon ✅
- Update logos from 0.14 to 0.16 (latest stable) - Update chumsky to 0.12 (latest stable) and update parser API usage - Add Hash and Eq traits to Token enum for chumsky compatibility - Add Hash trait to FeatureStatus enum for HashMap usage - Fix visitor import in backend/codegen.rs (use super::visitor) - Update parser function signatures with lifetime parameters for chumsky 0.12 - Remove Error type parameter from Parser trait bounds (not supported in 0.12) - Fix write_output return type (anyhow::Result -> std::io::Result) - Fix CliError::report() to clone self before passing to miette - Update to_parse_error function signature with lifetime parameter - Fix unused import warnings Core crates compile successfully: - morphir-common ✅ - morphir-design ✅ - morphir-ir ✅ - morphir-daemon ✅ All tests pass (105 tests in core crates) Note: morphir-gleam-binding still has chumsky 0.12 API compatibility issues that require parser refactoring (separate from edition migration)
Update parser to use chumsky 0.12 API: - Use IterInput for spanned token input handling - Update all parser function signatures with ValueInput bounds - Add .boxed() to recursive parsers for Clone trait requirements - Fix custom type body parser to use repeated() for Gleam syntax Fix backend visitor: - Handle all V4 ValueBody variants (NativeBody, ExternalBody, IncompleteBody) - Add Decimal literal handling Fix codegen: - Resolve OsVfs ownership issues All 10 library unit tests now pass.
- Add cargo-llvm-cov tool to mise.toml - Create check:coverage-report task (generates HTML report with incremental build) - Create check:coverage task (verifies per-crate thresholds) - Add coverage threshold to morphir-gleam-binding (40%) - Add CI coverage job with 14-day artifact retention Per-crate thresholds are configured via Cargo.toml metadata: [package.metadata.coverage] threshold = 40
There was a problem hiding this comment.
Pull request overview
This PR introduces significant enhancements to the Morphir Rust project including Gleam language binding improvements, developer experience features, VFS improvements, and comprehensive documentation.
Changes:
- Gleam binding enhancements with parser improvements, AST alignment, pretty printing, and roundtrip testing infrastructure
- Dev mode launcher support via CLI flag, environment variables, or configuration to run from local source
- VFS contract documentation and MemoryVfs consistency fixes
- Extensive documentation including tutorials, contributor guides, and CLI reference
- Core CLI commands promoted from experimental to stable (compile, generate, gleam subcommands)
- Coverage reporting infrastructure with per-crate thresholds
Reviewed changes
Copilot reviewed 124 out of 125 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| rustfmt.toml | Edition update (CRITICAL ISSUE: edition 2024 doesn't exist) |
| mise.toml | Added cargo-llvm-cov for coverage |
| Cargo.toml | Version bump to 0.2.0, edition/rust-version updates (CRITICAL ISSUE) |
| docs/tutorials/* | New tutorial documentation for getting started, Gleam, configuration, workflows |
| docs/contributors/* | New contributor guides for architecture, development, extension system |
| docs/cli/* | CLI reference documentation including new commands |
| crates/morphir/src/* | Core CLI implementation with compile, generate, gleam commands |
| crates/morphir-design/* | New design-time crate for configuration and extension discovery |
| crates/morphir-daemon/* | Extension registry enhancements for builtin extensions |
| crates/morphir-common/* | VFS improvements, pipeline extensions, loader enhancements |
| crates/morphir-gleam-binding/* | Parse stage emission, expanded frontend/backend, test infrastructure |
| .github/workflows/ci.yml | Added coverage workflow |
| completions/* | Updated shell completions for new commands |
Comments suppressed due to low confidence (1)
crates/morphir-common/src/vfs/memory.rs:72
- The VFS trait documentation states that
exists()should return true for directories, but the previous implementation in MemoryVfs only checked files. The updated implementation now correctly checks both files and directories by iterating through all stored paths to find prefixes. However, this could be inefficient for large filesystems with many files.
Consider maintaining a separate set of known directories to avoid O(n) iteration on every exists() call for directories.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
79
to
90
| // Emit parse stage JSON if enabled | ||
| if emit_parse_stage | ||
| && let Err(e) = emit_parse_stage_json(&output_dir, &source.path, &module_ir) | ||
| { | ||
| diagnostics.push(Diagnostic { | ||
| severity: DiagnosticSeverity::Warning, | ||
| code: Some("W001".into()), | ||
| message: format!("Failed to emit parse stage output: {}", e), | ||
| location: None, | ||
| related: vec![], | ||
| }); | ||
| } |
There was a problem hiding this comment.
The error handling in this compile function silently catches and logs errors when emitting parse stage JSON, but continues execution. This means parse stage output failures are treated as warnings rather than errors. If parse stage output is critical for debugging or tooling integration, this could hide important failures.
Consider making emit_parse_stage failures more visible or providing a configuration option to make them fatal errors.
Member
Author
There was a problem hiding this comment.
@copilot open a new pull request to apply changes based on this feedback
Comment on lines
104
to
146
| match visitor.visit_module_v4(&module_ir) { | ||
| Ok(_) => { | ||
| // Read format.json as IR representation | ||
| // output_dir is already .morphir/out/<project>/compile/<language>/ | ||
| let format_json_path = output_dir.join("format.json"); | ||
| let vfs = OsVfs; | ||
| if vfs.exists(&format_json_path) { | ||
| match vfs.read_to_string(&format_json_path) { | ||
| Ok(format_content) => { | ||
| match serde_json::from_str::<serde_json::Value>( | ||
| &format_content, | ||
| ) { | ||
| Ok(ir_json) => { | ||
| ir_modules.push(ir_json); | ||
| } | ||
| Err(e) => { | ||
| diagnostics.push(Diagnostic { | ||
| severity: DiagnosticSeverity::Error, | ||
| code: Some("E002".into()), | ||
| message: format!( | ||
| "Failed to parse format.json: {}", | ||
| e | ||
| ), | ||
| location: None, | ||
| related: vec![], | ||
| }); | ||
| } | ||
| } | ||
| } | ||
| Err(e) => { | ||
| diagnostics.push(Diagnostic { | ||
| severity: DiagnosticSeverity::Error, | ||
| code: Some("E003".into()), | ||
| message: format!("Failed to read format.json: {}", e), | ||
| location: None, | ||
| related: vec![], | ||
| }); | ||
| } | ||
| } | ||
| } else { | ||
| // Fallback: serialize ModuleIR directly | ||
| ir_modules.push(serde_json::to_value(&module_ir)?); | ||
| } |
There was a problem hiding this comment.
The compile command reads the entire format.json file back into memory after writing it, then includes it in the ir_modules response. For large projects, this could cause memory issues as the IR is essentially duplicated in memory.
Consider either: 1) returning a reference/path to the written file instead of the full content, or 2) keeping the IR in memory and avoiding the write-then-read cycle if it's needed immediately.
Member
Author
There was a problem hiding this comment.
@copilot open a new pull request to apply changes based on this feedback
Lets try option 2 first
|
@DamianReeves I've opened a new pull request, #38, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
@DamianReeves I've opened a new pull request, #39, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
|
DamianReeves
force-pushed
the
docs/installation-instructions
branch
from
16f6463 to
d86ab78
Compare
…e emission (#38) * Initial plan * feat(gleam-binding): add emitParseStageFatal option for configurable error handling Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com> --------- Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
…onse (#39) * Initial plan * refactor: avoid disk I/O for format.json in compile response Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com> --------- Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
DamianReeves
force-pushed
the
docs/installation-instructions
branch
from
d86ab78 to
8a89b89
Compare
- Update roundtrip_should_complete to actually perform full roundtrip - Add to_pascal_case helper for Gleam type constructor names - Fix type definition generation (constructors space-separated, not pipe) - Add generate_type_expr method for V4 JSON type conversion
- Add cli_tests_available() to check if morphir binary is accessible - Update find_workspace_root to locate correct Cargo.toml - Skip CLI steps gracefully when context is not available - Write source files to disk when CLI context is present
DamianReeves
force-pushed
the
docs/installation-instructions
branch
from
d073d40 to
3d23041
Compare
- Add build job that creates morphir binary artifact - Split test into unit tests (runs in parallel with build) - Add integration tests that download and use the binary artifact - This enables CLI e2e tests to run with the actual binary
- Add daemon and extension design documents from main morphir repo - Format all design documents for Jekyll with proper frontmatter - Add links to main Morphir site and llms.txt throughout - Add actor-based extension system design documents (13 files) - Add technical-writer skill adapted for Jekyll/GitHub Pages - Include scripts for link checking, structure validation, and API docs - Update extension README to reference actor-based design docs All documents are formatted for Jekyll and include proper navigation structure and cross-references to main Morphir documentation.
…t tagging - Add mise tasks for build and test workflows: - build:release - Stage release binary to .morphir/build/bin/ - build:debug - Stage debug binary - test:unit - Run unit tests only - test:integration - Run integration tests with staged binary - test:all - Run all tests - Create integration-tests crate (publish=false) for CLI tests - Move CLI test helpers from morphir-gleam-binding - Separate CLI integration tests that require binary - Set publish=false on morphir-tests crate - Mark failing acceptance tests as @wip with issue reference: - roundtrip.feature: order_validation and order_processing scenarios - cli.feature: All CLI tests (not implemented) - project.feature: Module name lookup issues - workspace.feature: Not implemented - codegen.feature: Not implemented - Update ci.yml to use mise tasks consistently - Update cli_helpers.rs to check staged binary first Fixes: #40
- morphir-cet: Switch from custom Gleam parser to official Gleam parser - morphir-rust-72s: Implement Python binding using Ruff parser crate
- morphir-rust-ykr: Implement TypeScript binding using SWC parser crate
- morphir-rust-kw8: Design and implement Morphir daemon and extension system - Phase 1: MVP Daemon - Phase 2: Extension System Design - Phase 3: LSP Implementation - Phase 4: Production Hardening - Phase 5: IDE Extensions
The mise-action installs Rust but doesn't include rustfmt/clippy components by default. Add explicit rustup component installation step.
When a new commit is pushed to the same branch/PR, any in-progress CI runs are automatically canceled to avoid wasting resources.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR includes significant improvements to the Gleam language binding, developer experience, and infrastructure:
Gleam Binding Enhancements
prettycrate.morphir/out/<project>/parse/<module>.jsonDeveloper Experience
morphir.shandmorphir.ps1now support running from source via:--devCLI flagMORPHIR_DEV=1environment variablelocal-devin.morphir-versionfiledev_mode = trueinmorphir.toml[morphir]sectionmorphir self dev: New command to show dev mode status and configurationVFS Improvements
Vfstrait semanticsexists()now returnstruefor directories (matchingOsVfs/std::path::Path::exists())Documentation
Test plan
cargo test -p morphir-gleam-binding- Unit tests passcargo clippy- No warningscargo fmt- Properly formatted🤖 Generated with Claude Code