feat: Add remote source support for IR migration#17
Merged
DamianReeves merged 12 commits intomainfrom Jan 23, 2026
Merged
Conversation
Implement spec-compliant V4 Morphir IR structures: - Literal, Type, Pattern, Value, TypeDefinition, ValueDefinition - TypeAttributes, ValueAttributes with source locations - V4-only constructs: Hole, Native, External, IncompleteTypeDefinition Add tagged array serde for Morphir IR format: - Custom Serialize/Deserialize for Type, Pattern, Value - Backward compatibility with lowercase tags (Classic V1) Add TransformVisitor pattern for IR conversion: - TypeTransformVisitor, PatternTransformVisitor, ValueTransformVisitor - Walker functions for recursive transformation - Support for V4-only variant handling during downgrade Fix Classic Module serialization to output tuple format for V2+ All 61 unit tests and 18 acceptance tests pass.
- Create PackageName newtype wrapping Path for package identifiers - Create ModuleName newtype wrapping Path for module identifiers - Update v4.rs to use proper newtypes instead of type aliases - Update converter, loader, and migrate command to use newtypes - Update acceptance tests to handle newtype conversions This provides type safety to distinguish between package paths and module paths at compile time, preventing accidental misuse.
- Add ClassicToV4Converter implementing TypeTransformVisitor and ValueTransformVisitor for upgrading Classic IR to V4 format - Add V4ToClassicConverter for downgrading V4 IR to Classic format - Add ConversionError type with CannotDowngrade variant for V4-only constructs (Hole, Native, External) that can't be represented in Classic - Add comprehensive tests for attribute transformation, type conversion, and error handling for unsupported V4 constructs The visitor-based converters provide strongly-typed transformation between IR formats, complementing the existing JSON-based conversion functions.
Phase 7 of IR V4 spec conformance - key changes:
- Add V4 object wrapper serialization for Type, Pattern, Value
(serde_v4.rs) following `{ "Tag": { content } }` format
- Restructure v4.rs with Distribution enum (Library, Specs, Application)
using wrapper object format
- Convert from Vec<TupleStruct> to IndexMap<String, T> for keyed objects:
- Dependencies, modules, types, values now use IndexMap
- Add wrapper object serde for NativeHint, HoleReason, ValueBody, TypeDefinition
- Add FQName canonical string methods (to_canonical_string, from_canonical_string)
- Update converter.rs to work with new IndexMap-based v4 structures
- Enable schemars indexmap2 feature for JsonSchema support
All 98 tests pass.
Add comprehensive documentation for the `morphir ir migrate` command: - Usage examples for Classic to V4 and V4 to Classic conversion - Format differences between Classic and V4 IR - Workflow integration examples (CI/CD, pre-commit hooks) - Troubleshooting guide for common issues - Limitations when downgrading V4-only constructs
Add comprehensive remote source support to the `ir migrate` command, allowing users to migrate Morphir IR from remote sources (HTTP, Git, GitHub, Gist) in addition to local files. Key changes: - Add new `remote` module in morphir-common with: - RemoteSource enum for parsing various source formats - RemoteSourceResolver for fetching and caching sources - SourceCache for local caching with TTL support - HTTP and Git fetchers for downloading sources - RemoteSourceConfig for allow/deny list configuration - Update migrate command to accept remote sources - Add --force-refresh and --no-cache CLI flags - Fix V4 format serialization: - Path/Name now accept both string (V4) and array (Classic) formats - PackageName serializes as canonical string for V4 compatibility - Update test fixtures for V4 wrapper object format Supported source formats: - Local paths: ./morphir-ir.json - HTTP/HTTPS: https://example.com/ir.json - GitHub shorthand: github:owner/repo[@ref][/path] - Git URL: https://github.com/org/repo.git - Gist: gist:id[#filename]
- Add documentation for new remote source input types (HTTP, GitHub, Git, Gist) - Add examples using the FINOS LCR IR as a comprehensive remote source - Document --force-refresh and --no-cache CLI flags - Add morphir.toml configuration examples for remote sources - Add troubleshooting section for remote source errors
- Add Jekyll configuration (_config.yml) with minimal theme - Add documentation homepage (index.md) with quick start guide - Add GitHub Actions workflow for Pages deployment - Add Jekyll front matter to ir-migrate.md The docs will be available at https://finos.github.io/morphir-rust/ once GitHub Pages is enabled in repository settings.
- Replace redundant closures with direct function references - Use is_none_or/is_some_and instead of map_or - Fix overindented doc list items - Add #[allow(clippy::enum_variant_names)] for Morphir spec-defined enums - Implement FromStr trait for PackageName and ModuleName - Replace unwrap_or_else(T::default) with unwrap_or_default() - Use strip_prefix/strip_suffix instead of manual string slicing - Change &PathBuf to &Path in function signatures - Use next_back() instead of last() for DoubleEndedIterator
- Apply cargo fmt formatting to converter.rs - Fix PathBuf comparison in source.rs tests
- Remove unneeded unit return types and expressions - Replace test_attrs() helper with direct () usage - Use non-PI float constant in literal test
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 467109ec14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+177
to
+181
| // Check for @ref suffix | ||
| if let Some(at_pos) = parts.find('@') { | ||
| let ref_part = parts[at_pos + 1..].to_string(); | ||
| parts = parts[..at_pos].to_string(); | ||
| reference = Some(Self::parse_git_ref(&ref_part)); |
There was a problem hiding this comment.
If a user follows the documented shorthand github:owner/repo@ref/path, this parser strips everything after the first @ into ref_part, so reference becomes ref/path and the subpath is lost. That makes the git fetch attempt use an invalid branch/tag like main/examples/basic/morphir-ir.json, causing the clone/checkout to fail for the documented input format. Consider parsing the ref after splitting out the subpath or supporting both orderings.
Useful? React with 👍 / 👎.
Comment on lines
+330
to
+332
| let mut result = format!("https://github.com/{}/{}", owner, repo); | ||
| if let Some(sub) = subpath { | ||
| result = format!("{}/{}", result, sub); |
There was a problem hiding this comment.
Allow/deny checks call to_url_string, but GitHub sources are normalized to https://github.com/... here. That means config patterns using the same shorthand users pass on the CLI (e.g. github:finos/* from the new docs) will never match, so GitHub sources get blocked whenever allow is non-empty unless trusted_github_orgs is set. Either normalize to github: for GitHub sources or document that allow/deny must use the HTTPS form.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements Phase 8 of the Morphir IR migration plan: Remote Source Support. Key changes include:
github:owner/repo), Git repos, and Gistsmorphir.tomlir migratecommand with examples including the LCR IRUsage Examples
Test plan
cargo clippy --workspace -- -D warningspassescargo fmt --all --checkpasses