Skip to content

Commit 14bc258

Browse files
committed
feat: Add type aliases and multi-file imports (#52)
Implements comprehensive type alias and import system for LUMOS, enabling DRY schemas and multi-file organization. FEATURES: - Type aliases with Rust-style syntax (type UserId = PublicKey) - JavaScript-style imports (import { Type1, Type2 } from "./file.lumos") - Recursive type resolution with circular reference detection - Multi-file schema support with automatic import discovery - Cross-file type validation (three-pass architecture) IMPLEMENTATION: Core: - Added Import and TypeAlias structs to AST with span tracking - Implemented multi-line import parser using regex - Created TypeAliasResolver with recursive alias resolution - Implemented FileResolver (340 lines) for multi-file resolution - Added TypeAliasDefinition to IR and TypeAlias variant - Updated all generators (Rust pub type, TypeScript export type) - Fixed 15+ non-exhaustive pattern matches across codebase Dependencies: - Added regex = "1.10" for import parsing CLI: - Updated generate command to use FileResolver - Automatic multi-file import discovery and loading Testing: - Added 4 new file_resolver tests (single, circular, multiple, validation) - All 202 tests passing (100% success rate) - E2E compilation tests verified EXAMPLES: - examples/type_aliases.lumos (200+ lines, 23 type definitions) - examples/imports/types.lumos (24 definitions + 2 enums) - examples/imports/accounts.lumos (180+ lines, 5 account structs) - examples/imports/instructions.lumos (240+ lines, 6 instruction enums) - examples/imports/README.md (comprehensive documentation) VALIDATION: - Circular import detection with clear error messages - Undefined type detection across all files - Enum import support (fixed validation timing issue) - Multi-line import syntax support FILES MODIFIED: - 21 files changed, +1500/-302 lines - packages/core/src/file_resolver.rs (new, 340 lines) - parser.rs, transform.rs, generators (major updates) - All test files updated and passing BREAKING CHANGES: None BACKWARD COMPATIBLE: Yes Phase 5.3 Advanced Type System: 60% complete (3/5 issues)
1 parent 3d91396 commit 14bc258

30 files changed

+2913
-302
lines changed

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

ROADMAP.md

Lines changed: 61 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ LUMOS continues rapid evolution with IDE integration and schema versioning:
2525
-**Interactive playground** - Live code generation at docs.lumos-lang.org/playground
2626
-**Performance benchmarks** - Comprehensive Borsh comparison suite
2727

28-
**Completed**: Phase 5.1 (Schema Evolution - 100%), Phase 6.3 (Security & Validation - 100%)
29-
**Active**: Phase 5.2 (IDE Integration - 80%)
30-
**Next**: Phase 5.3 (Advanced Type System), Phase 5.4 (Multi-Language Generation)
28+
**Completed**: Phase 5.1 (Schema Evolution - 100%), Phase 5.2 (IDE Integration - 100%), Phase 6.3 (Security & Validation - 100%)
29+
**Active**: Phase 5.3 (Advanced Type System - 60%)
30+
**Next**: Phase 5.4 (Multi-Language Generation), Phase 6.1 (Framework Integration)
3131

3232
---
3333

@@ -182,13 +182,57 @@ Transform from schema DSL → full programming language for type-safe Solana wor
182182

183183
**Goal**: Express complex Solana program constraints
184184

185-
**Issues to create:**
186-
- [ ] Add custom derive macros support to LUMOS [#50]
187-
- [ ] Add const generics support for fixed-size arrays in LUMOS [#51]
188-
- [ ] Add type aliases and imports to LUMOS [#52]
185+
**Status**: 3/5 complete (60%)
186+
187+
**Issues:**
188+
- [x] Add custom derive macros support to LUMOS [#50]**COMPLETE**
189+
- [x] Add const generics support for fixed-size arrays in LUMOS [#51]**COMPLETE**
190+
- [x] Add type aliases and imports to LUMOS [#52]**COMPLETE**
189191
- [ ] Add nested module support to LUMOS [#53]
190192
- [ ] Add generic struct/enum definitions to LUMOS [#54]
191193

194+
**Completed**:
195+
- #50 (Nov 24, 2025) - Custom derive macros support
196+
- Extended `AttributeValue` enum with `List(Vec<String>)` variant
197+
- Added `#[derive(...)]` parser with comma-separated list support
198+
- Added `custom_derives` field to IR `Metadata` struct
199+
- Implemented intelligent derive deduplication with `merge_derives()`
200+
- Context-aware generation: Anchor accounts get only custom derives
201+
- Created comprehensive example at examples/custom_derives.lumos
202+
- 17 new tests (parser, transform, generator, end-to-end)
203+
- All 196 tests passing
204+
- Future enhancement: TypeScript derive equivalents (#107)
205+
206+
- #51 (Nov 24, 2025) - Fixed-size arrays (const generics)
207+
- Added `FixedArray { element, size }` variant to AST and IR
208+
- Parser extracts size from `[T; N]` syntax with validation (1-1024)
209+
- Generates Rust `[T; N]` format correctly
210+
- Generates TypeScript `T[]` with `borsh.array(element, size)` (no length prefix!)
211+
- Size calculation: `element_size * count` (no 4-byte Vec prefix)
212+
- Supports nested arrays: `[[u8; 10]; 10]`
213+
- Example schema at examples/fixed_arrays.lumos
214+
- Updated 10+ files across codebase (parser, transform, generators, CLI, etc.)
215+
- All 116 tests passing
216+
- Common Solana patterns now supported: `[u8; 32]` for hashes, `[PublicKey; N]` for authority lists
217+
218+
- #52 (Nov 24, 2025) - Type aliases and imports
219+
- Added `Import` and `TypeAlias` structs to AST with span tracking
220+
- Implemented JavaScript-style import parser: `import { Type1, Type2 } from "./file.lumos"`
221+
- Implemented Rust-style type alias parser: `type UserId = PublicKey`
222+
- Added `TypeAliasDefinition` to IR and `TypeAlias` variant to `TypeDefinition`
223+
- Created `TypeAliasResolver` with recursive alias resolution and circular reference detection
224+
- Implemented multi-file `FileResolver` with automatic import discovery and circular import detection
225+
- Three-pass validation: collect aliases → resolve recursively → transform → validate across all files
226+
- Updated all generators to handle type aliases (Rust `pub type`, TypeScript `export type`)
227+
- Fixed 15+ non-exhaustive pattern matches across codebase
228+
- Updated CLI to use FileResolver for multi-file schema generation
229+
- Added regex dependency for import parsing (multi-line support)
230+
- Created comprehensive examples: `examples/type_aliases.lumos` (200+ lines, 23 types)
231+
- Created multi-file import examples: `examples/imports/{types,accounts,instructions}.lumos`
232+
- Added 4 new file_resolver tests (single file, circular imports, multiple files, validation)
233+
- All 202 tests passing (including E2E compilation tests)
234+
- **Feature complete**: Single-file type aliases + multi-file imports with full validation
235+
192236
**Success Metric**: Support 95% of Anchor program patterns
193237

194238
### 5.4 Multi-Language Code Generation
@@ -647,7 +691,7 @@ Core language free forever, monetize via cloud platform and premium extensions
647691

648692
### Phase 5: Advanced Features 🚧 (In Progress - Nov 2025)
649693

650-
**Overall Progress**: 12/23 features complete (52%)
694+
**Overall Progress**: 13/23 features complete (57%)
651695

652696
**5.1 Schema Evolution (100% complete) ✅:**
653697
- [x] Schema versioning with #[version] attribute (#40)
@@ -656,14 +700,17 @@ Core language free forever, monetize via cloud platform and premium extensions
656700
- [x] Deprecation warnings for schema fields (#43)
657701
- [x] Schema diff tool: `lumos diff` (#44)
658702

659-
**5.2 IDE Integration (80% complete):**
703+
**5.2 IDE Integration (100% complete):**
660704
- [x] Language Server Protocol implementation (#45)
661705
- [x] IntelliJ IDEA / Rust Rover plugin (#46)
662706
- [x] Neovim plugin with Tree-sitter grammar (#47)
663707
- [x] Emacs mode (#48)
708+
- [x] Sublime Text package (#49)
664709

665-
**5.3 Advanced Type System (0% complete):**
666-
- No issues started yet
710+
**5.3 Advanced Type System (60% complete):**
711+
- [x] Custom derive macros support (#50)
712+
- [x] Fixed-size arrays (const generics) (#51)
713+
- [x] Type aliases and imports (#52)
667714

668715
**5.4 Multi-Language Code Generation (0% complete):**
669716
- No issues started yet
@@ -779,6 +826,9 @@ See an opportunity to help? Check our [Contributing Guide](CONTRIBUTING.md) or:
779826
**Last Updated**: November 24, 2025
780827

781828
**Recent Updates**:
829+
- Nov 24, 2025: **Type aliases and imports COMPLETE** (#52) - Phase 5.3 at 60% 🎉 - Multi-file schemas with automatic import discovery
830+
- Nov 24, 2025: **Fixed-size arrays COMPLETE** (#51) - Phase 5.3 at 40% 🎉 - Const generics support for `[T; N]` syntax
831+
- Nov 24, 2025: **Custom derive macros support COMPLETE** (#50) - Phase 5.3 begins (20% complete) 🎉
782832
- Nov 24, 2025: **PHASE 5.2 IDE INTEGRATION COMPLETE** 🎉🎉🎉 - All 5 editors supported!
783833
- Nov 24, 2025: **Sublime Text package COMPLETE** (#49) - Full syntax + LSP + snippets 🎉
784834
- Nov 24, 2025: **Emacs mode COMPLETE** (#48) - Phase 5.2 at 80% 🎉

examples/imports/README.md

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
# Multi-File Imports Example
2+
3+
This example demonstrates **LUMOS's multi-file import system**, allowing you to organize large schemas across multiple files with shared type definitions.
4+
5+
## 📁 File Structure
6+
7+
```
8+
imports/
9+
├── types.lumos # Common type aliases and enums
10+
├── accounts.lumos # Account definitions (imports from types.lumos)
11+
├── instructions.lumos # Instruction enums (imports from types.lumos)
12+
└── README.md # This file
13+
```
14+
15+
## 🎯 What This Demonstrates
16+
17+
### 1. **Shared Type Definitions** (`types.lumos`)
18+
- Common type aliases (UserId, Lamports, TokenAmount, etc.)
19+
- Shared enums (AccountStatus, PermissionLevel)
20+
- Single source of truth for type definitions
21+
22+
### 2. **Cross-File Imports** (`accounts.lumos`, `instructions.lumos`)
23+
- JavaScript-style import syntax
24+
- Multiple imports from same file
25+
- Automatic dependency resolution
26+
27+
### 3. **Circular Import Detection**
28+
- LUMOS prevents circular dependencies
29+
- Files can't import from each other in a cycle
30+
31+
## 🚀 Usage
32+
33+
### Generate Code from Entry Point
34+
35+
```bash
36+
# Generate from accounts.lumos (will auto-discover types.lumos)
37+
lumos generate imports/accounts.lumos --output generated/
38+
39+
# Generate from instructions.lumos (will also discover types.lumos)
40+
lumos generate imports/instructions.lumos --output generated/
41+
```
42+
43+
### What Gets Generated
44+
45+
#### Rust Output (`generated.rs`)
46+
```rust
47+
// From types.lumos
48+
pub type UserId = solana_program::pubkey::Pubkey;
49+
pub type Lamports = u64;
50+
pub type TokenAmount = u64;
51+
// ... all type aliases
52+
53+
#[derive(BorshSerialize, BorshDeserialize, Debug, Clone)]
54+
pub enum AccountStatus {
55+
Uninitialized,
56+
Active,
57+
Frozen,
58+
Closed,
59+
}
60+
61+
// From accounts.lumos
62+
#[account]
63+
pub struct UserAccount {
64+
pub user_id: UserId, // Expands to Pubkey
65+
pub balance: Lamports, // Expands to u64
66+
pub token_balance: TokenAmount, // Expands to u64
67+
pub status: AccountStatus,
68+
// ...
69+
}
70+
```
71+
72+
#### TypeScript Output (`generated.ts`)
73+
```typescript
74+
// From types.lumos
75+
import { PublicKey } from '@solana/web3.js';
76+
77+
export type UserId = PublicKey;
78+
export type Lamports = number;
79+
export type TokenAmount = number;
80+
// ... all type aliases
81+
82+
export enum AccountStatus {
83+
Uninitialized = 0,
84+
Active = 1,
85+
Frozen = 2,
86+
Closed = 3,
87+
}
88+
89+
// From accounts.lumos
90+
export interface UserAccount {
91+
user_id: UserId; // = PublicKey
92+
balance: Lamports; // = number
93+
token_balance: TokenAmount; // = number
94+
status: AccountStatus;
95+
// ...
96+
}
97+
```
98+
99+
## 📝 Import Syntax
100+
101+
### Basic Import
102+
```rust
103+
import { UserId, Lamports } from "./types.lumos";
104+
```
105+
106+
### Multiple Imports
107+
```rust
108+
import {
109+
UserId,
110+
WalletAddress,
111+
Lamports,
112+
TokenAmount,
113+
AccountStatus
114+
} from "./types.lumos";
115+
```
116+
117+
### Relative Paths
118+
```rust
119+
// Same directory
120+
import { Type } from "./types.lumos";
121+
122+
// Parent directory
123+
import { Type } from "../types.lumos";
124+
125+
// Subdirectory
126+
import { Type } from "./shared/types.lumos";
127+
```
128+
129+
### Extension Optional
130+
```rust
131+
// Both work
132+
import { UserId } from "./types.lumos";
133+
import { UserId } from "./types";
134+
```
135+
136+
## 🎓 Key Concepts
137+
138+
### Type Resolution Order
139+
1. **Load**: LUMOS loads entry file and discovers imports
140+
2. **Parse**: All imported files are parsed
141+
3. **Collect**: Type aliases from ALL files are collected
142+
4. **Resolve**: Aliases are resolved recursively
143+
5. **Transform**: All files are transformed to IR with shared context
144+
6. **Generate**: Single output with all types
145+
146+
### Circular Import Detection
147+
```rust
148+
// ❌ This will fail
149+
// a.lumos
150+
import { B } from "./b.lumos";
151+
struct A { b: B }
152+
153+
// b.lumos
154+
import { A } from "./a.lumos";
155+
struct B { a: A }
156+
157+
// Error: Circular import detected: a.lumos -> b.lumos -> a.lumos
158+
```
159+
160+
### Type Alias Resolution
161+
```rust
162+
// types.lumos
163+
type UserId = PublicKey;
164+
type AccountId = UserId; // Alias of an alias
165+
type Creator = AccountId; // Nested alias
166+
167+
// accounts.lumos
168+
import { Creator } from "./types.lumos";
169+
170+
struct NFT {
171+
creator: Creator, // Fully resolved to PublicKey
172+
}
173+
```
174+
175+
## ✅ Benefits
176+
177+
### 1. **Better Organization**
178+
- Separate concerns: types, accounts, instructions
179+
- Smaller, focused files
180+
- Easier to navigate large projects
181+
182+
### 2. **Reusability**
183+
- Define common types once
184+
- Import where needed
185+
- No duplication
186+
187+
### 3. **Consistency**
188+
- Same type names across entire project
189+
- Single source of truth
190+
- Refactor once, updates everywhere
191+
192+
### 4. **Type Safety**
193+
- Import validation at compile time
194+
- Undefined type errors caught early
195+
- Circular dependency detection
196+
197+
### 5. **Team Collaboration**
198+
- Multiple developers can work on different files
199+
- Merge conflicts reduced
200+
- Clear module boundaries
201+
202+
## 🏗️ Project Structure Best Practices
203+
204+
### Small Projects
205+
```
206+
project/
207+
└── schema.lumos # Single file is fine
208+
```
209+
210+
### Medium Projects
211+
```
212+
project/
213+
├── types.lumos # Common types
214+
└── main.lumos # Main definitions
215+
```
216+
217+
### Large Projects
218+
```
219+
project/
220+
├── types/
221+
│ ├── common.lumos # Shared types
222+
│ ├── tokens.lumos # Token types
223+
│ └── governance.lumos # DAO types
224+
├── accounts/
225+
│ ├── user.lumos
226+
│ ├── vault.lumos
227+
│ └── dao.lumos
228+
└── instructions/
229+
├── token.lumos
230+
├── staking.lumos
231+
└── governance.lumos
232+
```
233+
234+
## 🐛 Common Issues
235+
236+
### Import Not Found
237+
```
238+
Error: Failed to read file './types.lumos': No such file or directory
239+
```
240+
**Solution**: Check relative path is correct from the importing file.
241+
242+
### Undefined Type
243+
```
244+
Error: Import error in 'accounts.lumos': type 'InvalidType' not found in './types.lumos'
245+
```
246+
**Solution**: Make sure the type is defined in types.lumos and is not just imported there.
247+
248+
### Circular Import
249+
```
250+
Error: Circular import detected: a.lumos -> b.lumos -> a.lumos
251+
```
252+
**Solution**: Restructure to move shared types to a third file.
253+
254+
## 📚 Related Examples
255+
256+
- [`../type_aliases.lumos`](../type_aliases.lumos) - Single-file type alias patterns
257+
- [`../custom_derives.lumos`](../custom_derives.lumos) - Custom derive macros
258+
- [`../enums/`](../enums/) - Enum variant patterns
259+
260+
## 💡 Next Steps
261+
262+
1. Try generating code from these examples
263+
2. Modify types.lumos and see changes propagate
264+
3. Create your own multi-file project structure
265+
4. Add more modules following these patterns
266+
267+
## 🤝 Contributing
268+
269+
Found an issue or have a suggestion? Please open an issue at:
270+
https://github.com/getlumos/lumos/issues

0 commit comments

Comments
 (0)