Website: add ledger-crate.md from handover

Author: dannywillems

website/docs/developers/ledger-crate.md

@@ -0,0 +1,309 @@
+---
+sidebar_position: 5
+title: Ledger Crate Architecture
+description:
+  Comprehensive architecture overview of the ledger crate implementation
+slug: /developers/ledger-crate
+---
+
+# Ledger Crate Architecture
+
+## Overview
+
+The [`ledger`](https://o1-labs.github.io/mina-rust/api-docs/ledger/index.html)
+crate is a comprehensive Rust implementation of the Mina protocol's ledger,
+transaction pool, staged ledger, scan state, proof verification, and zkApp
+functionality, providing a direct port of the OCaml implementation. For
+developers familiar with the OCaml codebase, this maintains the same
+architecture and business logic while adapting to Rust idioms.
+
+This document provides an in-depth architectural overview complementing the
+[API documentation](https://o1-labs.github.io/mina-rust/api-docs/ledger/index.html).
+
+## Architecture
+
+### Core Components
+
+#### BaseLedger Trait (`src/base.rs`)
+
+- Direct mapping to OCaml's `Ledger_intf.S`
+- Defines the fundamental ledger interface for account management, Merkle tree
+  operations, and state queries
+- All ledger implementations (Database, Mask) implement this trait
+- Provides consistent interface across different ledger storage backends
+
+#### Mask System (`src/mask/`)
+
+- Port of OCaml's `Ledger.Mask` with identical copy-on-write semantics
+- Provides layered ledger views for efficient state management
+- Uses `Arc<Mutex<MaskImpl>>` for cheap reference counting; `Mask::clone()` is
+  fast
+- Used extensively in transaction processing to create temporary ledger states
+- Enables efficient branching and merging of ledger states during block
+  processing
+
+#### Database (`src/database/`)
+
+- In-memory implementation (ondisk module exists but is not used)
+- Corresponds to OCaml's `Ledger.Db` interface
+- Handles account storage and Merkle tree management
+- Optimized for high-performance account lookups and updates
+
+### Transaction Processing
+
+#### Transaction Pool (`src/transaction_pool.rs`)
+
+Complete port of `Transaction_pool.Indexed_pool` with identical behavior:
+
+- **Fee-based transaction ordering**: Higher fee transactions prioritized
+- **Sender queue management**: Nonce-based ordering per sender
+- **Revalidation on best tip changes**: Ensures transaction validity
+- **VkRefcountTable**: Verification key reference counting for memory efficiency
+- **Mempool operations**: Transaction addition, removal, and replacement logic
+- **Expiration handling**: Automatic cleanup of stale transactions
+
+#### Staged Ledger (`src/staged_ledger/`)
+
+Maps directly to OCaml's staged ledger implementation:
+
+- **Diff structures**: Correspond to `Staged_ledger_diff` with same partitioning
+- **Transaction application**: Manages sequential application of transactions
+- **Block validation**: Ensures blocks meet protocol requirements
+- **Fee collection**: Handles coinbase and fee distribution
+- **State transitions**: Manages ledger state evolution during block processing
+
+#### Scan State (`src/scan_state/`)
+
+Port of the parallel scan state system:
+
+- **SNARK work coordination**: Manages proof generation across the network
+- **Parallel scan trees**: Efficient proof aggregation structures
+- **Work distribution**: Coordinates SNARK worker assignments
+- **Proof verification**: Validates submitted SNARK work
+- **State management**: Tracks scan state evolution across blocks
+
+### Proof System Integration
+
+#### Proofs Module (`src/proofs/`)
+
+Comprehensive proof generation and verification:
+
+- **Transaction proofs**: User command and payment verification
+- **Block proofs**: Blockchain state transition validation
+- **zkApp proofs**: Zero-knowledge application proof handling
+- **Merge proofs**: Efficient proof aggregation
+- **Verification**: Fast proof validation using precomputed indices
+
+#### Sparse Ledger (`src/sparse_ledger/`)
+
+- Minimal ledger representation optimized for proof generation
+- Contains only accounts needed for specific proof context
+- Reduces proof generation overhead by eliminating unnecessary data
+- Maintains cryptographic integrity through Merkle path verification
+
+### zkApp Support
+
+#### zkApps Module (`src/zkapps/`)
+
+Complete zkApp transaction processing:
+
+- **Account updates**: Manages complex multi-account operations
+- **Authorization handling**: Validates zkApp permissions and signatures
+- **State management**: Handles zkApp on-chain state updates
+- **Event emission**: Processes zkApp events and sequencing
+- **Precondition validation**: Ensures zkApp execution prerequisites
+
+### Account Management
+
+#### Account Structures (`src/account/`)
+
+- **Account types**: Standard accounts, zkApp accounts, token accounts
+- **Balance management**: Handles account balances and locked funds
+- **Permission systems**: Manages account access controls and delegates
+- **Token support**: Native support for custom tokens
+- **Timing constraints**: Handles vesting and unlock schedules
+
+#### Address Management (`src/address/`)
+
+- **Public key handling**: Account identification and verification
+- **Address derivation**: Deterministic address generation
+- **Token address mapping**: Links token accounts to parent accounts
+
+## Implementation Details
+
+### Memory Management
+
+#### Arc-based Sharing
+
+- Extensive use of `Arc<T>` for efficient memory sharing
+- Copy-on-write semantics through mask system
+- Minimizes memory overhead during state transitions
+
+#### Caching Strategies
+
+- Merkle tree path caching for performance
+- Account lookup optimization
+- Proof verification result caching
+
+### Concurrency Model
+
+#### Thread Safety
+
+- `Mutex` protection for mutable state
+- Lock-free operations where possible
+- Careful ordering to prevent deadlocks
+
+#### Parallel Processing
+
+- SNARK work can be processed in parallel
+- Transaction validation optimizations
+- Block application pipeline efficiency
+
+### Performance Optimizations
+
+#### Fast Account Lookups
+
+- Efficient hash-based account storage
+- Merkle tree optimization for batch operations
+- Memory-mapped storage preparation (ondisk module)
+
+#### Transaction Processing
+
+- Batch transaction validation
+- Optimized fee calculation
+- Efficient nonce management
+
+## Protocol Compliance
+
+### OCaml Compatibility
+
+- Identical business logic to OCaml implementation
+- Same data structures and serialization formats
+- Compatible proof generation and verification
+- Consistent state transition behavior
+
+### Network Interoperability
+
+- Full compatibility with OCaml nodes
+- Identical block validation rules
+- Same transaction pool behavior
+- Compatible SNARK work distribution
+
+## Development Guidelines
+
+### Working with Ledger Code
+
+#### Understanding Masks
+
+```rust
+// Create a new mask for temporary operations
+let temp_mask = ledger.mask().unwrap();
+
+// Make modifications without affecting parent
+temp_mask.create_account(account_id, account)?;
+
+// Commit changes back to parent or discard
+temp_mask.commit();
+```
+
+#### Transaction Processing Patterns
+
+```rust
+// Typical transaction application flow
+let staged_ledger = StagedLedger::create(ledger);
+let diff = staged_ledger.apply_diff(transactions)?;
+let new_ledger = staged_ledger.apply(diff)?;
+```
+
+#### Account Management
+
+```rust
+// Safe account operations
+if let Some(account) = ledger.get_account(&account_id)? {
+    let updated = account.set_balance(new_balance)?;
+    ledger.set_account(&account_id, &updated)?;
+}
+```
+
+### Testing Strategies
+
+#### Unit Tests
+
+- Individual component testing (masks, accounts, transactions)
+- Property-based testing for invariants
+- Compatibility tests against OCaml reference
+
+#### Integration Tests
+
+- End-to-end transaction processing
+- Block application and validation
+- SNARK work coordination
+
+#### Performance Tests
+
+- Memory usage validation
+- Transaction throughput benchmarks
+- Proof generation performance
+
+## Known Limitations
+
+### Technical Debt Areas
+
+#### Error Handling
+
+- Extensive use of `.unwrap()` and `.expect()` in well-understood code paths
+- Opportunities for more explicit error propagation
+- Inconsistent error handling patterns across modules
+
+#### Code Organization
+
+- Some large files with multiple responsibilities
+- Opportunities for better separation of concerns
+- Legacy patterns from OCaml port that could be Rust-ified
+
+#### Performance Opportunities
+
+- Memory allocation optimization potential
+- Further parallelization possibilities
+- Caching strategy improvements
+
+### Future Improvements
+
+#### Persistence Integration
+
+- Integration with planned persistence layer
+- Disk-based storage backend completion
+- Efficient state recovery mechanisms
+
+#### Performance Enhancements
+
+- Lock contention reduction
+- Memory usage optimization
+- Parallel transaction processing
+
+#### Code Quality
+
+- Error handling improvements
+- Module decomposition
+- Better separation of concerns
+
+## Related Documentation
+
+- [Ledger Crate API Docs](https://o1-labs.github.io/mina-rust/api-docs/ledger/index.html):
+  Complete API reference
+- [Circuits Documentation](circuits): Circuit generation and proof system
+  integration
+- [Persistence Design](persistence-design): Future storage layer plans
+- [SNARK Work](../researchers/snark-work): Protocol-level SNARK work details
+
+## Conclusion
+
+The ledger crate represents the heart of the Mina Rust node, implementing all
+core ledger functionality with full protocol compliance. While maintaining
+compatibility with the OCaml implementation, it provides the foundation for
+efficient transaction processing, proof generation, and zkApp execution.
+
+The architecture balances performance, correctness, and maintainability, though
+opportunities exist for continued refinement as the codebase matures.
+Understanding this crate is essential for developers working on any aspect of
+the Mina Rust node's core functionality.

website/sidebars.ts

@@ -68,6 +68,7 @@ const sidebars: SidebarsConfig = {
         'developers/why-rust',
         'developers/architecture',
         'developers/circuits',
+        'developers/ledger-crate',
       ],
     },
     {