|
| 1 | +# GitHub Copilot Instructions for Nydus |
| 2 | + |
| 3 | +## Project Overview |
| 4 | + |
| 5 | +Nydus is a high-performance container image service that implements a content-addressable file system on the RAFS format. It enhances the OCI image specification by enabling on-demand loading, chunk-level deduplication, and improved container startup performance. |
| 6 | + |
| 7 | +### Key Components |
| 8 | + |
| 9 | +- **nydusd**: User-space daemon that processes FUSE/fscache/virtiofs messages and serves Nydus images |
| 10 | +- **nydus-image**: CLI tool to convert OCI image layers to Nydus format |
| 11 | +- **nydusify**: Tool to convert entire OCI images to Nydus format with registry integration |
| 12 | +- **nydusctl**: CLI client for managing and querying nydusd daemon |
| 13 | +- **nydus-service**: Library crate for integrating Nydus services into other projects |
| 14 | + |
| 15 | +## Architecture Guidelines |
| 16 | + |
| 17 | +### Crate Structure |
| 18 | +``` |
| 19 | +- api/ # Nydus Image Service APIs and data structures |
| 20 | +- builder/ # Image building and conversion logic |
| 21 | +- rafs/ # RAFS filesystem implementation |
| 22 | +- service/ # Daemon and service management framework |
| 23 | +- storage/ # Core storage subsystem with backends and caching |
| 24 | +- utils/ # Common utilities and helper functions |
| 25 | +- src/bin/ # Binary executables (nydusd, nydus-image, nydusctl) |
| 26 | +``` |
| 27 | + |
| 28 | +### Key Technologies |
| 29 | +- **Language**: Rust with memory safety focus |
| 30 | +- **Filesystems**: FUSE, virtiofs, EROFS, fscache |
| 31 | +- **Storage Backends**: Registry, OSS, S3, LocalFS, HTTP proxy |
| 32 | +- **Compression**: LZ4, Gzip, Zstd |
| 33 | +- **Async Runtime**: Tokio (current thread for io-uring compatibility) |
| 34 | + |
| 35 | +## Code Style and Patterns |
| 36 | + |
| 37 | +### Rust Conventions |
| 38 | +- Use `#![deny(warnings)]` in all binary crates |
| 39 | +- Follow standard Rust naming conventions (snake_case, PascalCase) |
| 40 | +- Prefer `anyhow::Result` for error handling in applications |
| 41 | +- Use custom error types with `thiserror` for libraries |
| 42 | +- Apply `#[macro_use]` for frequently used external crates like `log` |
| 43 | +- Always format the code with `cargo fmt` |
| 44 | +- Use `clippy` for linting and follow its suggestions |
| 45 | + |
| 46 | +### Error Handling |
| 47 | +```rust |
| 48 | +// Prefer anyhow for applications |
| 49 | +use anyhow::{bail, Context, Result}; |
| 50 | + |
| 51 | +// Use custom error types for libraries |
| 52 | +use thiserror::Error; |
| 53 | + |
| 54 | +#[derive(Error, Debug)] |
| 55 | +pub enum NydusError { |
| 56 | + #[error("Invalid arguments: {0}")] |
| 57 | + InvalidArguments(String), |
| 58 | + #[error("IO error: {0}")] |
| 59 | + Io(#[from] std::io::Error), |
| 60 | +} |
| 61 | +``` |
| 62 | + |
| 63 | +### Logging Patterns |
| 64 | +- Use structured logging with appropriate levels (trace, debug, info, warn, error) |
| 65 | +- Include context in error messages: `.with_context(|| "description")` |
| 66 | +- Use `info!`, `warn!`, `error!` macros consistently |
| 67 | + |
| 68 | +### Configuration Management |
| 69 | +- Use `serde` for JSON configuration serialization/deserialization |
| 70 | +- Support both file-based and environment variable configuration |
| 71 | +- Validate configurations at startup with clear error messages |
| 72 | +- Follow the `ConfigV2` pattern for versioned configurations |
| 73 | + |
| 74 | +## Development Guidelines |
| 75 | + |
| 76 | +### Storage Backend Development |
| 77 | +When implementing new storage backends: |
| 78 | +- Implement the `BlobBackend` trait |
| 79 | +- Support timeout, retry, and connection management |
| 80 | +- Add configuration in the backend config structure |
| 81 | +- Consider mirror/proxy support for high availability |
| 82 | +- Implement proper error handling and logging |
| 83 | + |
| 84 | +### Daemon Service Development |
| 85 | +- Use the `NydusDaemon` trait for service implementations |
| 86 | +- Support save/restore for hot upgrade functionality |
| 87 | +- Implement proper state machine transitions |
| 88 | +- Use `DaemonController` for lifecycle management |
| 89 | + |
| 90 | +### RAFS Filesystem Features |
| 91 | +- Support both RAFS v5 and v6 formats |
| 92 | +- Implement chunk-level deduplication |
| 93 | +- Handle prefetch optimization for container startup |
| 94 | +- Support overlay filesystem operations |
| 95 | +- Maintain POSIX compatibility |
| 96 | + |
| 97 | +### API Development |
| 98 | +- Use versioned APIs (v1, v2) with backward compatibility |
| 99 | +- Implement HTTP endpoints with proper error handling |
| 100 | +- Support both Unix socket and TCP communication |
| 101 | +- Follow OpenAPI specification patterns |
| 102 | + |
| 103 | +## Testing Patterns |
| 104 | + |
| 105 | +### Unit Tests |
| 106 | +- Test individual functions and modules in isolation |
| 107 | +- Use `#[cfg(test)]` modules within source files |
| 108 | +- Mock external dependencies when necessary |
| 109 | +- Focus on error conditions and edge cases |
| 110 | + |
| 111 | +### Integration Tests |
| 112 | +- Place integration tests in `tests/` directory |
| 113 | +- Test complete workflows and component interactions |
| 114 | +- Use temporary directories for filesystem operations |
| 115 | +- Clean up resources properly in test teardown |
| 116 | + |
| 117 | +### Smoke Tests |
| 118 | +- Located in `smoke/` directory using Go |
| 119 | +- Test real-world scenarios with actual images |
| 120 | +- Verify performance and functionality |
| 121 | +- Use Bats framework for shell-based testing |
| 122 | + |
| 123 | +## Performance Considerations |
| 124 | + |
| 125 | +### I/O Optimization |
| 126 | +- Use async I/O patterns with Tokio |
| 127 | +- Implement prefetching for predictable access patterns |
| 128 | +- Optimize chunk size (default 1MB) for workload characteristics |
| 129 | +- Consider io-uring for high-performance scenarios |
| 130 | + |
| 131 | +### Memory Management |
| 132 | +- Use `Arc<T>` for shared ownership of large objects |
| 133 | +- Implement lazy loading for metadata structures |
| 134 | +- Consider memory mapping for large files |
| 135 | +- Profile memory usage in performance-critical paths |
| 136 | + |
| 137 | +### Caching Strategy |
| 138 | +- Implement blob caching with configurable backends |
| 139 | +- Support compression in cache to save space |
| 140 | +- Use chunk-level caching with efficient eviction policies |
| 141 | +- Consider cache warming strategies for frequently accessed data |
| 142 | + |
| 143 | +## Security Guidelines |
| 144 | + |
| 145 | +### Data Integrity |
| 146 | +- Implement end-to-end digest validation |
| 147 | +- Support multiple hash algorithms (SHA256, Blake3) |
| 148 | +- Verify chunk integrity on read operations |
| 149 | +- Detect and prevent supply chain attacks |
| 150 | + |
| 151 | +### Authentication |
| 152 | +- Support registry authentication (basic auth, bearer tokens) |
| 153 | +- Handle credential rotation and refresh |
| 154 | +- Implement secure credential storage |
| 155 | +- Support mutual TLS for backend connections |
| 156 | + |
| 157 | +## Specific Code Patterns |
| 158 | + |
| 159 | +### Configuration Loading |
| 160 | +```rust |
| 161 | +// Standard pattern for configuration loading |
| 162 | +let config = match config_path { |
| 163 | + Some(path) => ConfigV2::from_file(path)?, |
| 164 | + None => ConfigV2::default(), |
| 165 | +}; |
| 166 | + |
| 167 | +// Environment variable override |
| 168 | +if let Ok(auth) = std::env::var("IMAGE_PULL_AUTH") { |
| 169 | + config.update_registry_auth_info(&auth); |
| 170 | +} |
| 171 | +``` |
| 172 | + |
| 173 | +### Daemon Lifecycle |
| 174 | +```rust |
| 175 | +// Standard daemon initialization pattern |
| 176 | +let daemon = create_daemon(config, build_info)?; |
| 177 | +DAEMON_CONTROLLER.set_daemon(daemon); |
| 178 | + |
| 179 | +// Event loop management |
| 180 | +if DAEMON_CONTROLLER.is_active() { |
| 181 | + DAEMON_CONTROLLER.run_loop(); |
| 182 | +} |
| 183 | + |
| 184 | +// Graceful shutdown |
| 185 | +DAEMON_CONTROLLER.shutdown(); |
| 186 | +``` |
| 187 | + |
| 188 | +### Blob Access Pattern |
| 189 | +```rust |
| 190 | +// Standard blob read pattern |
| 191 | +let mut bio = BlobIoDesc::new(blob_id, blob_address, blob_size, user_io); |
| 192 | +let blob_device = factory.get_device(&blob_info)?; |
| 193 | +blob_device.read(&mut bio)?; |
| 194 | +``` |
| 195 | + |
| 196 | +## Documentation Standards |
| 197 | + |
| 198 | +### Code Documentation |
| 199 | +- Document all public APIs with `///` comments |
| 200 | +- Include examples in documentation |
| 201 | +- Document safety requirements for unsafe code |
| 202 | +- Explain complex algorithms and data structures |
| 203 | + |
| 204 | +### Architecture Documentation |
| 205 | +- Maintain design documents in `docs/` directory |
| 206 | +- Update documentation when adding new features |
| 207 | +- Include diagrams for complex interactions |
| 208 | +- Document configuration options comprehensively |
| 209 | + |
| 210 | +### Release Notes |
| 211 | +- Document breaking changes clearly |
| 212 | +- Include migration guides for major versions |
| 213 | +- Highlight performance improvements |
| 214 | +- List new features and bug fixes |
| 215 | + |
| 216 | +## Container and Cloud Native Patterns |
| 217 | + |
| 218 | +### OCI Compatibility |
| 219 | +- Maintain compatibility with OCI image spec |
| 220 | +- Support standard container runtimes (runc, Kata) |
| 221 | +- Implement proper layer handling and manifest generation |
| 222 | +- Support multi-architecture images |
| 223 | + |
| 224 | +### Kubernetes Integration |
| 225 | +- Design for Kubernetes CRI integration |
| 226 | +- Support containerd snapshotter pattern |
| 227 | +- Handle pod lifecycle events appropriately |
| 228 | +- Implement proper resource cleanup |
| 229 | + |
| 230 | +### Cloud Storage Integration |
| 231 | +- Support major cloud providers (AWS S3, Alibaba OSS) |
| 232 | +- Implement proper credential management |
| 233 | +- Handle network interruptions gracefully |
| 234 | +- Support cross-region replication patterns |
| 235 | + |
| 236 | +## Build and Release |
| 237 | + |
| 238 | +### Build Configuration |
| 239 | +- Use `Cargo.toml` workspace configuration |
| 240 | +- Support cross-compilation for multiple architectures |
| 241 | +- Implement proper feature flags for optional components |
| 242 | +- Use consistent dependency versioning |
| 243 | + |
| 244 | +### Release Process |
| 245 | +- Tag releases with semantic versioning |
| 246 | +- Generate release binaries for supported platforms |
| 247 | +- Update documentation with release notes |
| 248 | +- Validate release artifacts before publishing |
| 249 | + |
| 250 | +Remember to follow these guidelines when contributing to or working with the Nydus codebase. The project emphasizes performance, security, and compatibility with the broader container ecosystem. |
0 commit comments