[Phase 2.1] Implement OSS/S3 backend using object_store

[zhexuany]

Summary

Implements full OSS/S3 storage backend using the object_store crate for Alibaba Cloud OSS and other S3-compatible storage services.

Changes

• OssConfig: Builder pattern for OSS configuration (bucket, endpoint, credentials, prefix, region, internal endpoint)
• OssStorage: Full Storage trait implementation with reader, writer, exists, size, metadata, list, delete, copy
• OssWriter: Buffered upload writer with configurable max buffer size and automatic upload on flush/drop
• StorageFactory: Updated to use OssConfig for instantiation

Test plan

• All 48 storage unit tests pass
• Clippy passes with -D warnings
• Build succeeds with --features cloud-storage
• Code formatted with cargo fmt

Environment Variables

OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_ENDPOINT (or ALIYUN_* prefixes)

URL Format

oss://bucket/path/to/file.txt?endpoint=oss-cn-hangzhou.aliyuncs.com&internal=true

Closes #13

[greptile-apps]

Greptile Overview

Greptile Summary

Implemented OSS/S3 storage backend using object_store crate, replacing stub implementation with working cloud storage support.

Key changes:

• Added OssConfig struct with builder pattern for configuration (bucket, endpoint, credentials, region, prefix)
• Implemented all Storage trait methods (reader, writer, exists, size, metadata, list, delete, copy)
• Created OssWriter that buffers writes and uploads on flush/drop
• Uses single-threaded tokio runtime with block_on for sync API over async object_store
• Added region support to StorageFactory for S3 compatibility

Critical issues:

• OssWriter auto-upload logic has a bug where data written after 100MB threshold may be lost (sets uploaded=true but continues accepting writes)
• Uses unstable let if syntax requiring Rust 1.87+ which may cause compilation failures
• Negative timestamp handling casts i64 to u64 causing incorrect dates
• endpoint_url() has dead code where use_internal_endpoint flag produces identical output in both branches

Confidence Score: 2/5

• not safe to merge due to data loss bug in writer and syntax incompatibility
• two critical bugs will cause production issues: OssWriter loses data after 100MB auto-upload, and let-chains syntax breaks compilation on stable Rust
• src/storage/oss.rs requires immediate fixes to writer logic (line 474) and Drop implementation (line 489)

Important Files Changed

Filename	Overview
src/storage/oss.rs	implemented OSS/S3 backend with object_store, but has critical bugs in writer auto-upload logic and unstable let-chains syntax
src/storage/factory.rs	integrated OssConfig into factory with region support, changes look correct
src/storage/mod.rs	exported OssConfig alongside OssStorage, trivial change

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant Factory as StorageFactory
    participant Config as OssConfig
    participant OssStorage as OssStorage
    participant Runtime as Tokio Runtime
    participant ObjectStore as object_store
    participant OSS as OSS/S3 Backend

    App->>Factory: create("oss://bucket/path")
    Factory->>Factory: parse URL
    Factory->>Factory: load credentials from env
    Factory->>Config: new(bucket, endpoint, key_id, key_secret)
    Factory->>Config: with_region(region)
    Factory->>OssStorage: with_config(config)
    
    OssStorage->>Runtime: create current_thread runtime
    OssStorage->>ObjectStore: AmazonS3Builder::new()
    ObjectStore->>ObjectStore: configure credentials & endpoint
    ObjectStore-->>OssStorage: Arc<dyn ObjectStore>
    OssStorage-->>Factory: OssStorage instance
    Factory-->>App: Arc<dyn Storage>

    Note over App,OSS: Read Operation
    App->>OssStorage: reader(path)
    OssStorage->>ObjectStore: get(key)
    ObjectStore->>Runtime: block_on(async get)
    Runtime->>OSS: HTTP GET request
    OSS-->>Runtime: object bytes
    Runtime-->>ObjectStore: GetResult
    ObjectStore->>Runtime: block_on(bytes())
    Runtime-->>OssStorage: Vec<u8>
    OssStorage-->>App: Cursor<Vec<u8>>

    Note over App,OSS: Write Operation
    App->>OssStorage: writer(path)
    OssStorage-->>App: OssWriter
    App->>OssWriter: write(data)
    OssWriter->>OssWriter: buffer.extend(data)
    opt buffer > 100MB
        OssWriter->>ObjectStore: put(key, payload)
        ObjectStore->>Runtime: block_on(async put)
        Runtime->>OSS: HTTP PUT request
        OSS-->>Runtime: success
    end
    App->>OssWriter: flush()
    OssWriter->>ObjectStore: put(key, payload)
    ObjectStore->>Runtime: block_on(async put)
    Runtime->>OSS: HTTP PUT request
    OSS-->>Runtime: success
    Runtime-->>OssWriter: Ok()

Loading

[greptile-apps]

_{3 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

auto-upload clears buffer but continues accepting writes

After auto-uploading at 100MB, buffer is cleared and uploaded=true, but subsequent write() calls will append to buffer without uploading again. This means data after 100MB may be lost.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/storage/oss.rs
Line: 474:476

Comment:
auto-upload clears buffer but continues accepting writes

After auto-uploading at 100MB, buffer is cleared and `uploaded=true`, but subsequent `write()` calls will append to buffer without uploading again. This means data after 100MB may be lost.

How can I resolve this? If you propose a fix, please make it concise.