feat: add background compaction for Lance fragments

[beinan]

Background

Long-running agents append many small fragments to the Lance dataset. Without compaction, read performance degrades and storage costs increase.

Proposal

• Introduce an asynchronous compaction service that periodically merges small fragments once thresholds are exceeded (size, count, or age).
• Expose tuning knobs via configuration (e.g., target fragment size, max concurrent compactions, quiet hours).
• Provide a manual trigger API for hosts that prefer explicit control.

Design Notes

• Reuse Lance's existing optimize/compaction APIs to avoid reimplementing fragment merging logic.
• Run the background job in a separate Tokio task so writes are non-blocking; gate work behind a rate limiter to avoid competing with foreground operations.
• Emit metrics/logs so operators can observe compaction frequency and latency; integrate with tracing.

Acceptance Criteria

• Small-fragment datasets automatically consolidate after sustained ingestion, improving scan throughput in benchmarks.
• Compaction respects configured thresholds and can be disabled for environments with external schedulers.
• Documentation outlines operational guidance and default settings.