refactor(kafka-reader): Separates kafka constructs for better separation of concerns & reuse.

[owen-d]

Refactor Kafka Reader and Partition Committer

This PR refactors the Kafka ingestion pipeline components to achieve better separation of concerns and cleaner interfaces. The changes split the existing Reader into three distinct components:

1. A ReaderIfc that handles pure Kafka interactions
2. A partitionCommitter that manages async offset commits (already existed, but I've refactored it to use the interface)
3. A ReaderService that coordinates the overall lifecycle

Key Changes

Note: As of now, this just adds 3 new files with refactored versions -- if we're happy I'll replace the existing ones, but the current structure mirrors how I developed against the prior versions.

Disclaimer: I haven't hooked this up to testware yet; if we're happy with this direction, I'll do that next. edit: done.

New Reader Interface

Created a focused interface for Kafka operations:

type ReaderIfc interface {
    Topic() string
    Partition() int32
    ConsumerGroup() string
    FetchLastCommittedOffset(ctx context.Context) (int64, error)
    FetchPartitionOffset(ctx context.Context, position SpecialOffset) (int64, error)
    Poll(ctx context.Context) ([]Record, error)
    Commit(ctx context.Context, offset int64) error
    // Set the target offset for consumption. reads will begin from here.
    SetOffsetForConsumption(offset int64)
}

Improved Committer Design

• Moved the committer to depend on ReaderIfc instead of containing Kafka logic
• Maintains async commit functionality but delegates actual commits to the reader
• Cleaner separation between offset management and Kafka operations
• Added explicit SpecialOffset type to make partition offset fetching more type-safe

Service Lifecycle

• ReaderService now coordinates between the reader and committer
• Clearer separation of metrics between components
• More explicit error handling throughout

Benefits

• Maintainability: Responsibilities are clearly separated between Kafka operations, offset management, and service lifecycle
• Flexibility: The ReaderIfc can be used independently of the service wrapper, which is how I started down this path. I'd like to reuse the underlying reader in the block-builder code.