feat!(clp-package): Add support for providing multiple S3 object keys as input for compression.

[LinZhihao-723]

Description

Before this PR, the CLP package compression only accepted a single S3 URL as the input. The S3 URL is decomposed into a bucket, a region code, and a key prefix. Everything under the key prefix will be compressed.

This PR adds support for providing multiple S3 object keys as input for compression. Now, users can specify a list of S3 URLs, where each URL is treated as a URL to an actual key in the bucket. Key-prefix-based ingestion is still supported, however, users must explicitly specify that the input is a prefix by using --s3-single-prefix option.

The current implementation has the following requirements for the given input URLs:

• All the specified objects must belong to the same bucket.
• All the specified objects must be in the same region.
• The keys must share a common prefix.
• There should be no duplicate keys.

To support multi-keys, we update S3InputConfig to store an optional keys field. If this field is not set, prefix-based ingestion will be used as before. Otherwise, we will traverse the bucket to collect object metadata of the given keys to create compression jobs.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.
• Tested locally that prefix-based ingestion still works as previously.
• Tested locally that with a list of valid keys given, they can be compressed as expected.
• Tested locally that missing keys will be properly reported.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[LinZhihao-723]

Please check whether this flag name makes sense @kirkrodrigues @junhaoliao

[LinZhihao-723]

Please check whether passing keys in this way makes sense @kirkrodrigues @junhaoliao.
In eithere key ingestion or prefix ingestion, the key prefix will be required so S3Config base is untouched.

[junhaoliao]

(I haven't taken a very careful look at the implementation. sorry if i misunderstood anything.)

In what case do we expect the keys to be None and when not? I'm guessing that
1. Non-None and Non-empty: When the input config is created by an S3 scanner, or (to be implemented, ) when the user provides multiple s3 keys when submitting a compression job.
2. None: when the user only provides a single s3 prefix when submitting a compression job

Since this is a list where a length of 0 is allowed, do you think we can make it non-optional?

[LinZhihao-723]

You're right. But actually length of 0 is not allowed. With Pydantic, shall we make a validator?