Support defining bucket ownership

[dlvenable]

Is your feature request related to a problem? Please describe.

The S3 source validates that an S3 bucket is owned by the same account as the SQS queue. This can protect against reading from buckets in unknown accounts. This approach uses S3's bucket ownership verification.

Pipeline authors can disable this by setting disable_bucket_ownership_validation to true. This completely disables the bucket ownership validation. There is no way to validate that buckets are owned by specific accounts other than the SQS queue account.

Describe the solution you'd like

Provide two options for bucket validation:

• bucket_owners - A simple map of bucket name to expected owner.
• default_bucket_owner - A scalar value with an accountId to use for any bucket not in the map above. If specified, this will override the SQS accountId.

s3:
  sqs:
      queue_url: "https://sqs.us-east-1.amazonaws.com/000000000000/MyQueue"
  bucket_owners:
    my-bucket-01: 123456789012
    my-bucket-02: 99999999999
  default_bucket_owner: 111111111111

In the example above, the S3 source will set an expectation that my-bucket-01 is owned by 123456789012. It will expect that my-bucket-02 is owned by 99999999999. It would expect that any other bucket (say my-bucket-02) is owned by 111111111111. It will never expect any bucket to be owned by the SQS queue - 000000000000.

Describe alternatives you've considered (Optional)

The existing functionality allows for skipping validation. So it is possible that no additional functionality is needed. But, then there is no bucket validation. However, the S3 documentation recommends validation: We recommend using bucket owner condition whenever you perform a supported S3 operation and know the account ID of the expected bucket owner.

Additional context

Original PR adding the current functionality: #1526

[sharraj]

Alternatively we should consider option where customer wants to disable this check completely instead of specifying account IDs. We give control to customer to do this based on their deployment needs. Like just 1 knob s3-bucket-sqs-ownership-check: disable.

[dlvenable]

Alternatively we should consider option where customer wants to disable this check completely instead of specifying account IDs. We give control to customer to do this based on their deployment needs. Like just 1 knob s3-bucket-sqs-ownership-check: disable.

@sharraj , This is already available as the disable_bucket_ownership_validation configuration. However, this is not recommended as S3 suggests checking the bucket owner to improve security.

https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/s3/#configuration