Feature: S3 Remote Signing

[c-thiel]

Currently we do not respect the s3.remote-signing* attributes in FileIO.
It would be great to support remote-signing for access federation to MinIO like storages.

Follow-up of #505

[Xuanwo]

Copied my reply for this feature below:

I checked how iceberg-java implemetns remote-signing:

https://github.com/apache/iceberg/blob/f7585932a6d89b04c0d45b5f9dfe6f45483efd0b/aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java#L65-L76

It seems we need to implement this signer in iceberg-rust and pass it as a customized signer to opendal.

[flaneur2020]

can i take this issue?

[Xuanwo]

Of course, welcome!

[flaneur2020]

the pr which allows passing a customized Sign trait has already been merged into reqsign. however, there still needs some work to let it integrated into opendal.

it may need some more time before this issue is ready to be worked on.

sorry i'd unassign myself from this issue at the moment 🥲, i'll keep an eye on the progress and sync this issue when it's ready to be tackled again.

[Xuanwo]

Hi, @flaneur2020. Thank you for your work, and sorry for the delay with reqsign and opendal. I hope we can integrate them in a more organized way, which will lead to a significant refactor of the entire reqsign codebase. I'll ping you here once it's ready.

[delta003]

hii, can I help with this somehow?

If I understand the state currently: reqsign is going through a refactor which makes it possible to configure a signer (based on apache/opendal-reqsign#482). OpenDAL needs to pick up that and expose pluggable signer, so that iceberg-rust can implement remote signing in the same way as iceberg-{java, python}.

There are quite a lot of pieces involved, so I'm wondering: Do you see any way I can help with this that's not too disruptive to your other work? Or is it best to find a workaround?

[liurenjie1024]

hii, can I help with this somehow?

If I understand the state currently: reqsign is going through a refactor which makes it possible to configure a signer (based on Xuanwo/reqsign#482). OpenDAL needs to pick up that and expose pluggable signer, so that iceberg-rust can implement remote signing in the same way as iceberg-{java, python}.

There are quite a lot of pieces involved, so I'm wondering: Do you see any way I can help with this that's not too disruptive to your other work? Or is it best to find a workaround?

cc @Xuanwo Could you take a look?

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.