Add AWS overlay for production users

[Jeffwan]

Currently, platform-agnostic works well on AWS for user who like to experience KFP features. We(AWS) get some feedbacks users want to use more stable AWS services to replace some of the components in KFP. S3 and RDS start to be widely voiced.

I think pipeline team build good abstraction here, for example, GCP users can easily build minio-gcs-gateway and cloudsql proxy to override base manifest.

In AWS, we'd like to adopt similar solution, minio-s3-gateway is straighforward, however, RDS-proxy recently released is kind of complex to setup and it raise bar to setup KFP in production environment.

We probably like to plug RDS to KFP components directly if sql proxy is not that straightforward on AWS. There're some work already one in Kubeflow v1.0.x. Just create this issue to track our work on standalone manifest.

• https://www.kubeflow.org/docs/aws/pipeline/
• https://www.kubeflow.org/docs/aws/rds/

Please leave comments or suggestions and we can have more discussion.

/cc @PatrickXYS

[PatrickXYS]

/assign

[Jeffwan]

Jeffwan@9ae0464
I make some changes and this will help replace minio if user doesn't like to use gateway. However, the client sdk are still minio which means user may not leverage latest S3 features.

[Bobgy]

FYI, there's a community maintained AWS overlay https://github.com/e2fyi/kubeflow-aws/tree/master/pipelines

[Jeffwan]

@Bobgy Yeah, I talked with @eterna2 in the past. I think we would like to leverage upstream base as more as possible. We will keep link to e2fyi as alternative. I know some users are already on it. At the same time, we will try to further simplify the setups and add cloud resources in the scope. for example, one CFN/ deploy manager to help setup aws resources