[MD][Meta] Support SigV4 auth types for data source

[noCharger]

No description provided.

[zhongnansu]

Problem Statement

Currently in Multiple Datasource we have supported these auth types

• No auth
• basic auth

We want to give user a way to add AWS OpenSearch domains with IAM policy as data sources, which means we need to support Sigv4 signing, as a new auth type

In general there're 3 types of IAM access policy.

1. Resource-based policies
2. Identity-based policies
3. IP-based policies

We'll introduce "resource based" IAM access policy in Multiple Datasource, because as the name indicates the "resources" are "data sources" in Multiple Datasource feature. And there're also 2 types of resource based access policy, the implementation will differ

1. User based: IAM user comes with IAM access key & secret key that can be used to sign the request.
2. Role based: Assume role with STS service on behalf of a master user, that internally generates "ACCESS KEY", "SECRET KEY" and "SESSION TOKEN" to authenticate the request. **We may not need to support this because it doesn't make sense for user to configure a master user then assume a role. User can just configure a IAM user in the policy and use approach 1 to access the domain **

POC

• research/poc on using https://www.npmjs.com/package/http-aws-es with client
• Set up test domain that enables IAM user/role in access policy
  • set up VPC test domain in aws opensearch [doc]
  • set up and programmatically access AWS Opensearch domain with IAM user-based access policy [doc]

     curl --request GET \
       'https://search-os-test-cognito-zdooa7o3mj3hg6tvpdibrt5k4u.us-west-2.es.amazonaws.com' \
       --aws-sigv4 aws:amz:us-west-2:es \
       --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
       --header 'Accept: application/json'
     {
       "name" : "45fb95e575edc5ee359680cd09594015",
       "cluster_name" : "841677925608:os-test-cognito",
       "cluster_uuid" : "A_x-L1mcQqi6ZoJt_3i3_Q",
       "version" : {
         "distribution" : "opensearch",
         "number" : "1.0.0",
         "build_type" : "tar",
         "build_hash" : "unknown",
         "build_date" : "2021-11-11T13:03:46.725100Z",
         "build_snapshot" : false,
         "lucene_version" : "8.8.2",
         "minimum_wire_compatibility_version" : "6.8.0",
         "minimum_index_compatibility_version" : "6.0.0-beta1"
       },
       "tagline" : "The OpenSearch Project: https://opensearch.org/"
     }
    

  • Set up IAM role-based access policy in test domain, and create user that can assume role and connect. [ref]
• poc using latest opensearch-js client with sigV4 natively supported [doc link]
  • Use hardcoded IAM access key & secret key (through AWS credential object) + opensearch client to connect to AOS domains
  • Use aws-sdk sts client + opensearch-js client

Implementation Tasks

• support SigV4 with User based IAM access policy [MD] Support connection to AWS OpenSearch domain with user based Sigv4 #2921
• support SigV4 with Role based IAM access policy [may not implement]
• [Multiple Datasource] Integrate SigV4 auth with test connection feature #3418
• Improve testing(UT, IT)
• documentation website update: [DOC] Document a new SigV4 authentication type in Dashboards Multiple Datasource documentation-website#2667

[zhongnansu]

demo video with PR #3058

Screen.Recording.2023-02-08.at.12.46.11.PM.mov

@KrooshalUX please leave UI/UX feedbacks on this thread

[KrooshalUX]

@zhongnansu This is a great demo, thanks for putting it together and syncing earlier.

Two small bits of feedback:

• I recommend removing the confirmations in the update password UI on the object details page (also as @kgcreative mentioned in the PR)
• A nice to have to consider is converting the authentication type radio buttons to OuiSelect since we now have more than 2. I understand that currently the project only has the 3 and tbd on plans for adding more, so this is not blocking feedback as it can be added in a future iteration as well.

Great job team!

[zhongnansu]

Updated UI for credential update modal

[zhongnansu]

Reopen. Not all task are completed

[zhongnansu]

This can be closed, as we released this feature in 2.6.0