Add option to use LakeFormation in S3Glue data source.

[asuresh8]

Description

Enables OpenSearch users to use LakeFormation option when querying the S3Glue data source.
This is important because some Glue tables are managed by LakeFormation. The customer will need to pass in a property for LakeFormation in order to not break backwards compataibility.

Note that this feature is launching in EMR serverless next month, so an error is thrown. Adding it in now so it's in the next release which is in line with the launch of this feature on EMR serverless.

Testing

Built locally with 2.13 as version and then copied over to cluster running 2.13.

Validating that property is read and correct configuration is added to Spark config

curl --request POST  --url http://localhost:9200/_plugins/_query/_datasources   --header 'content-type: application/x-ndjson'   --data '{"name": "mygdc","description": "","connector": "S3GLUE","allowedRoles": [],"properties": {"glue.auth.type": "iam_role","glue.auth.role_arn": "arn:aws:iam::xxxxxxxxxxxx:role/flint-opensearch-execution-role","glue.indexstore.opensearch.uri": "http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:9200","glue.indexstore.opensearch.auth": "noauth", "glue.lakeformation.enabled": "true"}}'
curl --request  POST   --url http://localhost:9200/_plugins/_async_query   --header 'content-type: application/x-ndjson'   --data '{"datasource": "mygdc","lang": "sql","query": "SELECT * FROM mygdc.amazon_security_lake_glue_db_us_east_1.amazon_security_lake_table_us_east_1_vpc_flow_2_0 LIMIT 1"}'
{
  "status": 500,
  "error": {
    "type": "RuntimeException",
    "reason": "There was internal problem at backend",
    "details": "Internal Server Error."
  }
cat ~/tmp/opensearch.log 
com.amazonaws.services.emrserverless.model.ValidationException: Lake Formation configuration can only be specified at the application level, not at job run level. (Service: AWSEMRServerless; Status Code: 400; Error Code: ValidationException; Request ID: 28566859-cdfc-45fa-bb0a-21ad69d0a88a; Proxy: null)

Validating backwards compatibility

curl --request POST   --url http://localhost:9200/_plugins/_query/_datasources   --header 'content-type: application/x-ndjson'   --data '{"name": "mygdc2","description": "","connector": "S3GLUE","allowedRoles": [],"properties": {"glue.auth.type": "iam_role","glue.auth.role_arn": "arn:aws:iam::xxxxxxxxxx:role/flint-opensearch-execution-role","glue.indexstore.opensearch.uri": "http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:9200","glue.indexstore.opensearch.auth": "noauth"}}'
curl --request  POST   --url http://localhost:9200/_plugins/_async_query   --header 'content-type: application/x-ndjson'   --data '{"datasource": "mygdc2","lang": "sql","query": "SELECT * FROM mygdc2.amazon_security_lake_glue_db_us_east_1.amazon_security_lake_table_us_east_1_vpc_flow_2_0 LIMIT 1"}'
{
  "queryId": "b1d5WnhqYkdoMm15Z2RjMg==",
  "sessionId": "eFhDVk85ZUZMMG15Z2RjMg=="
}
curl --request  GET --url http://localhost:9200/_plugins/_async_query/b1d5WnhqYkdoMm15Z2RjMg==
{
  "status": "SUCCESS",
   ...
}

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vamsimanohar]

@asuresh8 can you update the docs over here with the new property: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.37%. Comparing base (2c97be7) to head (1c8998e).
Report is 1 commits behind head on main.

❗ Current head 1c8998e differs from pull request most recent head 08ccd67. Consider uploading reports for the commit 08ccd67 to get more accurate results

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2624   +/-   ##
=========================================
  Coverage     95.37%   95.37%           
  Complexity     5133     5133           
=========================================
  Files           490      490           
  Lines         14431    14434    +3     
  Branches        971      971           
=========================================
+ Hits          13763    13766    +3     
  Misses          643      643           
  Partials         25       25           

Flag	Coverage Δ
sql-engine	95.37% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dai-chen]

This new config is configurable for open source user, right? Could you update user manual if so, maybe here https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst?

[dai-chen]

I assume these account and domain info are internal? Can we replace with fake one for test?

[asuresh8]

It looks like these values are used for all tests. I can update as part of a follow-up PR.

[dai-chen]

As discussed earlier, we may need to disable covering index rewrite if LF enabled (until better solution): https://github.com/opensearch-project/opensearch-spark/pull/318/files#diff-cf86f425a16363ac48fd59d28648420bc5c961b1b472edfb54075360633be338R136

[asuresh8]

This new config is configurable for open source user, right? Could you update user manual if so, maybe here https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst?

Done

As discussed earlier, we may need to disable covering index rewrite if LF enabled (until better solution):

Done

[dai-chen]

This new config is configurable for open source user, right? Could you update user manual if so, maybe here https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst?

Done

As discussed earlier, we may need to disable covering index rewrite if LF enabled (until better solution):

Done

Thanks a lot! Will get my PR merged soon.