Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

V2 migration glue #2052

Merged
merged 26 commits into from
Jul 22, 2024
Merged

V2 migration glue #2052

merged 26 commits into from
Jul 22, 2024

Conversation

aimethed
Copy link
Contributor

Description of changes:
Migrate AWS Glue to v2. Part of the larger effort to migrate the codebase to AWS SDK v2 for java.

Also includes dependency reductions to shrink down the jar/zip size for hbase and bigquery.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

aimethed and others added 21 commits May 22, 2024 15:09
@aimethed
update secrets manager to v2 in sdk
d5f5b8f
@aimethed
update ~half of connectors to use secrets manager v2
f889729
@aimethed
all others except ddb secrets v2
b717ea6
@aimethed
update ddb to secrets manager v2
cd75a48
@aimethed
remaining changes missed with initial passes
fb591ee
@aimethed
updated glue to v2 in sdk
f11d7d0
@aimethed
glue v2 for docdb and ddb
dfc3164
@aimethed
glue v2 for hbase and elasticsearch
2616d10
@aimethed
glue v2 for gcs
68a392c
@aimethed
glue v2 for neptune, redis, timestream
4839562
@aimethed
glue v2 for kafka
deeb559
@aimethed
glue v2 for msk
40d76e4
@aimethed
removed hbase dependencies to shrink jar size
4e1fc93
@aimethed
Merge branch 'v2-migration-secrets' into v2-migration-glue
79d63b9 
Merging in secrets change
@aimethed
swap to apache http client for jar size reasons
04bd9f6
@Trianz-Akshay
unused dependency remove hbase connector (#2021)
37ff821
@Trianz-Akshay
unused dependency remove bigquery connector (#2022)
35ee207
@aimethed
Merge branch 'master' into v2-migration-secrets
3ccf6e0
@aimethed
Merge branch 'master' into v2-migration-glue
d35ea4b
@aimethed
update ddb pom to use same v2 versions as sdk
28b42dd
@aimethed
secrets manager v2 for clickhouse
ff03307
@aimethed aimethed changed the base branch from master to v2-migration-secrets June 27, 2024 21:24
@aimethed
Copy link
Contributor Author

aimethed commented Jul 9, 2024

Linking to tracking issue: #2079

@aimethed aimethed self-assigned this Jul 16, 2024
@aimethed aimethed changed the base branch from v2-migration-secrets to v2-master July 22, 2024 19:59
@aimethed aimethed merged commit 6d25424 into v2-master Jul 22, 2024
5 checks passed
aimethed added a commit that referenced this pull request Sep 25, 2024
@aimethed @Trianz-Akshay @Jithendar12
V2 migration glue (#2052)
c1335dd 
Co-authored-by: Trianz-Akshay <108925344+Trianz-Akshay@users.noreply.github.com>
Co-authored-by: Jithendar Trianz <106380520+Jithendar12@users.noreply.github.com>
github-actions bot pushed a commit that referenced this pull request Dec 17, 2024
Cut release 2024.50.1
fa0f8ff 
  - Panama sdkv2 gdcv2 (#2352)
  - Support DECIMAL type properly when using DDB type NUMBER in sets (#2483)
  - Add SECRET_NAME_PATTERN unit test (#2471)
  - Update neptune documentation (#2472)
  - Enabled RDS Certs for Oracle DB (#2473)
  - Oracle Data Types Clean up (#2453)
  - check legacy IS_FIPS_ENABLED (#2470)
  - revert bump (#2469)
  - remove Trianz label (#2468)
  - various cleanup fixes (#2467)
  - remove dynamodb:ListSchemas
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - Disabling Auto Approval Bot (#2466)
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.37 to 20.00.00.38 (#2455)
  - build(deps): bump com.teradata.jdbc:terajdbc
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.29.24 to 2.29.29 (#2462)
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs
  - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 3.17.4 to 3.17.5 (#2459)
  - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier
  - build(deps): bump com.microsoft.sqlserver:mssql-jdbc from 12.8.1.jre11 to 12.9.0.jre11-preview (#2458)
  - build(deps): bump com.microsoft.sqlserver:mssql-jdbc
  - build(deps): bump net.jqwik:jqwik from 1.9.1 to 1.9.2 (#2463)
  - build(deps): bump net.jqwik:jqwik from 1.9.1 to 1.9.2
  - build(deps): bump io.confluent:kafka-protobuf-serializer from 7.7.2 to 7.8.0 (#2464)
  - build(deps): bump io.confluent:kafka-protobuf-serializer
  - build(deps): bump org.apache.kafka:kafka-clients from 7.7.2-ce to 7.8.0-ce (#2461)
  - build(deps): bump org.apache.kafka:kafka-clients
  - build(deps): bump org.apache.maven.plugins:maven-javadoc-plugin from 3.11.1 to 3.11.2 (#2460)
  - build(deps): bump org.apache.maven.plugins:maven-javadoc-plugin
  - build(deps): bump io.confluent:kafka-avro-serializer from 7.7.2 to 7.8.0 (#2456)
  - build(deps): bump io.confluent:kafka-avro-serializer from 7.7.2 to 7.8.0
  - build(deps): bump aws-sdk-v2.version from 2.29.20 to 2.29.29 (#2454)
  - build(deps): bump aws-sdk-v2.version from 2.29.20 to 2.29.29
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.29.20 to 2.29.24 (#2446)
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs
  - build(deps): bump io.lettuce:lettuce-core from 6.5.0.RELEASE to 6.5.1.RELEASE (#2450)
  - build(deps): bump io.lettuce:lettuce-core
  - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib-jdk8 from 2.0.21 to 2.1.0 (#2444)
  - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib-jdk8
  - build(deps): bump org.codehaus.mojo:license-maven-plugin from 2.4.0 to 2.5.0 (#2449)
  - build(deps): bump org.codehaus.mojo:license-maven-plugin
  - build(deps-dev): bump log4j2Version from 2.24.1 to 2.24.2 (#2426)
  - build(deps-dev): bump log4j2Version from 2.24.1 to 2.24.2
  - build(deps): bump io.confluent:kafka-avro-serializer from 7.7.1 to 7.7.2 (#2451)
  - build(deps): bump io.confluent:kafka-avro-serializer from 7.7.1 to 7.7.2
  - build(deps): bump org.apache.kafka:kafka-clients from 7.7.1-ce to 7.7.2-ce (#2445)
  - build(deps): bump org.apache.kafka:kafka-clients
  - build(deps): bump io.confluent:kafka-protobuf-serializer from 7.7.1 to 7.7.2 (#2447)
  - build(deps): bump io.confluent:kafka-protobuf-serializer
  - build(deps): bump io.confluent:kafka-protobuf-provider from 7.7.1 to 7.7.2 (#2448)
  - build(deps): bump io.confluent:kafka-protobuf-provider
  - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib from 2.0.21 to 2.1.0 (#2443)
  - build(deps): bump org.jetbrains.kotlin:kotlin-stdlib
  - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier from 3.17.3 to 3.17.4 (#2442)
  - build(deps-dev): bump nl.jqno.equalsverifier:equalsverifier
  - build(deps): bump fasterxml.jackson.version from 2.18.1 to 2.18.2 (#2441)
  - build(deps): bump fasterxml.jackson.version from 2.18.1 to 2.18.2
  - build(deps): bump org.jetbrains.kotlin:kotlin-reflect from 2.0.21 to 2.1.0 (#2438)
  - build(deps): bump org.jetbrains.kotlin:kotlin-reflect
  - Snowflake case insensitive search improve with ANNOTATION added back for backward compatibility (#2437)
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - build(deps): bump com.amazon.redshift:redshift-jdbc42 from 2.1.0.30 to 2.1.0.31 (#2424)
  - build(deps): bump com.amazon.redshift:redshift-jdbc42
  - build(deps): bump com.google.cloud:google-cloud-storage from 2.44.1 to 2.45.0 (#2432)
  - build(deps): bump com.google.cloud:google-cloud-storage
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs from 2.29.15 to 2.29.20 (#2431)
  - build(deps): bump software.amazon.awssdk:cloudwatchlogs
  - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql from 5.0.3 to 5.1.0 (#2428)
  - build(deps): bump org.eclipse.rdf4j:rdf4j-repository-sparql
  - build(deps): bump com.microsoft.azure:msal4j from 1.17.2 to 1.17.3 (#2429)
  - build(deps): bump com.microsoft.azure:msal4j from 1.17.2 to 1.17.3
  - build(deps): bump com.teradata.jdbc:terajdbc from 20.00.00.34 to 20.00.00.37 (#2430)
  - build(deps): bump com.teradata.jdbc:terajdbc
  - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client from 8.16.0 to 8.16.1 (#2427)
  - build(deps): bump org.elasticsearch.client:elasticsearch-rest-client
  - build(deps): bump aws-sdk-v2.version from 2.29.15 to 2.29.20 (#2423)
  - build(deps): bump aws-sdk-v2.version from 2.29.15 to 2.29.20
  - Changed message to debug (#2422)
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - oracle casing flag (#2415)
  - Change default endpoint (#2416)
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - Fix secret issue if field is integer (#2412)
  - glue connection reference fix (#2414)
  - fixing to use quoted password (#2413)
  - Fix docdb connection string (#2407)
  - [panama-sdkv2-gdcv2] Oracle Panama Testing Fix  (#2397)
  - add quotes around oracle password (#2399)
  - update to use SSL oracle url (#2400)
  - Enable case insensitive username/password in secret and allow secret … (#2398)
  - Synapse panama issue fix, updated connection string prefix. (#2394)
  - [panama-sdkv2-gdcv2] Fix Db2 JDBC Connection String (#2395)
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - Passing snowflake JDBC parameter into parameters fields instead of wi… (#2391)
  - Fix issue with secret being not populated for default EnvironmentProp… (#2393)
  - add default to oracle connection kmskeyid (#2392)
  - Check if auth is not null (#2388)
  - Snowflake case insensitive match instead of upper case by default (#2387)
  - add ImageConfig back to postgres connection yaml (#2386)
  - Increase glue connection timeout
  - Use proper SecretNamePrefix instead of SecretName
  - Use glue gamma as default
  - Use gamma glue endpoint if environment variable specified
  - update connections yaml files with new image uri
  - Merge branch 'master' into panama-sdkv2-gdcv2
  - Merge branch 'v2-master' into panama-sdkv2-gdcv2
  - Update DDB new errors to v2
  - Merge branch 'v2-master'
  - reverting PR #2273 for gbq connector as it's not required for cdk. (#2311)
  - Fix lambda issues with glue (#2308)
  - fix checkstyle
  - Merge branch 'v2-master' into panama-sdkv2-gdcv2
  - V2 final changes (#2297)
  - Use prod
  - Fix teradata (#2294)
  - SDK V2 ec2 changes (#2293)
  - Merge pull request #20 from aimethed/update-connections-yaml
  - remove unneeded Ref
  - Merge in v2-master 9/25 (#19)
  - fix missed error from merge
  - Merge branch 'v2-master'
  - V2 rds (#2273)
  - v2 DocDB (#2282)
  - v2 Cloudformation (#2281)
  - migrate awslogs to cloudwatchlogs (v1 to v2) (#2272)
  - Add RedshiftCompositeHandler class to populate environment from glue connections (#18)
  - v2 sdk changes redshift (#2289)
  - v2 sdk changes emr (#2288)
  - Merge pull request #17 from aimethed/update-connections-yaml
  - fix mysql connection yaml
  - update connectors with mux handlers to use normal in connections yaml
  - update all ImageURI to use correct repo name (no prod)
  - Merge pull request #16 from aimethed/update-connections-yaml
  - missed a Runtime property
  - update image uri to not include partition
  - Implement EnvironmentProperties for remaining non-jdbc connectors (#12)
  - v2 changes for timestream (#2239)
  - update new uses of semantic version (#2258)
  - update connections yaml files to use ecr image
  - fixing yaml files with correct parameters (#13)
  - Merge pull request #11 from awslabs/v2-master
  - V2 master merge 9/10 (#2255)
  - v2 migration elasticsearch (#2243)
  - Migrate Elasticache to AWS SDK v2 (#2238)
  - V2 image deployment (#2253)
  - Add rest of JDBC connectors excluding redshift (#10)
  - Split CFN templates into original and glue connections
  - V2 merge jsii (#2240)
  - Refactor glue connection logic to be more abstract (#9)
  - Migrate Cloudwatch Metrics connector to v2 (#2182)
  - Add GDCv2 properties to environment
  - Patch issues from initial merge
  - Merge v2-master into Panama project
  - Panama phase 1 squashed commit
  - Update v2-master with master
  - update v2-master with msk dependency change (#2208)
  - v2 migration vertica issue fix (#2147)
  - Fix return statement in getGlueSchemaType method (#2199)
  - fixed SpillLocationVerifier merge errors
  - merge in master, to be tested
  - V2 migration s3 (#2108)
  - V2 migration lambda (#2077)
  - V2 migration KMS (#2107)
  - V2 migration athena (#2055)
  - v2 migration sts (#2054)
  - V2 migration jmes (#2053)
  - V2 migration glue (#2052)
  - V2 migration secrets (#2051)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chngpe chngpe chngpe approved these changes

Assignees

@aimethed aimethed

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aimethed @chngpe @Trianz-Akshay @Jithendar12