Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cosmos spark config infra #17702

Merged
merged 3 commits into from
Nov 20, 2020
Merged

Cosmos spark config infra #17702

merged 3 commits into from
Nov 20, 2020

Conversation

moderakh
Copy link
Contributor

  • added infra for passing user config
  • infra for config validation
  • test for configs

note: user configs are validated and parsed from Map[Stirng, String] to case classes (e.g. CosmosAccountConfig)

TODO:

  • how to merge SparkConf application config to User Config?

@ghost ghost added the Cosmos label Nov 20, 2020
FabianMeiswinkel
FabianMeiswinkel approved these changes Nov 20, 2020
View reviewed changes
Copy link
Member

@FabianMeiswinkel FabianMeiswinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Thanks

@moderakh moderakh merged commit bb658bf into Azure:feature/cosmos/spark30 Nov 20, 2020
@moderakh moderakh added the cosmos:spark3 Cosmos DB Spark3 OLTP Connector label Dec 8, 2020
@moderakh moderakh linked an issue Dec 18, 2020 that may be closed by this pull request
Config foundation. #18238
Closed
@moderakh moderakh mentioned this pull request Dec 18, 2020
Closed
@moderakh moderakh deleted the users/moderakh/spark-config-infra branch February 8, 2021 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FabianMeiswinkel FabianMeiswinkel FabianMeiswinkel approved these changes

@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli

@kushagraThapar kushagraThapar Awaiting requested review from kushagraThapar

@mbhaskar mbhaskar Awaiting requested review from mbhaskar

@milismsft milismsft Awaiting requested review from milismsft

@simplynaveen20 simplynaveen20 Awaiting requested review from simplynaveen20

@xinlian12 xinlian12 Awaiting requested review from xinlian12

Assignees
No one assigned
Labels
cosmos:spark3 Cosmos DB Spark3 OLTP Connector Cosmos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Config foundation.
2 participants
@moderakh @FabianMeiswinkel