Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS EMR and Athena components #1286

Merged
merged 3 commits into from
May 7, 2019

Conversation

Jeffwan
Copy link
Member

@Jeffwan Jeffwan commented May 5, 2019

  1. Add AWS EMR and Athena components
  2. Add example to use EMR to run Spark pipelines.

This change is Reviewable

Copy link
Contributor

@animeshsingh animeshsingh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm from syntax/semantic perspective


# Athena query is aync call, we need to fetch results and wait for execution
state = 'RUNNING'
max_execution = 5 # TODO: this should be an optional parameter from users. or use timeout
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - can be a default param in input

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I will give to our users and see how do they like it. Change will be made and expose to users later either with max_execution or max_waiting_time.

Ark-kun and others added 2 commits May 6, 2019 15:28
@Ark-kun
Copy link
Contributor

Ark-kun commented May 7, 2019

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit f5c464a into kubeflow:master May 7, 2019
@Jeffwan Jeffwan deleted the add_bigdata_components branch May 7, 2019 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants