Skip to content

[SPARK-5846] Correctly set job description and pool for SQL jobs #4630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

kayousterhout
Copy link
Contributor

@marmbrus am I missing something obvious here? I verified that this fixes the problem for me (on 1.2.1) on EC2, but I'm confused about how others wouldn't have noticed this?

@SparkQA
Copy link

SparkQA commented Feb 16, 2015

Test build #27575 has finished for PR 4630 at commit 2022ad4.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

test this please

@SparkQA
Copy link

SparkQA commented Feb 17, 2015

Test build #27596 has finished for PR 4630 at commit 2022ad4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

/cc @liancheng

@liancheng
Copy link
Contributor

This LGTM. This PR also fixes SPARK-2472. For most queries, the sql call is lazy, so the job wasn't submitted until collect() or toLocalIterator() are called. The problem occurs when DDLs, commands, and CATS statements are executed, since they, or part of them (for CATS) are evaluated eagerly. Thanks for fixing this!

asfgit pushed a commit that referenced this pull request Feb 19, 2015
marmbrus am I missing something obvious here? I verified that this fixes the problem for me (on 1.2.1) on EC2, but I'm confused about how others wouldn't have noticed this?

Author: Kay Ousterhout <kayousterhout@gmail.com>

Closes #4630 from kayousterhout/SPARK-5846_1.3 and squashes the following commits:

2022ad4 [Kay Ousterhout] [SPARK-5846] Correctly set job description and pool for SQL jobs

(cherry picked from commit e945aa6)
Signed-off-by: Cheng Lian <lian@databricks.com>
@asfgit asfgit closed this in e945aa6 Feb 19, 2015
@liancheng
Copy link
Contributor

Merged into master and branch-1.3.

asfgit pushed a commit that referenced this pull request Feb 19, 2015
This is #4630 but modified for the 1.2 branch, because I'm guessing it makes sense to fix this issue in that branch (again, unless I missed something obvious here...)

Author: Kay Ousterhout <kayousterhout@gmail.com>

Closes #4631 from kayousterhout/SPARK-5846_1.2.1 and squashes the following commits:

ffe8ff2 [Kay Ousterhout] [SPARK-5846] Correctly set job description and pool for SQL jobs
@kayousterhout
Copy link
Contributor Author

Thanks for merging this Cheng!

On Wed, Feb 18, 2015 at 6:03 PM, Cheng Lian notifications@github.com
wrote:

Merged into master and branch-1.3.


Reply to this email directly or view it on GitHub
#4630 (comment).

markhamstra pushed a commit to markhamstra/spark that referenced this pull request Feb 24, 2015
This is apache#4630 but modified for the 1.2 branch, because I'm guessing it makes sense to fix this issue in that branch (again, unless I missed something obvious here...)

Author: Kay Ousterhout <kayousterhout@gmail.com>

Closes apache#4631 from kayousterhout/SPARK-5846_1.2.1 and squashes the following commits:

ffe8ff2 [Kay Ousterhout] [SPARK-5846] Correctly set job description and pool for SQL jobs
@kayousterhout kayousterhout deleted the SPARK-5846_1.3 branch April 12, 2017 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants