Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merging master #11

Merged
merged 6 commits into from
Dec 9, 2020
Merged

merging master #11

merged 6 commits into from
Dec 9, 2020

Commits on Dec 9, 2020

  1. [SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCH… 

    …AR types

### What changes were proposed in this pull request?

Add migration guide for CHAR VARCHAR types

### Why are the changes needed?

for migration

### Does this PR introduce _any_ user-facing change?

doc change

### How was this patch tested?

passing ci

Closes #30654 from yaooqinn/SPARK-33641-F.

Authored-by: Kent Yao <yaooqinn@hotmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
    @yaooqinn @cloud-fan
    yaooqinn authored and cloud-fan committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    c88edda View commit details
    Browse the repository at this point in the history

  2. [SPARK-33669] Wrong error message from YARN application state monitor… 

    … when sc.stop in yarn client mode

### What changes were proposed in this pull request?
This change make InterruptedIOException to be treated as InterruptedException when closing YarnClientSchedulerBackend, which doesn't log error with "YARN application has exited unexpectedly xxx"

### Why are the changes needed?
For YarnClient mode, when stopping YarnClientSchedulerBackend, it first tries to interrupt Yarn application monitor thread. In MonitorThread.run() it catches InterruptedException to gracefully response to stopping request.

But client.monitorApplication method also throws InterruptedIOException when the hadoop rpc call is calling. In this case, MonitorThread will not know it is interrupted, a Yarn App failed is returned with "Failed to contact YARN for application xxxxx;  YARN application has exited unexpectedly with state xxxxx" is logged with error level. which confuse user a lot.

### Does this PR introduce _any_ user-facing change?
Yes

### How was this patch tested?
very simple patch, seems no need?

Closes #30617 from sqlwindspeaker/yarn-client-interrupt-monitor.

Authored-by: suqilong <suqilong@qiyi.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
    suqilong authored and Mridul Muralidharan committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    48f93af View commit details
    Browse the repository at this point in the history

  3. [SPARK-33655][SQL] Improve performance of processing FETCH_PRIOR 

    ### What changes were proposed in this pull request?
Currently, when a client requests FETCH_PRIOR to Thriftserver, Thriftserver reiterates from the start position. Because Thriftserver caches a query result with an array when THRIFTSERVER_INCREMENTAL_COLLECT feature is off, FETCH_PRIOR can be implemented without reiterating the result. A trait FeatureIterator is added in order to separate the implementation for iterator and an array. Also, FeatureIterator supports moves cursor with absolute position, which will be useful for the implementation of FETCH_RELATIVE, FETCH_ABSOLUTE.

### Why are the changes needed?
For better performance of Thriftserver.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
FetchIteratorSuite

Closes #30600 from Dooyoung-Hwang/refactor_with_fetch_iterator.

Authored-by: Dooyoung Hwang <dooyoung.hwang@sk.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
    @HyukjinKwon
    Dooyoung Hwang authored and HyukjinKwon committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    a713a7e View commit details
    Browse the repository at this point in the history

  4. [SPARK-33719][DOC] Add make_date/make_timestamp/make_interval into th… 

    …e doc of ANSI Compliance

### What changes were proposed in this pull request?

Add make_date/make_timestamp/make_interval into the doc of ANSI Compliance

### Why are the changes needed?

Users can know that these functions throw runtime exceptions under ANSI mode if the result is not valid.
### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Build doc and check it in browser:
![image](https://user-images.githubusercontent.com/1097932/101608930-34a79e80-39bb-11eb-9294-9d9b8c3f6faa.png)

Closes #30683 from gengliangwang/improveDoc.

Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
    @gengliangwang @HyukjinKwon
    gengliangwang authored and HyukjinKwon committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    9959d49 View commit details
    Browse the repository at this point in the history

  5. [SPARK-33071][SPARK-33536][SQL][FOLLOW-UP] Rename deniedMetadataKeys … 

    …to nonInheritableMetadataKeys in Alias

### What changes were proposed in this pull request?

This PR is a followup of #30488. This PR proposes to rename `Alias.deniedMetadataKeys` to `Alias.nonInheritableMetadataKeys` to make it less confusing.

### Why are the changes needed?

To make it easier to maintain and read.

### Does this PR introduce _any_ user-facing change?

No. This is rather a code cleanup.

### How was this patch tested?

Ran the unittests written in the previous PR manually. Jenkins and GitHub Actions in this PR should also test them.

Closes #30682 from HyukjinKwon/SPARK-33071-SPARK-33536.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
    @HyukjinKwon
    HyukjinKwon committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    b5399d4 View commit details
    Browse the repository at this point in the history

  6. [SPARK-33722][SQL] Handle DELETE in ReplaceNullWithFalseInPredicate 

    ### What changes were proposed in this pull request?

This PR adds `DeleteFromTable` to supported plans in `ReplaceNullWithFalseInPredicate`.

### Why are the changes needed?

This change allows Spark to optimize delete conditions like we optimize filters.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

This PR extends the existing test cases to also cover `DeleteFromTable`.

Closes #30688 from aokolnychyi/spark-33722.

Authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
    @aokolnychyi @dongjoon-hyun
    aokolnychyi authored and dongjoon-hyun committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    fa9ce1d View commit details
    Browse the repository at this point in the history