Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
c281373
[SPARK-29248][SQL] Pass in number of partitions to WriteBuilder
edrevo Sep 27, 2019
903d863
move numpartitions to physicalbuildinfo
edrevo Nov 15, 2019
c6b95f5
Merge branch 'master' of https://github.com/apache/spark into add-par…
edrevo Nov 15, 2019
30c8800
fixes
edrevo Nov 15, 2019
15096b2
fixes
edrevo Nov 15, 2019
389afee
lint fixes
edrevo Nov 15, 2019
4f10e54
[SPARK-29655][SQL] Read bucketed tables obeys spark.sql.shuffle.parti…
wangyum Nov 15, 2019
ca4e894
PR feedback
edrevo Nov 15, 2019
16e3c4a
more pr feedback
edrevo Nov 15, 2019
ee4784b
[SPARK-26499][SQL][FOLLOW-UP] Replace `update` with `setByte` for Byt…
maropu Nov 15, 2019
1521889
[SPARK-29902][DOC][MINOR] Add listener event queue capacity configura…
shahidki31 Nov 15, 2019
848bdfa
[SPARK-29829][SQL] SHOW TABLE EXTENDED should do multi-catalog resolu…
planga82 Nov 15, 2019
c0507e0
[SPARK-29833][YARN] Add FileNotFoundException check for spark.yarn.jars
ulysses-you Nov 16, 2019
7720781
[SPARK-29127][SQL][PYTHON] Add a clue for Python related version info…
HyukjinKwon Nov 16, 2019
16e7195
[SPARK-29834][SQL] DESC DATABASE should look up catalog like v2 commands
fuwhu Nov 16, 2019
6d6b233
[SPARK-29343][SQL][FOLLOW-UP] Remove floating-point Sum/Average/Centr…
maropu Nov 16, 2019
1112fc6
[SPARK-29867][ML][PYTHON] Add __repr__ in Python ML Models
huaxingao Nov 16, 2019
f77c10d
[SPARK-29923][SQL][TESTS] Set io.netty.tryReflectionSetAccessible for…
dongjoon-hyun Nov 16, 2019
40ea4a1
[SPARK-29807][SQL] Rename "spark.sql.ansi.enabled" to "spark.sql.dial…
xuanyuanking Nov 16, 2019
d0470d6
[MINOR][TESTS] Ignore GitHub Action and AppVeyor file changes in testing
dongjoon-hyun Nov 16, 2019
5336473
[SPARK-29476][WEBUI] add tooltip for Thread
PavithraRamachandran Nov 16, 2019
e88267c
[SPARK-29928][SQL][TESTS] Check parsing timestamps up to microsecond …
MaxGekk Nov 17, 2019
cc12cf6
[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API
dongjoon-hyun Nov 17, 2019
388a737
[SPARK-29858][SQL] ALTER DATABASE (SET DBPROPERTIES) should look up c…
fuwhu Nov 17, 2019
a9959be
[SPARK-29456][WEBUI] Improve tooltip for Session Statistics Table col…
PavithraRamachandran Nov 17, 2019
e1fc38b
[SPARK-29932][R][TESTS] lint-r should do non-zero exit in case of errors
dongjoon-hyun Nov 17, 2019
5eb8973
[SPARK-29930][SQL] Remove SQL configs declared to be removed in Spark…
MaxGekk Nov 17, 2019
c5f644c
[SPARK-16872][ML][PYSPARK] Impl Gaussian Naive Bayes Classifier
zhengruifeng Nov 18, 2019
d83cacf
[SPARK-29907][SQL] Move DELETE/UPDATE/MERGE relative rules to dmlStat…
Nov 18, 2019
f280c6a
[SPARK-29378][R][FOLLOW-UP] Remove manual installation of Arrow depen…
HyukjinKwon Nov 18, 2019
42f8f79
[SPARK-29936][R] Fix SparkR lint errors and add lint-r GitHub Action
dongjoon-hyun Nov 18, 2019
9ff8ac7
javadoc fixes
edrevo Nov 18, 2019
ee3bd6d
[SPARK-25694][SQL] Add a config for `URL.setURLStreamHandlerFactory`
jiangzho Nov 18, 2019
7391237
[SPARK-29020][SQL] Improving array_sort behaviour
Nov 18, 2019
5cebe58
[SPARK-29783][SQL] Support SQL Standard/ISO_8601 output style for int…
yaooqinn Nov 18, 2019
50f6d93
[SPARK-29870][SQL] Unify the logic of multi-units interval string to …
yaooqinn Nov 18, 2019
c32e228
[SPARK-29859][SQL] ALTER DATABASE (SET LOCATION) should look up catal…
fuwhu Nov 18, 2019
ae6b711
[SPARK-29941][SQL] Add ansi type aliases for char and decimal
yaooqinn Nov 18, 2019
ea010a2
[SPARK-29873][SQL][TEST][FOLLOWUP] set operations should not escape w…
yaooqinn Nov 18, 2019
9514b82
[SPARK-29777][SPARKR] SparkR::cleanClosure aggressively removes a fun…
falaki Nov 19, 2019
8469614
[SPARK-25694][SQL][FOLLOW-UP] Move 'spark.sql.defaultUrlStreamHandler…
HyukjinKwon Nov 19, 2019
882f54b
[SPARK-29870][SQL][FOLLOW-UP] Keep CalendarInterval's toString
HyukjinKwon Nov 19, 2019
28a502c
[SPARK-28527][FOLLOW-UP][SQL][TEST] Add guides for ThriftServerQueryT…
wangyum Nov 19, 2019
a834dba
Revert "[SPARK-29644][SQL] Corrected ShortType and ByteType mapping t…
shivsood Nov 19, 2019
3d45779
[SPARK-29728][SQL] Datasource V2: Support ALTER TABLE RENAME TO
imback82 Nov 19, 2019
118d81f
[SPARK-29248][SQL] Add PhysicalWriteInfo with number of partitions
edrevo Nov 19, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lint fixes
  • Loading branch information
edrevo committed Nov 15, 2019
commit 389afeee3b51116533ecaa845391f0d4a96227e6
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
* An interface that defines how to write the data to data source for batch processing.
*
* The writing procedure is:
* 1. Create a writer factory by {@link #createBatchWriterFactory(PhysicalWriteInfo)}, serialize and send it to all
* the partitions of the input data(RDD).
* 1. Create a writer factory by {@link #createBatchWriterFactory(PhysicalWriteInfo)}, serialize
* and send it to all the partitions of the input data(RDD).
* 2. For each partition, create the data writer, and write the data of the partition with this
* writer. If all the data are written successfully, call {@link DataWriter#commit()}. If
* exception happens during the writing, call {@link DataWriter#abort()}.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

/**
* :: Experimental ::
* This interface contains physical (i.e. RDD) write information that data sources can use when generating a
* {@link DataWriterFactory} or a {@link StreamingDataWriterFactory}.
* This interface contains physical (i.e. RDD) write information that data sources can use when
* generating a {@link DataWriterFactory} or a {@link StreamingDataWriterFactory}.
*/
@Experimental
public interface PhysicalWriteInfo {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import org.apache.spark.sql.connector.catalog.Table;
import org.apache.spark.sql.connector.catalog.TableCapability;
import org.apache.spark.sql.connector.write.streaming.StreamingWrite;
import org.apache.spark.sql.types.StructType;

/**
* An interface for building the {@link BatchWrite}. Implementations can mix in some interfaces to
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@

package org.apache.spark.sql.connector.write;

import org.apache.spark.annotation.Evolving;
import org.apache.spark.annotation.Experimental;
import org.apache.spark.sql.types.StructType;

Expand All @@ -26,7 +25,7 @@
* This interface contains write information that data sources can use when generating a
* {@link WriteBuilder}.
*/
@Evolving
@Experimental
public interface WriteInfo {
/**
* @return `queryId` is a unique string of the query. It's possible that there are many queries
Expand Down