Skip to content

[SPARK-12318][SPARKR] Save mode in SparkR should be error by default #10290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-12318][SPARKR] Save mode in SparkR should be error by default #10290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4f2e56b
[SPARK-12318][SPARKR] Save mode in SparkR should be error by default
zjffdu Dec 14, 2015
3e71b06
Add doc to this change
zjffdu Dec 15, 2015
d72a3af
Add migration doc and update R doc
zjffdu Dec 15, 2015
8eb181c
Update doc
zjffdu Dec 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions R/pkg/R/DataFrame.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1886,7 +1886,7 @@ setMethod("except",
#' @param df A SparkSQL DataFrame
#' @param path A name for the table
#' @param source A name for external data source
#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it is 'error' by default)
#'
#' @family DataFrame functions
#' @rdname write.df
Expand All @@ -1903,7 +1903,7 @@ setMethod("except",
#' }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add the default to this doc line above:

 #' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixcheung Is it necessary to do that ? Because I notice there's no doc for the default value in other R APIs. Actually user can see the default value in the function signature which is also in the R doc.

## S4 method for signature 'DataFrame,character'
write.df(df, path, source = NULL,
  mode = "error", ...)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I think that's just good practice, but it's fine both ways.

setMethod("write.df",
signature(df = "DataFrame", path = "character"),
function(df, path, source = NULL, mode = "append", ...){
function(df, path, source = NULL, mode = "error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
Expand All @@ -1928,7 +1928,7 @@ setMethod("write.df",
#' @export
setMethod("saveDF",
signature(df = "DataFrame", path = "character"),
function(df, path, source = NULL, mode = "append", ...){
function(df, path, source = NULL, mode = "error", ...){
write.df(df, path, source, mode, ...)
})

Expand All @@ -1951,7 +1951,7 @@ setMethod("saveDF",
#' @param df A SparkSQL DataFrame
#' @param tableName A name for the table
#' @param source A name for external data source
#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it is 'error' by default)
#'
#' @family DataFrame functions
#' @rdname saveAsTable
Expand All @@ -1968,7 +1968,7 @@ setMethod("saveDF",
setMethod("saveAsTable",
signature(df = "DataFrame", tableName = "character", source = "character",
mode = "character"),
function(df, tableName, source = NULL, mode="append", ...){
function(df, tableName, source = NULL, mode="error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
Expand Down
9 changes: 8 additions & 1 deletion docs/sparkr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ printSchema(people)
</div>

The data sources API can also be used to save out DataFrames into multiple file formats. For example we can save the DataFrame from the previous example
to a Parquet file using `write.df`
to a Parquet file using `write.df` (Until Spark 1.6, the default mode for writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala API)

<div data-lang="r" markdown="1">
{% highlight r %}
Expand Down Expand Up @@ -387,3 +387,10 @@ The following functions are masked by the SparkR package:
Since part of SparkR is modeled on the `dplyr` package, certain functions in SparkR share the same names with those in `dplyr`. Depending on the load order of the two packages, some functions from the package loaded first are masked by those in the package loaded after. In such case, prefix such calls with the package name, for instance, `SparkR::cume_dist(x)` or `dplyr::cume_dist(x)`.

You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/search.html)


# Migration Guide

## Upgrading From SparkR 1.6 to 1.7

- Until Spark 1.6, the default mode for writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala API.