[R] Better messaging for currently unsupported CSV options in open_dataset

[asfimport]

While read_csv_arrow()'s signature matches readr,  the readr_to_csv_parse_options() function (called by way of open_dataset()) only appears to capture a subset of those options:

(

arrow/r/R/csv.R

Line 464 in 883eb57

delimiter = delim,

)

readr_to_csv_parse_options <- function(delim = ",",
 quote = '"',
 escape_double = TRUE,
 escape_backslash = FALSE,
 skip_empty_rows = TRUE)

I ran into this trying to use a non-standard 'na' value:

 

open_dataset("/path/to/csv/directory/", schema = sch, partitioning=NULL, format="csv", delim=";", na="\\N", escape_backslash=TRUE, escape_double=FALSE`)
Error in readr_to_csv_parse_options(...) : unused argument (na = "\\N")

 

Reporter: Gabriel Bassett
Assignee: Ian Cook / @ianmcook

Related issues:

• [C++][Dataset] Add ConvertOptions and ReadOptions to CsvFileFormat (relates to)

PRs and other links:

• GitHub Pull Request #9143

_{Note: This issue was originally created as ARROW-10463. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Thanks for the report. This is actually ARROW-8631--the single file reader supports this option but CSV datasets do not. The Arrow CSV reader collects options into three separate objects, and na goes into ConvertOptions:

arrow/r/R/csv.R

Line 178 in 883eb57

na,

. But the Dataset module only knows about ReadOptions, so it tries to interpret all ... args as ReadOptions

arrow/r/R/dataset-format.R

Line 125 in 883eb57

readr_to_csv_parse_options(...)

ARROW-8631 is the ticket to implement this feature; I'll use this ticket to make the error message clearer in R in the meantime.

[asfimport]

Ian Cook / @ianmcook:
PR #9143 improves the error message that occurs when the user specifies unsupported readr-style options. The example above would yield this error message:

 Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "na" 

The added code matches both full and partial option names and improves handling of several other modes of unsupported options.

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 9143
#9143