[R] Accept format-specific scan options in collect()

[asfimport]

ARROW-9749 and ARROW-8631 added format/scan-specific options. In R, the most natural place to accept these is in collect(), but this isn't yet done.

Reporter: David Li / @lidavidm

Related issues:

• [C++][Dataset] Add ConvertOptions and ReadOptions to CsvFileFormat (relates to)
• [C++][Dataset] Extract format-specific scan options from FileFormat (relates to)

_{Note: This issue was originally created as ARROW-12059. Please see the migration documentation for further details.}

[asfimport]

Jonathan Keane / @jonkeane:
Working on an independent task I ran into this (and followed the issues to make sure we've got it covered).

I'm not totally sure that collect() is the most natural place to put this from an R-user's perspective.

The code I first tried was:

ds <- open_dataset("cranlogs", partitioning = c("year", "month", "day"), format = "csv", na = c("", "NA"))

# only ~17% of cran queries include version
since_41 <- ds %>%
  filter(date > as.Date("2021-05-18")) %>%
  filter(r_version != "NA") %>%
  select(date, r_version, r_os, package) %>%
  collect()

This is a pretty common (simple) version of this. Other readers like {vroom} that support reading multiple files do this at the read/open step:

library(vroom)

table <- read_csv(list_of_files, na = c("", "NA"))

I don't think that this doesn't have to be one or the other, I suspect we could support specifying it in both places, but we should implement it at the open_dataset() step if at all possible to match with other paradigms.

[asfimport]

Neal Richardson / @nealrichardson:
Is this still valid?