Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-3439: [R] R language bindings for Feather format #2947

Closed
wants to merge 15 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
+ feather prefix -> feather_table_writer and feather_table_reader
  • Loading branch information
romainfrancois committed Nov 14, 2018
commit da9df5d69dfe5c082edaa87096718dcc3e65c9c5
16 changes: 8 additions & 8 deletions r/NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ S3method(buffer,numeric)
S3method(buffer,raw)
S3method(buffer_reader,"arrow::Buffer")
S3method(buffer_reader,default)
S3method(feather_table_reader,"arrow::io::RandomAccessFile")
S3method(feather_table_reader,"arrow::ipc::feather::TableReader")
S3method(feather_table_reader,character)
S3method(feather_table_reader,default)
S3method(feather_table_reader,fs_path)
S3method(feather_table_writer,"arrow::io::OutputStream")
S3method(fixed_size_buffer_writer,"arrow::Buffer")
S3method(fixed_size_buffer_writer,default)
S3method(length,"arrow::Array")
Expand Down Expand Up @@ -48,12 +54,6 @@ S3method(record_batch_file_reader,character)
S3method(record_batch_file_reader,fs_path)
S3method(record_batch_stream_reader,"arrow::io::InputStream")
S3method(record_batch_stream_reader,raw)
S3method(table_reader,"arrow::io::RandomAccessFile")
S3method(table_reader,"arrow::ipc::feather::TableReader")
S3method(table_reader,character)
S3method(table_reader,default)
S3method(table_reader,fs_path)
S3method(table_writer,"arrow::io::OutputStream")
S3method(write_arrow,"arrow::RecordBatch")
S3method(write_arrow,"arrow::Table")
S3method(write_arrow,data.frame)
Expand Down Expand Up @@ -91,6 +91,8 @@ export(date32)
export(date64)
export(decimal)
export(dictionary)
export(feather_table_reader)
export(feather_table_writer)
export(file_open)
export(file_output_stream)
export(fixed_size_buffer_writer)
Expand Down Expand Up @@ -123,8 +125,6 @@ export(schema)
export(str.integer64)
export(struct)
export(table)
export(table_reader)
export(table_writer)
export(time32)
export(time64)
export(timestamp)
Expand Down
30 changes: 15 additions & 15 deletions r/R/feather.R
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@
#' @param stream an OutputStream
#'
#' @export
table_writer <- function(stream) {
UseMethod("table_writer")
feather_table_writer <- function(stream) {
UseMethod("feather_table_writer")
}

#' @export
`table_writer.arrow::io::OutputStream` <- function(stream){
`feather_table_writer.arrow::io::OutputStream` <- function(stream){
unique_ptr(`arrow::ipc::feather::TableWriter`, ipc___feather___TableWriter__Open(stream))
}

Expand Down Expand Up @@ -107,7 +107,7 @@ write_feather_RecordBatch <- function(data, stream) {
#' @export
#' @method write_feather_RecordBatch arrow::io::OutputStream
`write_feather_RecordBatch.arrow::io::OutputStream` <- function(data, stream) {
ipc___TableWriter__RecordBatch__WriteFeather(table_writer(stream), data)
ipc___TableWriter__RecordBatch__WriteFeather(feather_table_writer(stream), data)
}

#' A arrow::ipc::feather::TableReader to read from a file
Expand All @@ -117,44 +117,44 @@ write_feather_RecordBatch <- function(data, stream) {
#' @param ... extra parameters
#'
#' @export
table_reader <- function(file, ...){
UseMethod("table_reader")
feather_table_reader <- function(file, ...){
UseMethod("feather_table_reader")
}

#' @export
table_reader.default <- function(file, ...) {
feather_table_reader.default <- function(file, ...) {
stop("unsupported")
}

#' @export
table_reader.character <- function(file, mmap = TRUE, ...) {
table_reader(fs::path_abs(file), mmap = mmap, ...)
feather_table_reader.character <- function(file, mmap = TRUE, ...) {
feather_table_reader(fs::path_abs(file), mmap = mmap, ...)
}

#' @export
table_reader.fs_path <- function(file, mmap = TRUE, ...) {
feather_table_reader.fs_path <- function(file, mmap = TRUE, ...) {
stream <- if(isTRUE(mmap)) mmap_open(file, ...) else file_open(file, ...)
table_reader(stream)
feather_table_reader(stream)
}

#' @export
`table_reader.arrow::io::RandomAccessFile` <- function(file, ...){
`feather_table_reader.arrow::io::RandomAccessFile` <- function(file, ...){
unique_ptr(`arrow::ipc::feather::TableReader`, ipc___feather___TableReader__Open(file))
}

#' @export
`table_reader.arrow::ipc::feather::TableReader` <- function(file){
`feather_table_reader.arrow::ipc::feather::TableReader` <- function(file){
file
}

#' Read a feather file
#'
#' @param file a arrow::ipc::feather::TableReader or whatever the [table_reader()] function can handle
#' @param file a arrow::ipc::feather::TableReader or whatever the [feather_table_reader()] function can handle
#' @param ... additional parameters
#'
#' @return an arrow::Table
#'
#' @export
read_feather <- function(file, ...){
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It occurs to me that the semantics of this function are different from feather::read_feather (it returns an arrow::Table instead of data.frame), if we care

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO it would be a nice incentive for people to abandon the feather package in favor of arrow if we could say arrow::read_feather() / arrow::write_feathe()r are drop-in replacements.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the goals of arrow is to make Array structures palpable in R, which feather didn't do. But yeah I guess we need both, i.e.

  • a function that returns an arrow::Table, not sure how to call this one.
  • a function, perhaps read_feather if this need to look like feather::read_feather that returns a tibble by calling as_tibble on the result of the first one.

Right now, it makes sense to go straight to tibble, but soon enough this (or other) packages will have more tooling to manipulate and compute directly on arrow memory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could add an as_tibble = TRUE argument to read_feather so this can be toggled on and off

table_reader(file, ...)$Read()
feather_table_reader(file, ...)$Read()
}
6 changes: 3 additions & 3 deletions r/man/table_reader.Rd → r/man/feather_table_reader.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions r/man/table_writer.Rd → r/man/feather_table_writer.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion r/man/read_feather.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.