ARROW-8792: [C++][Python][R][GLib] New Array compute kernels implementation and execution framework

[wesm]

This patch is a major reworking of our development strategy for implementing array-valued functions and applying them in a query processing setting.

The design was partly inspired by my previous work designing Ibis (https://github.com/ibis-project/ibis -- the "expr" subsystem and the way that operators validate input types and resolve output types). Using only function names and input types, you can determine the output types of each function and resolve the "execute" function that performs a unit of work processing a batch of data. This will allow us to build deferred column expressions and then (eventually) do parallel execution.

There are a ton of details, but one nice thing is that there is now a single API entry point for invoking any function by its name:

Result<Datum> CallFunction(ExecContext* ctx, const std::string& func_name,
                           const std::vector<Datum>& args,
                           const FunctionOptions* options = NULLPTR);

What occurs when you do this:

• A Function instance is looked up in the global FunctionRegistry
• Given the descriptors of args (their types and shapes -- array or scalar), the Function searches for Kernel that is able to process those types and shapes. A kernel might be able to do array[T0], array[T1] or only scalar[T0], scalar[T1], for example. This permits kernel specialization to treat different type and shape combinations
• The kernel is executed iteratively against args based on what args contains -- if there are ChunkedArrays, they will be split into contiguous pieces. Kernels never see ChunkedArray, only Array or Scalar
• The Executor implementation is able to split contiguous Array inputs into smaller chunks, which is important for parallel execution. See ExecContext::set_exec_chunksize

To summarize: the REGISTRY contains FUNCTIONS. A FUNCTION contains KERNELS. A KERNEL is a specific implementation of a function that services a particular type combination.

An additional effort in this patch is to radically simplify the process of creating kernels that are based on a scalar function. To do this, there is a growing collection of template-based kernel generation classes in compute/kernels/codegen_internal.h that will surely be the topic of much debate. I want to make it a lot easier for people to add new kernels.

There are some other incidental changes in the PR, such as changing the convenience APIs like Cast to return Result. I'm afraid we may have to live with the API breakage unless someone else wants to add backward compatibility code for the old APIs.

I have to apologize for making such a large PR. I've been working long hours on this for nearly a month and the process of porting all of our existing functionality and making the unit tests pass caused much iteration in the "framework" part of the code, such that it would have been a huge time drain to review incomplete iterations of the framework that had not been proven to capture the functionality that previously existed in the project.

Given the size of this PR and that fact that it completely blocks any work into src/arrow/compute, I don't think we should let this sit unmerged for more than 4 or 5 days, tops. I'm committed to responding to all of your questions and working to address your feedback about the design and improving the documentation and code comments. I tried to leave copious comments to explain my thought process in various places. Feel free to make any and all comments in this PR or in whatever form you like. I don't think that merging should be blocked on stylistic issues.

[wesm]

For code reviewers:

• The code diff is a bit deceiving because many kernel files were renamed to make their taxonomy more clear
• I suggest focusing your energies on the files in the arrow/compute top level directory. This is almost entirely new code, what was there before is almost all gone

I am aware that there are some obviously messy things that will need to be cleaned up a bit, but we have to start somewhere.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8792

[wesm]

Aside from fixing the builds, I'm going to do a few things to help with this:

• Add as many comments as possible to the new header files
• Add some high level explanation to compute/README.md about how kernels work and how to begin to develop a new one
• Add simple Python bindings for FunctionRegistry, Function, and Kernel so that we can inspect the contents of the registry and the available kernels (and their type signatures) for each function

[wesm]

leaving a handful of cleaning TODOs for myself

[pitrou]

Some questions that came while reading through the API.

[pitrou]

It's weird to have this as part of the cast options.

[wesm]

Yes, I'm aware, but I wasn't able to figure out another way to communicate the output type to the standard kernel APIs

[nealrichardson]

The R failure:

  ── 1. Error: filter() on timestamp columns (@test-dataset.R#381)  ──────────────
  NotImplemented: Function >= has no kernel matching input types (array[timestamp[us, tz=UTC]], scalar[timestamp[us, tz=UTC]])
  Backtrace:
    1. arrow:::expect_equivalent(...)
    2. dplyr::filter(., ts >= lubridate::ymd_hms("2015-05-04 03:12:39"))
    2. dplyr::filter(., part == 1)
    2. dplyr::select(., ts)
   10. dplyr::collect(.)
   13. Scanner$create(x)$ToTable()
   16. arrow:::dataset___Scanner__ToTable(self)

I'm happy to help debug but that sounds pretty straightforward.

[wesm]

@nealrichardson yeah, I will fix (wasn't thinking about timezones when adding the dispatch rules for comparisons)

[kou]

Port GLib bindings

Can I start working on this on this branch?

[wesm]

@kou yes please go ahead, I'm sure you'll be careful about rebasing since I'm going to push some commits to the branch later today probably in response to the comments above

[kou]

OK!

[wesm]

I rebased without pushing any new changes

[emkornfield]

CC @brills I'm not sure if this impacts TFX at all.

[wesm]

@emkornfield @brills no Python APIs are changed (not sure how much of the C++ API TFX uses)

[emkornfield]

nit: Void or Null for name?

[wesm]

Hmm https://en.wikipedia.org/wiki/Arity

[emkornfield]

nit: VarArgs

It also seems like it might be more readable for callers to have to always specify min_args (especially because it isn't clear to me why 1 is the default and not 0 or 2

[wesm]

You need at least one argument spec for type checking

[wesm]

I'm changing the name, but will leave the default issue for follow up work

[wesm]

I got sucked into some other things today, I'll focus on getting the CI passing tomorrow so we can hopefully go into the weekend with a green build

[wesm]

@jorisvandenbossche I fixed the segfault you hit and now have changed the implementation of Array.filter to be

pc = _pc()
options = pc.FilterOptions(null_selection_behavior)
return pc.call_function('filter', [self, mask], options)

I also rebased -- I am going to merge this when the build is passing so we do not continue to stack changes that need to be reviewed on this PR. I welcome further code reviews on this PR or follow up JIRAs

[wesm]

+1

[wesm]

The PR has been merged, but the code review feature is still available. Please continue to leave comments and I will address them

[wesm]

I'll leave this PR open until next week sometime to allow more time for comments

[wesm]

Closing the PR again. Please open JIRA issues or review other PRs to provide more feedback.