Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Aggregate functions to take builder parameters #859

Merged
merged 35 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1825394
Add NullTreatment enum wrapper and add filter option to approx_distinct
timsaucer Sep 4, 2024
3b96b9d
Small usability on aggregate
timsaucer Sep 4, 2024
c434b4e
Adding documentation and additional unit test for approx_median
timsaucer Sep 6, 2024
1a5e138
Update approx_percentil_cont with builder parameters it uses, which i…
timsaucer Sep 6, 2024
c931c06
Update approx_percentil_cont_with_weight with builder parameters it u…
timsaucer Sep 6, 2024
2cc7c94
Update array_agg to use aggregate options
timsaucer Sep 7, 2024
52e33ac
Update builder options for avg aggregate function
timsaucer Sep 7, 2024
3701631
move bit_and bit_or to use macro to generaty python fn
timsaucer Sep 7, 2024
176092c
Update builder arguments for bitwise operators
timsaucer Sep 7, 2024
fdee791
Use macro for bool_and and bool_or
timsaucer Sep 7, 2024
4f93736
Update python wrapper for arguments appropriate to bool operators
timsaucer Sep 7, 2024
62f3d2c
Set corr to use macro for pyfunction
timsaucer Sep 7, 2024
32d8ddd
Update unit test to make it easier to debug
timsaucer Sep 7, 2024
9543626
Update corr python wrapper to expose only builder parameters used
timsaucer Sep 7, 2024
8d16a3c
Update count and count_star to use macro for exposing
timsaucer Sep 7, 2024
55ebc17
Update count and count_star with approprate aggregation options
timsaucer Sep 7, 2024
7e42e6c
Move covar_pop and covar_samp to use macro for aggregates
timsaucer Sep 7, 2024
ceb65c6
Updateing covar_pop and covar_samp with builder option
timsaucer Sep 7, 2024
b7262ba
Use macro for last_value and move first_value to be near it
timsaucer Sep 7, 2024
91e5f7d
Update first_value and last_value with the builder parameters that ar…
timsaucer Sep 7, 2024
fde7e70
Remove grouping since it is not actually implemented upstream
timsaucer Sep 7, 2024
22826d4
Move median to use macro
timsaucer Sep 7, 2024
85df127
Expose builder options for median
timsaucer Sep 7, 2024
3296e1a
Expose nth value
timsaucer Sep 7, 2024
a0e24b4
Updating linear regression functions to use filter and macro
timsaucer Sep 8, 2024
2325223
Update stddev and stddev_pop to use filter and macro
timsaucer Sep 8, 2024
6be2094
Expose string_agg
timsaucer Sep 8, 2024
6420f07
Add string_agg to python wrappers and add unit test
timsaucer Sep 8, 2024
529de88
Switch sum to use macro in rust side and expose correct options in py…
timsaucer Sep 8, 2024
e352ee3
Use macro for exposing var_pop and var_samp
timsaucer Sep 8, 2024
1857468
Add unit tests for filtering on var_pop and var_samp
timsaucer Sep 8, 2024
7148dcb
Move approximation functions to use macro when possible
timsaucer Sep 8, 2024
b55ff88
Update user documentation to explain in detail the options for aggreg…
timsaucer Sep 8, 2024
ba09df1
Update unit test to handle Python 3.10
timsaucer Sep 8, 2024
62ab0ea
Clean up commented code
timsaucer Sep 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Switch sum to use macro in rust side and expose correct options in py…
…thon wrapper
  • Loading branch information
timsaucer committed Sep 8, 2024
commit 529de8833eae04d658d7f3c415dcc16e0ac68f0e
20 changes: 17 additions & 3 deletions python/datafusion/functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -1797,9 +1797,23 @@ def min(expression: Expr, filter: Optional[Expr] = None) -> Expr:
return Expr(f.min(expression.expr, filter=filter_raw))


def sum(arg: Expr) -> Expr:
"""Computes the sum of a set of numbers."""
return Expr(f.sum(arg.expr))
def sum(
expression: Expr,
filter: Optional[Expr] = None,
) -> Expr:
"""Computes the sum of a set of numbers.

This aggregate function expects a numeric expression.

If using the builder functions described in ref:`_aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.

Args:
expression: Values to combine into an array
filter: If provided, only compute against rows for which the filter is True
"""
filter_raw = filter.expr if filter is not None else None
return Expr(f.sum(expression.expr, filter=filter_raw))


def stddev(expression: Expr, filter: Optional[Expr] = None) -> Expr:
Expand Down
1 change: 1 addition & 0 deletions python/datafusion/tests/test_aggregation.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ def test_aggregation_stats(df, agg_expr, calc_expected):
False,
),
(f.avg(column("b"), filter=column("a") != lit(1)), pa.array([5.0]), False),
(f.sum(column("b"), filter=column("a") != lit(1)), pa.array([10]), False),
(f.count(column("b"), distinct=True), pa.array([2]), False),
(f.count(column("b"), filter=column("a") != 3), pa.array([2]), False),
(f.count(), pa.array([3]), False),
Expand Down
6 changes: 1 addition & 5 deletions src/functions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,6 @@ pub fn approx_percentile_cont_with_weight(
add_builder_fns_to_aggregate(agg_fn, None, filter, None, None)
}

#[pyfunction]
pub fn sum(args: PyExpr) -> PyExpr {
functions_aggregate::expr_fn::sum(args.expr).into()
}

#[pyfunction]
pub fn var_samp(expression: PyExpr) -> PyExpr {
functions_aggregate::expr_fn::var_sample(expression.expr).into()
Expand Down Expand Up @@ -664,6 +659,7 @@ aggregate_function!(array_agg);
aggregate_function!(max);
aggregate_function!(min);
aggregate_function!(avg);
aggregate_function!(sum);
aggregate_function!(bit_and);
aggregate_function!(bit_or);
aggregate_function!(bit_xor);
Expand Down