Provide a syntax to perform row aggregations fast

[ppalmes]

• All() operates on collection of columns and typically, you need to operate in many columns so it is easier to process vector or tuple of rows instead of using positional arguments.
• All() -> ByRow(sum or mean) won't work including other statistical metrics because All returns positional arguments and most functions process either vectors or tuples.
• All()=>ByRow((x...) -> sum(x)) or AsTable(All()) => (x->sum(x)) is needed but it is messy because why do you have to transform All() into a table because from the user perspective, dataframe is a table already. It is not clear why you need AsTable with All(). This happens because All() returns a positional value instead of a collection like vector or tuple
• most functions return scalar, or tuple, or vector. it is confusing to find All() returning positional argument for typical user. it is convenient to deal with two or three columns but many use-cases require operating each row across several columns

[bkamins]

All() -> ByRow(sum or mean) won't work

All() => ByRow(+) works, as for mean indeed you have to write combine(df, AsTable(All()) => ByRow(mean))

AsTable(All()) => (x->sum(x)) is needed

It is enough to write:

All() => +

or

AsTAble(All()) => sum

in this case

but many use-cases require operating each row across several columns

As a side note. These operations will be inefficient using a DataFrame, if you have such a use case probably a different data structure should be used (DataFrames.jl is optimized for column access not for row access).

---

Having said that, we could have another wrapper similar to AsTable, e.g. AsTuple that would produce a Tuple instead.

[pdeffebach]

Thanks for the issue report!

I acknowledge that All() is awkward in this context since it has an undefined number of inputs so it mirrors splatting more. However note that

1. Plenty of functions allow spatting, i.e. +, string, *, so these things will just work with current behavior.
2. I disagree that "working with many many columns" is the dominant use-case. See some code I wrote for a personal project here. Plenty of uses of transform with positional arguments.

3. into a table because from the user perspective, dataframe is a table already. It is not clear why you need AsTable with All(). This happens because All() returns a positional value instead of a collection like vector or tuple

note that you aren't advocating making a DataFrame the default output of All(), so this problem wouldn't be fixed by your proposal.

4. It is definitely confusing that All() doesn't return a "thing". I agree, but we are trying to emulate the anonymous function syntax here a little bit, i.e. [:x, :y] => fun is "kind of" like (x, y) -> fun. It's not 1:1 but thats the mental model I think users should have.

5. Note that all the functionality you request is currently available in AsTable. So these are things you can currently do. I presume you would like AsTable to return a DataFrame.

If you are really working with 100s of columns, then AsTable might be slow due to compilation. I think we should maybe have an AsDataFrame type for working with many many columns. But then you might as well just not do piping all together. That's always an option.

[ppalmes]

i think the documentation can also be improved on this. because when i first tried the examples, i really am confused why i selected columns and then do some summary statistics to them and then have the error that sum/mean/median/ etc cannot operate on those columns you selected. it is typical for example that you iterate on df based on certain groupings, do transformations column-wise or row-wise, and expect that these operations have similar view whether you are doing it columnwise or rowwise from the user perspective. i get a row, i expect that i can just get the mean elements of this slice. my understanding is if you get a slice of table, you get a vector-like data you can operate. for example, it will be normal to do All(r"x*") or r"x*" to gather all x's to get mean of them. with positional args, you have to do (x...)->f(x) for most of aggregation functions. i think the use-case of + or * are less than many of aggregation functions for vectors.

[ppalmes]

a good use case is that you have x1,..., x100 columns each is a sensor value located in certain region. it is a time-series so your rows are time periods for example. to estimate the average temperature value per unit of time for example, you need to get the mean per row of all those x's but you need to use regular expressions because another region y's which is a ph sensor have also similar data in the same dataframe. basically, you want a slice using All() or r"". unfortunately you are not getting a vector/tuple per row so your typical functions for summary stats or even just maximum/minimum value or scaling won't work. + and * are rarely needed in statistical summary. even sum does not work or mean which is more often use than just + or *.

[bkamins]

a good use case is that you have

I think this is a very important scenario. I have this situation very often. However, you should decide how you want to work with this data (it is exactly the same situation as in RDBMS):

• if you want to work with them column-wise and want maximum speed: use the design you proposed
• if you want to work with them row-wise then create one column, call it :x and store in it tuples or vectors; then working on such data will be very fast row-wise, and reasonably fast column wise (not as fast as the first option, because of CPU cache misses, but quite good still as things will be type stable)

sum or +

These are exactly the same functions but they just have a different API: sum gets one argument and + gets many; but in the end they do the same (I omit some corner cases and concentrate on the core issue)

---

Still - as I have written - we can consider adding AsTuple wraper if you find it useful, that would pass a Tuple to a function. We could also similarly have AsVector.

[ppalmes]

I just think that AsTuple or AsVector should be hidden away from ordinary user. they should expect the regular expression to return something that behaves like a table or elements of table which is a vector. by having that function, the user has to know the difference between positional thing vs tuple vs vector. They can be added but I'm thinking of the default behavior when you slice a column or select a column by regular expression. for df[:, r"x"], i expect a dataframe? but if i do transform(df, r"x"......) the r"x" passed to anonymous function is not a tuple/vector which is the one that confuses me or maybe others.

[ppalmes]

ByRow(max) works but ByRow(sum) which looks similar because they aggregate things don't work. both says, get the rows, sum them. get the rows, give me the max value. this is what the user thinks.

[bkamins]

As @nalimilan - this was the default in the past. We have changed it because of compilation latency.

What we have now was selected as a default because it reduces compilation times in the most common cases (and there were user reports that using what we had as a legacy solution caused so many recompilations that it became unacceptable).

So the default behavior is something to be learned unfortunately.

However, as said - I am open to having different inputs to functions using special wrappers.

[ppalmes]

yeah, i think we have to hide away the complexity from the user. i think it's just a matter of what is expected by the user who is not familiar with the existence of special commands like AsTuple or AsVector. regular expression is really good in slicing the columns so ByRow(fun) confuses me because it does not deal in collections of elements.

[pdeffebach]

I agree that it would make sense to have AsTable be the default. But then we would have users complain about unnecessary indexing for functions with positional arguments. We would have to introduce a AsPositional term.

I think what people find the default and intuitive is very subjective. Its a tough balancing act.

[ppalmes]

the errors can be intimidating by missing the AsTable() thing. i think positional thing is less common in terms of expectations. seldom i encounter function that returns not a scalar value, tuple, or vector.

[ppalmes]

collection of positional thing as returned value is the main thing that confused me the most. i did not expect that return value.

[pdeffebach]

As noted above, with data cleaning you absolutely want positional arguments a lot. Many of us come from a social science background, which probably lead to that being the default choice.