Skip to content

PERF: combine ops can be block based #5484

Closed
@jreback

Description

@jreback

related #5482 / #5481

in theory an op like:

df * df2 can be done block-based rather than column-by-column, so for large numbers of columns would see nice speedups. Though, passing to pd.eval is almost always faster because its passed to numexpr (at the cost of an upcasted dtype on the way back)

the 2 frames are aligned so that they have the same columns; then need to cast
each of the other (df2) in this case to a joint common type (happens in any event when you operate.

Then can just do block-by-block ops

prob BlockManager.eval can handle this with a kw arg to distinguish

Metadata

Metadata

Assignees

No one assigned

    Labels

    EnhancementInternalsRelated to non-user accessible pandas implementationNumeric OperationsArithmetic, Comparison, and Logical operationsPerformanceMemory or execution speed performance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions