[EPIC] A collection of Interval arithmetic (not Intervals of time) improvements

[alamb]

Is your feature request related to a problem or challenge?

DataFusion contains a powerful general purpose interval arithmetic library, in intervals, first introduced in #5322

This library is currently used for statistic and selectivity estimation, and to provide additional guarantees during expression simplification.

This ticket tracks additional items needed to improve the sophistication of DataFusion in this regard

• Enhance Interval arithmetic to support String type.  #7098
• Add support for OR to interval arithmetic #7883
• Consolidate interval analysies from Interval and PruningPredicate #7887

[berkaysynnada]

I am working on some new supports and improvements in interval arithmetic. I also plan to move the code into a more accessible place (possibly datafusion_expr). I will complete my work next week and link them here.

[alamb]

Thanks @berkaysynnada -- note I just proposed three relatively small changes . Perhaps you have some time to look at them:

• Minor: Clarify Boolean Interval handling and verify it with a test #7885
• Support Interval analysis for OR expressions #7884
• Minor: Fix bug in AND interval analysis tests (not code), and add more coverage #7886

I also plan to move the code into a more accessible place (possibly datafusion_expr)

What would you think about making a datafusion-interval crate to hold the interval analysis library ? that would keep it nicely isolated and likely improve the development cycle time (I found it quite painful to work on in datafusion-physical-expr)

[alamb]

I also filed #7887

[berkaysynnada]

Thanks @berkaysynnada -- note I just proposed three relatively small changes . Perhaps you have some time to look at them:

• Minor: Clarify Boolean Interval handling and verify it with a test #7885
• Support Interval analysis for OR expressions #7884
• Minor: Fix bug in AND interval analysis tests (not code), and add more coverage #7886

I also plan to move the code into a more accessible place (possibly datafusion_expr)

What would you think about making a datafusion-interval crate to hold the interval analysis library ? that would keep it nicely isolated and likely improve the development cycle time (I found it quite painful to work on in datafusion-physical-expr)

Thank you for the care and effort you have put into this issue 😌. I tried to answer the current behavior and how it should actually be in the reviews. However, as I mentioned in them, I am currently working on a comprehensive interval library update. It will eliminate many of the bugs you noticed and provide a more open, error-free API for users. I don't want to keep you waiting, but I think it will be easier to solve these issues after I quickly finalize my PR and present it to you.

As for making the new interval crate, it would be better to answer after discussing it with my teammates. Thanks again.

[alamb]

. I don't want to keep you waiting, but I think it will be easier to solve these issues after I quickly finalize my PR and present it to you.

There is no particular time pressure from my end. If you are working to revamp the code, I'll wait until that lands (and try to help review it promptly) and then we can pick up the PRs.

BTW is there a ticket somewhere that tracks what you plan to do (mostly I am thinking of any 'end user visible bugs/ behaviors' that might result)?

As for making the new interval crate, it would be better to answer after discussing it with my teammates. Thanks again.

Sounds good -- we can break it into a new crate at some later time too, if that is appropriate. There is no reason to do it as part of your in-filght refactoring

[berkaysynnada]

BTW is there a ticket somewhere that tracks what you plan to do (mostly I am thinking of any 'end user visible bugs/ behaviors' that might result)?

There is no ticket here but I can summarize them as follows:

• Relocating the code.
• Propagate arithmetic API change (the a-b<0 assumption will no longer be used for a<b propagation, which will also be useful for easy integration of supports such as OR).
• Mul and Div support
• Interval construct API (including checks)
• The relationship of the next_value() function with infinite bounds.

Sounds good -- we can break it into a new crate at some later time too, if that is appropriate. There is no reason to do it as part of your in-filght refactoring

👍