Wrong pl.when matching

[EpicUsaMan]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

pl.read_parquet("test.parquet").with_columns(
            pl.when(pl.col("timestamp") < pl.lit(split_date, pl.Datetime("ns")))
            .then(pl.col("price") / (split_to / split_from))
            .otherwise(pl.col("price"))
            .cast(pl.Float64)
            .alias("price"),
        )

split_date is 2020-08-31
split_to = 1
split_from = 10

Log output

┌────────┬───────────┬──────┬───────────────────────────────┬────────────────┬────────────────┬────────────────────┬────────────────┐
│ ticker ┆ price     ┆ size ┆ timestamp                     ┆ not_updates_oc ┆ not_updates_hl ┆ not_updates_volume ┆ extended_hours │
│ ---    ┆ ---       ┆ ---  ┆ ---                           ┆ ---            ┆ ---            ┆ ---                ┆ ---            │
│ cat    ┆ f64       ┆ u32  ┆ datetime[ns]                  ┆ bool           ┆ bool           ┆ bool               ┆ bool           │
╞════════╪═══════════╪══════╪═══════════════════════════════╪════════════════╪════════════════╪════════════════════╪════════════════╡
│ AAPL   ┆ 0.02207   ┆ 153  ┆ 2024-09-03 23:25:53.316335601 ┆ false          ┆ false          ┆ false              ┆ true           │
│ AAPL   ┆ 0.0220675 ┆ 1    ┆ 2024-09-03 23:25:53.905632305 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.022078  ┆ 1    ┆ 2024-09-03 23:25:55.326620515 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.0220665 ┆ 1    ┆ 2024-09-03 23:25:55.482855962 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.0220665 ┆ 1    ┆ 2024-09-03 23:26:02.722102327 ┆ true           ┆ true           ┆ false              ┆ true           │
│ …      ┆ …         ┆ …    ┆ …                             ┆ …              ┆ …              ┆ …                  ┆ …              │
│ AAPL   ┆ 0.02209   ┆ 1    ┆ 2024-09-03 23:59:52.389425558 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.022095  ┆ 50   ┆ 2024-09-03 23:59:52.838793885 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.022095  ┆ 25   ┆ 2024-09-03 23:59:56.590786958 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.0220875 ┆ 1    ┆ 2024-09-03 23:59:57.925981270 ┆ true           ┆ true           ┆ false              ┆ true           │
│ AAPL   ┆ 0.0220875 ┆ 1    ┆ 2024-09-03 23:59:58.891216420 ┆ true           ┆ true           ┆ false              ┆ true           │
└────────┴───────────┴──────┴───────────────────────────────┴────────────────┴────────────────┴────────────────────┴────────────────┘

Issue description

Columns are divided even when they must be not divided

Expected behavior

Columns are not divided even when they must be not divided

Installed versions

--------Version info---------
Polars:              1.6.0
Index type:          UInt32
Platform:            Linux-6.8.0-41-generic-x86_64-with-glibc2.39
Python:              3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0]

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          3.0.0
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               2024.6.1
gevent               24.2.1
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         1.6.0
numpy                1.26.4
openpyxl             <not installed>
pandas               2.2.2
pyarrow              16.1.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           2.0.31
torch                2.3.1+rocm6.0
xlsx2csv             <not installed>
xlsxwriter           <not installed>

[cmdlineluser]

I think this came up in #18375 recently.

price = pl.lit(220.92, pl.Decimal(precision=10, scale=4))

pl.select(
    price = price,
    true = pl.when(True).then(price / 2).otherwise(price),
    false = pl.when(False).then(price / 2).otherwise(price)
)

# shape: (1, 3)
# ┌───────────────┬──────────────┬──────────────┐
# │ price         ┆ true         ┆ false        │
# │ ---           ┆ ---          ┆ ---          │
# │ decimal[10,4] ┆ decimal[*,8] ┆ decimal[*,8] │
# ╞═══════════════╪══════════════╪══════════════╡
# │ 220.9200      ┆ 110.46000000 ┆ 0.02209200   │
# └───────────────┴──────────────┴──────────────┘

You end up with different scale/precision even in the false case.

(I don't know enough about Decimal to know what is the expected result is here.)

[mcrumiller]

Can you include the input data? Also, your output data doesn't have a price_new column, although your code creates one, so the output is not from the code.

If we can just copy/paste the code and run it, it is much easier to diagnose, i.e. can you also define your split_date, split_to, and split_from variables?

[EpicUsaMan]

Can you include the input data? Also, your output data doesn't have a price_new column, although your code creates one, so the output is not from the code.

If we can just copy/paste the code and run it, it is much easier to diagnose, i.e. can you also define your split_date, split_to, and split_from variables?

Edited code, but we can actually close it

It's really decimal issue here

[mcrumiller]

Edited code, but we can actually close it

Thanks--but FYI, from your edited code, we cannot see the old price value so there is still no way to diagnose.

[EpicUsaMan]

test.zip

Here is test file

Edited code, but we can actually close it

Thanks--but FYI, from your edited code, we cannot see the old price value so there is still no way to diagnose.

[cmdlineluser]

The then/otherwise values have their "supertype" computed so there is essentially an implicit cast going on.

price = pl.lit(1.5)

pl.select(
    price = price,
    true  = pl.when(True).then(pl.lit("foo")).otherwise(price),
    false = pl.when(False).then(pl.lit("foo")).otherwise(price)
)

# shape: (1, 3)
# ┌───────┬──────┬───────┐
# │ price ┆ true ┆ false │
# │ ---   ┆ ---  ┆ ---   │
# │ f64   ┆ str  ┆ str   │ # false=str
# ╞═══════╪══════╪═══════╡
# │ 1.5   ┆ foo  ┆ 1.5   │
# └───────┴──────┴───────┘

I'm not sure if there are plans for a strict version of this operation?

[EpicUsaMan]

I think everything must be running in strict by default

Because data must be safe and only if you are know what you are doing -- only then you should disable "strict"

It's exactly the way how rust behaves, so I believe that polars must go with the same approach

[cmdlineluser]

Yes, I know there are plans to change it for fill_null where a similar "gotcha" exists: #13789

pl.DataFrame({"foo": [1, None]}).select(pl.col.foo.fill_null("two"))
# shape: (2, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ str │ # <- cast
# ╞═════╡
# │ 1   │
# │ two │
# └─────┘

Perhaps the same will happen for when/then?

The Decimal example was extra confusing to me as I did not realise the arithmetic operation was changing the "type".