lazy join_asof + select with by_left != by_right leads to column not found error (eager api functions correctly)

[mrwelty]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df1 = pl.DataFrame({'on1':0,'by1':0})
df2 = pl.DataFrame({'on1':0,'by2':0})

# lazy join_asof + select errors when by_left != by_right 

(
    df1.lazy()
    .join_asof(
        df2.lazy(),
        on = 'on1',
        by_left = 'by1',
        by_right = 'by2',
    )
    .select(pl.all())
    .collect()
)

# -> PanicException: called `Result::unwrap()` on an `Err` value: ColumnNotFound(ErrString("by2"))

Log output

thread '<unnamed>' panicked at crates/polars-plan/src/utils.rs:358:79:
called `Result::unwrap()` on an `Err` value: ColumnNotFound(ErrString("by2"))
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[4], line 17
      4 df2 = pl.DataFrame({'on1':0,'by2':0})
      6 # lazy join_asof + select errors when by_left != by_right 
      8 (
      9     df1.lazy()
     10     .join_asof(
     11         df2.lazy(),
     12         on = 'on1',
     13         by_left = 'by1',
     14         by_right = 'by2',
     15     )
     16     .select(pl.all())
---> 17     .collect()
     18 )

File /opt/conda/lib/python3.12/site-packages/polars/lazyframe/frame.py:2026, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, no_optimization, streaming, engine, background, _eager, **_kwargs)
   2024 # Only for testing purposes
   2025 callback = _kwargs.get("post_opt_callback", callback)
-> 2026 return wrap_df(ldf.collect(callback))

PanicException: called `Result::unwrap()` on an `Err` value: ColumnNotFound(ErrString("by2"))

Issue description

Within the lazy api, performing a 'join_asof' when the 'by_left' and 'by_right' keys are different followed by a 'select' statement results in a column not found error. The same operation done within the eager api functions properly (no error).

In this example I have used '.select(pl.all())' however the issue [I believe] is not specific to using 'pl.all()'. I also suspect it is not a coincidence that the column which ends up being not found [hence throwing the error] is the 'by_right' key.

Expected behavior

import polars as pl

df1 = pl.DataFrame({'on1':0,'by1':0})
df2 = pl.DataFrame({'on1':0,'by2':0})

### Related examples below which do not error ###

# eager join_asof + select
# (performing the same operations which resulted in the bug but now via the eager api)
(
    df1
    .join_asof(
        df2,
        on = 'on1',
        by_left = 'by1',
        by_right = 'by2',
    )
    .select(pl.all())
)

# lazy rename + join_asof + select 
# (renaming the by_right column to match by_left before joining circumvents this bug)
(
    df1.lazy()
    .join_asof(
        df2.lazy().rename({'by2':'by1'}),
        on = 'on1',
        by_left = 'by1',
        by_right = 'by1',
    )
    .select(pl.all())
    .collect()
)

# lazy join_asof + with_columns 
# (naively I would have assumed .select(pl.all()) was equivalent to .with_columns(), this bug suggests otherwise)
(
    df1.lazy()
    .join_asof(
        df2.lazy(),
        on = 'on1',
        by_left = 'by1',
        by_right = 'by2',
    )
    .with_columns()
    .collect()
)

Installed versions

--------Version info---------
Polars:               1.3.0
Index type:           UInt32
Platform:             Linux-6.6.22-linuxkit-x86_64-with-glibc2.36
Python:               3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          2.2.1
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2024.3.1
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               0.10.0
matplotlib:           3.8.4
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             3.1.2
pandas:               2.2.2
pyarrow:              14.0.2
pydantic:             2.5.3
pyiceberg:            <not installed>
sqlalchemy:           2.0.30
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[cmdlineluser]

Can reproduce. (It seems to be a little worse, as it's actually causing a PanicException)

If it's useful: the eager API is essentially a wrapper around: lazy().collect(no_optimization=True)

In this instance, it seems to be related to the projection_pushdown optimization:

.collect(projection_pushdown=False)

shape: (1, 2)
┌─────┬─────┐
│ on1 ┆ by1 │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 0   ┆ 0   │
└─────┴─────┘