BUG: Pyarrow dependency warning starts with newline which makes it impossible to filter out by message with -W or PYTHONWARNINGS

[lesteve]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Issue Description

I would expect that the Pyarrow warning is easily ignorable by message i.e. something like the following does not produce any warning:

python -W 'ignore:Pyarrow:DeprecationWarning' -c 'import pandas'

The issue is that because the warning starts with a newline it can not be targetted by message. This seems to be the case of any warning that starts with some kind of whitespace character.

I tried a variety of things like -W 'ignore:\nPyarrow:DeprecationWarning' but I could not make it work. I think the reason is the "ignoring any whitespace at the start or end of message" from the warnings documentation:

In -W and PYTHONWARNINGS, message is a literal string that the start of the warning message must contain (case-insensitively), ignoring any whitespace at the start or end of message.

In scikit-learn tests warnings are turned into errors with pytest -W and we would like to use -W too to ignore the Pyarrow dependency warning for now. A possible work-around is to use filterwarnings which accept a regex, see Note in https://docs.pytest.org/en/stable/how-to/capture-warnings.html#controlling-warnings for more details, but being able to use -W or PYTHONWARNINGS seems desirable e.g. to control warnings from the command-line outside of pytest.

Expected Behavior

There is an easy way with -W or PYTHONWARNINGS to ignore the Pyarrow DeprecationWarning e.g.

python -W 'ignore:Pyarrow:DeprecationWarning' -c 'import pandas'

Installed Versions

INSTALLED VERSIONS ------------------ commit : f538741 python : 3.11.4.final.0 python-bits : 64 OS : Linux OS-release : 6.7.0-arch3-1 Version : #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14 +0000 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2
Cython : 0.29.35
pytest : 7.4.0
hypothesis : None
sphinx : 6.0.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.18.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.0
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.12.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None

[jorenham]

This works:
(?s).*Pyarrow will become a required dependency of pandas:DeprecationWarning"

[lesteve]

python -W or PYTHONWARNINGS does not handle regexes, see above link to the Python documentation, so no this does not work:

❯ python -W 'ignore:(?s).*Pyarrow:DeprecationWarning' -c 'import pandas' 
<string>:1: DeprecationWarning: 
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466

As the doc says, regexes are accepted in warning.filterwarnings and your regex likely works with warning.filterwarnings

[jorenham]

Ah I didn't know that. I assumed it would, since it did work within my pytest config.

[lesteve]

Yeah filterwarnings accepts regex contrary to -W quite a quirk which I did not know about either before very recently. This is mentioned in the pytest doc: https://docs.pytest.org/en/stable/how-to/capture-warnings.html#controlling-warnings

[rhshadrach]

Thanks for the report, a linked PR is already up to resolve this.

[lesteve]

Nice, thanks! This is #57003 indeed. I looked for an open PR but didn't find it somehow ...