CI: matplotlib 3.7.0 broke ci

[phofl]

Started seeing this on the arm builds

https://app.circleci.com/pipelines/github/pandas-dev/pandas/24736/workflows/0c1b7766-2149-40ab-9130-18e134e87e4c/jobs/56289

[mroeschke]

I think matplotlib/matplotlib#23573 impacted pandas/tests/plotting/frame/test_frame.py::TestDataFramePlots::test_plain_axes

[mroeschke]

Opened up matplotlib/matplotlib#25219 for the other failures