BUG: bar a line plots are not aligned on the x-axis/xticks

[diegodebrito]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame(
    {
        "bars": {
            1: 0.5,
            2: 1.0,
            3: 3.0,
            4: 3.5,
            5: 1.5,
        },
        "pct": {
            1: 4.0,
            2: 2.0,
            3: 2.0,
            4: 2.0,
            5: 8.0,
        },
    }
)

ax=df["bars"].plot(kind="bar")
df["pct"].plot(kind="line", ax=ax,)

Issue Description

Bar and line plot are not aligned on the x-axis when plotting with Pandas. I saw some somewhat related issues, but they were not exactly this type of plot.

This is the plot generated from the sample code above:

Expected Behavior

Line also starts in index=1, and not index=2 like in the plot above.

Installed Versions

INSTALLED VERSIONS

commit : a671b5a
python : 3.11.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.1.4
numpy : 1.26.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.2.1
Cython : None
pytest : 7.4.3
hypothesis : None
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.17.2
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 14.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.4
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, confirmed on main. Further investigations and PRs to fix are welcome!

[sharonwoo]

I'm new to the codebase but this caught my interest because I've never encountered it, but it would be really annoying if I did. I'm still trying to understand the code but while reading it I tried to get a sense of where to look by plotting a few graphs myself.

• In the above example, if it's only lines that are plotted, they are plotted on the correct axes until a bar graph gets added. Then the lines all get shifted by 1.

• Also if the axes ran from 0-4, the graph is plotted normally, ie with the bars and lines on the correct axes:

import pandas as pd

df = pd.DataFrame(
    {
        "bars": {
            0: 0.5,
            1: 1.0,
            2: 3.0,
            3: 3.5,
            4: 1.5,
        },
        "pct": {
            0: 4.0,
            1: 2.0,
            2: 2.0,
            3: 2.0,
            4: 8.0,
        },
        "test": {
            0: 8.0,
            1: 1.0,
            2: 2.0,
            3: 2.0,
            4: 10.0,
        },
    }
)

ax1 = df["pct"].plot(kind="line")
ax2 = df["bars"].plot(kind="bar")
ax3 = df["test"].plot(kind="line")

But if I change my axes for all 3 to 3-7, then I get a shift of 3 instead for the lines (starting at 6 instead of 3):

Code to reproduce this to save some editing:

import pandas as pd

df = pd.DataFrame(
    {
        "bars": {
            3: 0.5,
            4: 1.0,
            5: 3.0,
            6: 3.5,
            7: 1.5,
        },
        "pct": {
            3: 4.0,
            4: 2.0,
            5: 2.0,
            6: 2.0,
            7: 8.0,
        },
        "test": {
            3: 8.0,
            4: 1.0,
            5: 2.0,
            6: 2.0,
            7: 10.0,
        },
    }
)

ax1 = df["pct"].plot(kind="line")
ax2 = df["bars"].plot(kind="bar")
ax3 = df["test"].plot(kind="line")

Annnnnd consistent with expectations, we get a shift of -1 for the lines if the axes all start with -1.

[sharonwoo]

Opened an incomplete PR (logic only, I need to fix CI) for further discussion - this would fix the issue, but should we fix it in the first place?

Tldr, BarPlot uses self.tick_pos = np.arange(len(data)) to set x, and I've put in some logic that lets it do something similar to LinePlot if the data is a series.

[sharonwoo]

take

[rhshadrach]

Thanks for the investigations here. We also get incorrect results if all the Series used for the plots do not have the same index, e.g.

bars = pd.Series({"a": 0.5, "b": 1.0})
pct = pd.Series({"b": 4.0, "a": 2.0})
ax = bars.plot(kind="bar")
pct.plot(kind="line", ax=ax)

It does seem reasonable to me for users to expect the order of the bars is preserved, even for numeric indexes:

pct = pd.Series({1: 4.0, 10: 2.0, 3: 3.0})
pct.plot(kind="bar")

The bars currently appear with x-ticks 1, 10, 3.

It seems difficult to determine appropriate results when there are different indexes in the data in general. I wonder if when plotting and ax is provided we can just detect when the xticks do not agree.

[sharonwoo]

Thanks for the thoughtful comment, let me investigate further.

I found some related issues raised previously: #55508, #50508, #48806

[sharonwoo]

Thanks for the investigations here. We also get incorrect results if all the Series used for the plots do not have the same index, e.g.

bars = pd.Series({"a": 0.5, "b": 1.0})
pct = pd.Series({"b": 4.0, "a": 2.0})
ax = bars.plot(kind="bar")
pct.plot(kind="line", ax=ax)

It does seem reasonable to me for users to expect the order of the bars is preserved, even for numeric indexes:

pct = pd.Series({1: 4.0, 10: 2.0, 3: 3.0})
pct.plot(kind="bar")

The bars currently appear with x-ticks 1, 10, 3.

It seems difficult to determine appropriate results when there are different indexes in the data in general. I wonder if when plotting and ax is provided we can just detect when the xticks do not agree.

I tried, and Matplotlib does this chart correctly:

import matplotlib.pyplot as plt
bars = {"a": 1.5, "b": 3.5}
pct = {"b": 4.0, "a": 2.0}

fig, ax = plt.subplots()

ax.bar(bars.keys(), bars.values(), color='blue')
ax.plot(pct.keys(), pct.values(), color='red')

plt.show()

However for the out of order series, the PR implementation is similar to what Matplotlib currently does (ie bars are out of order):

import matplotlib.pyplot as plt
import pandas as pd

pct = pd.Series({1: 4.0, 10: 2.0, 3: 3.0})
plt.figure()
plt.bar(pct.index, pct.values, color='blue')
plt.show()

[rhshadrach]

I haven't been able to find documentation on this, but as far as I can tell matplotlib's behavior is:

• If a series of x values contains both numeric and non-numeric, raise.
• If a series of x values is numeric, use the value to determine location
• If a series of x values is non-numeric, use the position (0, 1, 2, ...) to determine the location

The third bullet point above extends to when a series of non-numeric values is added to the plot: if there is a symbol not yet seen, it is added as the next bar. If it's already been seen, it is stacked.

From the matplotlib docs, I think we may also need to consider "if x has units (e.g. datetime)" as a separate case. I've yet to see how that interacts with the logic above.

It seems to me we can replicate this behavior (MultiIndex entries treated as tuples and hence non-numeric). One thing I'm still wondering is why we don't just offload this logic to matplotlib's default behavior.