Pandas Series Construction Extremely Slow for Array of Large Series

[ChoiwahChow]

Code Sample, a copy-pastable example if possible

# Your code here

Problem description

[this should explain why the current behaviour is a problem and why the expected output is a better solution.]

Note: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!

Note: Many problems can be resolved by simply upgrading pandas to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if master addresses this issue, but that is not necessary.

For documentation-related issues, you can check the latest versions of the docs on master here:

https://pandas-docs.github.io/pandas-docs-travis/

If the issue has not been resolved there, go ahead and file it in the issue tracker.

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

[ChoiwahChow]

Code Sample:

import pandas as pd

a = pd.Series([[100.0]]*1000000)
b = pd.Series([a]*100)

Problem description:
The above code snippet completes in less than 1 ms when run with pandas version 0.22.0, but takes 3.376 seconds when run with pandas version 0.23.4 - a 3000-fold longer. We do have use cases that are impacted by this change.

It is easy to trace through the code that in the new version, the bottleneck is in line 1228 in pd.core.dtypes.cast.construct_1d_object_array_from_listlike:

    result[:] = values

The original performance can be restored by replacing the above line with

for i, value in enumerate(values):
    result[i] = value

pd.show_versions() for pandas 0.23.4:
INSTALLED VERSIONS

commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.4
scipy: 1.2.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.3
patsy: 0.5.0
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

pd.show_versions() for pandas 0.22.0
INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: 3.4.0
pip: 9.0.1
setuptools: 38.5.1
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.4
feather: None
matplotlib: 3.1.2
openpyxl: 2.5.0
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: None
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

Could you please check your code sample? I couldn't copy / paste as the variable a is undefined

[ChoiwahChow]

Sorry, missed copying one line
a = pd.Series([[100.0]]*1000000)

That is, a is a pandas series of a million rows.

[WillAyd]

Do you see same performance regression on master? IIRC we fixed something with this recently @jbrockmendel

[TomAugspurger]

Looks like a dupe of #24368

The original performance can be restored by replacing the above line with

for i, value in enumerate(values):
    result[i] = value

does that pass all the constructor tests? Especially for Series with nested data of different lengths?

[ChoiwahChow]

From the discussion chain of 23368, I don't think the problem was fixed yet. In fact, performance degradation was worse on 0.24.1 and on master - on 0.23.4, it was 3.4 second, but from 0.24.1 onward, the same test takes over 12 seconds.

[ChoiwahChow]

As for #23368, it does have a lot in common with this one, although that one seems to be addressing a lot other issues with dataframes.
As for the solution for this problem, no, I have not started thorough testing (although it does work for our use cases), as there may be a lot of contending (or even contradicting) ways of fixing the same problem, so it is just a starting point.
Judging from the code in master, there are more places to fix:

        if dtype is not None:
            try:
                subarr = _try_cast(data, False, dtype, copy,
                                   raise_cast_failure)
            except Exception:
                if raise_cast_failure:  # pragma: no cover
                    raise
                subarr = np.array(data, dtype=object, copy=copy)
                subarr = lib.maybe_convert_objects(subarr)

        else:
            subarr = maybe_convert_platform(data)

A new place to fix is

                subarr = np.array(data, dtype=object, copy=copy)

It is just as slow as the "else" branch (this branch eventually calls construct_1d_object_array_from_listlike, which is slow) that follows.

[jreback]

are you actually trying to create a Series with list elements?

[ChoiwahChow]

As a work around, we can use numpy array of pd.Series. But this requires the user to actually know the pitfall.

[WillAyd]

What is the use case for doing this? Understood it's a performance regression but unless I'm missing something why wouldn't you opt for a DataFrame here?

Just asking as I'd be hesitant to burden the codebase with a potential fix for this if it doesn't have a practical application

[TomAugspurger]

From the discussion chain of 23368, I don't think the problem was fixed yet.

Right, it's still open.

why wouldn't you opt for a DataFrame here?

Potentially for ragged arrays, where each element is of a different length.

---

@ChoiwahChow can you edit the original post to include a (nicely formatted) minimal example? We'll keep this issue open specifically for the Series constructor, likely focusing on fixing or avoiding the line in pd.core.dtypes.cast.construct_1d_object_array_from_listlike. We'll leave #24368 for the dataframe constructor.