Insertion speed of new dataset elements

[hmaarrfk]

What is your issue?

In #7221 I showed that a major contributor the slowdown in inserting a new element was the cost associated with an internal only debugging assert statement.

The benchmarks results 7221 and 7222 are pretty useful to look at.

Thank you for encouraging the creation of a "benchmark" so that we can monitor the performance of element insertion.

Unfortunately, that was the only "free" lunch I got.

A few other minor improvements can be obtained with:
#7222

However, it seems to me that the fundamental reason this is "slow" is because element insertion is not so much "insertion" as it is:

• Dataset Merge
• Dataset Replacement of the internal methods.

This is really solidified in the https://github.com/pydata/xarray/blob/main/xarray/core/dataset.py#L4918

In my benchmarks, I found that in the limit of large datasets, list comprehensions of 1000 elements or more were often used to "search" for variables that were "indexed"

xarray/xarray/core/merge.py

Line 267 in ca57e5c

indexed_elements = [

I think a few speedsups can be obtained by avoiding these kinds of "searches" and list comprehensions. However, I think that the dataset would have to provide this kind of information to the merge_core routine, instead of the merge_core routine recreating it all the time.

Ultimately, I think you trade off "memory footprint" (due to the potential increase of datastructures you keep around) of a dataset, and "speed".

Anyway, I just wanted to share where I got.

[Illviljan]

I have thought a little about this as well and went and looked in my old code. Creating a data_vars dict with the data and then at the end creating the dataset seems to be the way to go:

import numpy as np
import xarray as xr
from time import perf_counter


# %% Inputs
names = np.core.defchararray.add("long_variable_name", np.arange(0, 100).astype(str))
time = np.array([0, 1])
coords = dict(time=time)
value = np.array(["0", "b"], dtype=str)

# %% Insert to Dataset with DataArray:
time_start = perf_counter()

ds = xr.Dataset(coords=coords)
for v in names:
    ds[v] = xr.DataArray(data=value, coords=coords)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Insert to Dataset with DataArray:", time_elapsed)

# %% Insert to Dataset with Variable:
time_start = perf_counter()

ds = xr.Dataset(coords=coords)
for v in names:
    ds[v] = xr.Variable("time", value)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Insert to Dataset with Variable:", time_elapsed)

# %% Insert to Dataset with tuple:
time_start = perf_counter()

ds = xr.Dataset(coords=coords)
for v in names:
    ds[v] = ("time", value)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Insert to Dataset with tuple:", time_elapsed)


# %% Dict of DataArray then create Dataset:
time_start = perf_counter()

data_vars = dict()
for v in names:
    data_vars[v] = xr.DataArray(data=value, coords=coords)
ds = xr.Dataset(data_vars=data_vars, coords=coords)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Dict of DataArrays then create Dataset:", time_elapsed)

# %% Dict of Variables then create Dataset:
time_start = perf_counter()

data_vars = dict()
for v in names:
    data_vars[v] = xr.Variable("time", value)
ds = xr.Dataset(data_vars=data_vars, coords=coords)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Dict of Variables then create Dataset:", time_elapsed)

# %% Dict of tuples then create Dataset:
time_start = perf_counter()

data_vars = dict()
for v in names:
    data_vars[v] = ("time", value)
ds = xr.Dataset(data_vars=data_vars, coords=coords)

time_end = perf_counter()
time_elapsed = time_end - time_start
print("Dict of tuples then create Dataset:", time_elapsed)

Insert to Dataset with DataArray: 0.3787728999996034
Insert to Dataset with Variable: 0.3083788999997523
Insert to Dataset with tuple: 0.30018929999960164
Dict of DataArrays then create Dataset: 0.07277609999982815
Dict of Variables then create Dataset: 0.005166500000086671
Dict of tuples then create Dataset: 0.003186699999787379  # Winner! :)

[hmaarrfk]

Ok, I don't think I have the right tools to really get to the bottom of this. The spyder profiler just seems to slowdown code too much. Any other tools to recommend?

[hmaarrfk]

xref: pandas-dev/pandas#49393