Graph_obj.py slow performance when .append_trace on large amount of subplots

[depettit]

I just started using Plotly with Python and I think I have found a significant speed up improvement in the graph_objs.py module.

Here are some details:

• I am creating a plot with 24 subplots that has 60 traces in each subplot
  • Each trace contains 6 points.
• To generate this plot it took 1945 seconds, which seems like way too long
  • I had a print state for each subplot and it progressively takes longer for each subplot
• I ran “kernprof” on the code looking specifically at the subroutine that I use to loop through the data to .append_trace to the figure
  • I noticed that all the time was spent on the .append_trace function that is called out of graph_objs.py
• As I dug through graph_objs.py and found that line 936 contained “self[‘data’] += [trace]”
  • In my limited understanding of python, I always heard this was a very inefficient way to do large string appends
• I changed line 936 to “self[‘data’].append(trace)”, which I believe is equivalent in function to “self[‘data’] += [trace]” but I think I have read that this is a much more efficient way to append
  • This sped up the script to 8 seconds! A 243x improvement in speed.

From what I can tell, I think it is valid to change “self[‘data’] += [trace]” to “self[‘data’].append(trace)” but I am not sure of all the ramifications to other functionality. Is this a valid change to make and if so can graph_objs.py be updated with this?

Thanks!

[yankev]

@depettit Hey thanks for suggesting this. The speedup is because using the += method creates a whole new list somewhere in memory, which would take a long time if your original list was already very large, in addition to adding the new trace. Where as appending a trace only incurs a cost of adding the trace object.

[yankev]

Hmm though it seems that we over-ride the += with extend here: https://github.com/plotly/plotly.py/blob/master/plotly/graph_objs/graph_objs.py#L180-L183

So technically the change shouldn't have much of an effect, because extending a trace should be equivalent to appending it. However, if this results in significant speed improvements, I don't think it hurts to change it to append().

@theengineear: What do you think?

[depettit]

Kevin, you asked if I used go.Data when defining my trace. I think I did but I don't have access to my code at the moment. Here is a snap shot that I was able to dig out of my email.

print "Plotting "+str(title)+" in Location: "+str(yLoc)+","+str(xLoc)
for dataCond in plotOver:
temp_df = df
dataCond = list(dataCond)
for i in range(len(dataCond)):
temp_df = temp_df[temp_df[dataNames[i]] == dataCond[i]]
trace = go.Scatter(name=str(dataCond), x=temp_df[xAxisName], y=temp_df[yAxisName], mode = 'lines+markers')
fig.append_trace(trace, yLoc, xLoc)

[theengineear]

@depettit

If you do have an instance of Data, I wouldn't expect much gain since extend just maps to append.

very inefficient way to do large string appends

Note that this is not a string append it's a dict that's being appended to a list so neither self[‘data’] += [trace] nor self[‘data’].append(trace) would create a large object in memory. trace in this case is just a dict (a mutable pointer to a Python object), there's no copy happening. The only thing being being extraneously created is a length one list holding a reference to a dict.

Now, that said. We do some validation on incoming objects, so the validation step may actually be the thing that has bottlenecks:

https://github.com/plotly/plotly.py/blob/master/plotly/graph_objs/graph_objs.py#L236

As a first step so either myself or @yankev can confirm the speedups you're seeing, would you mind:

Pasting a test script in here so that we have some standard piece of test code to compare with

Thanks for looking into this with a profiler! We haven't had time to do that and I'm sure there are plenty of bottlenecks that could be fixed!

[depettit]

@theengineear

I am new to the idea of using classes so I didn't really use them.

I may be doing this inefficiently by using definitions.

I create the figure using:

fig = tls.make_subplots(rows=len(vplotConds), cols=len(hplotConds), subplot_titles = subPlotTitles)

I pass 'fig' to a new definition that loops through and append_trace the data to the subplot.

for dataCond in plotOver:
    temp_df = df
    dataCond = list(dataCond)
    for i in range(len(dataCond)):
        temp_df = temp_df[temp_df[dataNames[i]] == dataCond[i]]
    trace = go.Scatter(name="", x=temp_df[xAxisName], y=temp_df[yAxisName], mode = 'lines+markers', text = str(dataCond), hoverinfo='x+y+text')
    fig.append_trace(trace, yLoc, xLoc)

Some explanation: I pass a dataframe 'df' in the subroutine that then getes reduced down to a unique set of XY data. 'dataCond' is a list of data conditions that reduces the 'df'. The scatter trace doesn't have a name but rather uses the hover info to display the trace name. This keeps the trace name from being cut off.

I can't give you the data file that I am working with that the 'df' is generated from but I know that the bottle neck is in the 'fig.append_trace' command. If I need to come up with a dummy data set, I can try to accommodate.

If there is anything from my end that I can change in the code and test, I can easily revert back to the original '+=' method.

I am always open to learn more!

[kb1lqc]

Can confirm just ran into this using Plotly 1.13.0. Waited 15 minutes for a plot to finish without it actually finishing, I stopped it. My dataset has roughly 10 figures (Histograms) with about 130 traces per histogram.

Changing https://github.com/plotly/plotly.py/blob/master/plotly/graph_objs/graph_objs.py#L1213 to

self['data'].append(trace)

resulted in completely finishing my graph in < 30 seconds. I am not sure if the append change is the best option, it appears @theengineear has a good handle on how the code should be written. I can confirm however this was a huge PITA working with a large dataset and am happy to let you all know it's still there and there appears to be something that fixes it.

Unfortunately my dataset is proprietary so I can't send anything from it.

[theengineear]

@depettit and @kb1lqc thanks for the info and linking to the line that's causing the bottleneck! I'll take a peek this weekend (should have some free cycles). If it's as easy as changing that line, I'll make a new release (though it will be in 2.0.0 at this point).

If it's not as easy as changing that line, I'll at least drop some breadcrumbs in here.

Thanks for all the info reproducibility help!

[theengineear]

As-promised. I dug in a little this weekend. I looks like creating the figure in the first place is taking a ton of time. This mostly happens because of the validation we do to ensure that graph_objects remain valid.

My current approach is to basically memoize the computation-heavy calls that happen in graph_reference.py. I say basically memoize because one of the functions allows a * value for the value argument which wouldn't be helpful to memoize.

I was going to use lru_cache and wrap that * value in another function which can me memoized, but it's not available in Python < 3.2.

The better solution for the future would be to better-parallelize the options to use graph_objs vs native Python dicts when making subplots. At the core, you should always be able to use dicts and lists (much, much more performant, but no validation) in place of the graph_objs that plotly supplies.

One goal of this library is to default to hand-holding to make the plotting attributes/syntax as self-discoverable as possible, BUT, allow power-users to turn the hand-holding off when it gets in the way of performance.

I'll ping you when I get a PR up.

[theengineear]

I'm not quite to an order of magnitude speedup, but getting there!

# debug.py
import time
from plotly import tools as tls
t0 = time.time()
tls.make_subplots(rows=100, cols=100, print_grid=False)
print(time.time() - t0)

# Current master
$ python debug.py
8.2746360302

# With changes
$ python debug.py
1.21838903427

[Teoretic6]

I'm afraid we should reopen the issue.

'+=' is still there:
https://github.com/plotly/plotly.py/blob/master/plotly/graph_objs/graph_objs.py#L1279

The significantly slow perfomance in "append_trace" is still there.

Still fixes by changing:
self['data'] += [trace]
to
self['data'].append(trace)