Poor Performance

[nathancole]

I'm experiencing very poor performance with large datasets. We're loading 10000 values (just random values) and it becomes almost unusable.

[lblb]

Hi, this is expected, because C3 is based on D3, which uses SVG to render graphics. Every SVG element is created as a separate dom node in browser. Hence the more elements, the more nodes you have to deal with. Consequently performance drops exponential with new elements added. You can find many tests online comparing SVG vs Canvas vs Flash performance and you'll see that SVG is reasonably fast until number of elements is relatively small. You can also get a proof yourself: create a test function that draw and animate 10000 circles alone and you'll see CPU getting hot. You can take a look at this example to get the first impression as well: http://tommykrueger.com/projects/d3tests/performance-test.php

The only way to work with large datasets in SVG library like C3 is to reduce or aggregate values before plotting them. Unfortunately C3 doesn't have any methods for that. My experience is that 500 members in a series is upper limit after which charts become a mess (both visually and performance vice). So it's necessary to design interface of page differently with drill down functionality, so that user could always look for fine details if needed.

p.s.
dc.js is a nice charting library that uses crossfilter for data reduction. (http://nickqizhu.github.io/dc.js/). But it has it's own limitations too.

[masayuki0812]

I found this issue before and agree with @lblb . It would be difficult to improve the performance dramatically without reducing the number of elements. I'll consider to introduce a feature for reducing or aggregation of data.

[lblb]

Hi Masayuki, data analysis is a an extensive topic on it own. I'd better suggest focus on charting only and leave that alone. In my case, I do data manipulations on server side and retrieve results on demand. When using knockout.js it's rather easy to bind multiple charts and have them react to data changes. Your functions to load and remove data (and also transform chart to different type) are awesome and work well for me. Other users may prefer retrieving large datasets once and do further manipulations on client side (probably using worker thread so that UI stays responsive). There is no standard approach that fits everyone needs.

I may suggest adding notes to subchart and zoom functions of C3 explaining that they are just experimental and not intelligent at all. User may get a false impression that you can dump megadata to the chart and then zoom in for details.

[masayuki0812]

@lblb Thank you for the advice. I agree with you and I'll add some notes about the amount of data c3 can process. I know the documentation is completely poor, so I'll add as soon as I can.

[kjavia]

We found Crossfilter.js library (also written by Mike Bostok who wrote D3) to be extremely useful for data aggregation client side. Integrating Crossfilter and C3.js on the same app is pretty easy and no need to add aggregation features in C3.

[masayuki0812]

Hi @KetanJavia , Thank you for telling. I'll try that library.

[masayuki0812]

Please try this extension implemented by @danelkhen as #310 . Sample is here:
https://github.com/masayuki0812/c3/blob/master/htdocs/samples/zoom_reduction.html

I think this would a effective solution, so I'll close this issue now.

[danelkhen]

Thanks masayuki0812, if anyone needs help using this feature let me know. Cheers.

[webxl]

I think this should be reopened. Just because this library uses SVG/d3 doesn't mean it can't handle large data sets performantly.

As a stress test, I have a dataset consisting of 1000 series, with 36 points each. Here are stats from several other SVG-based libraries and C3 (I've turned off points, tooltips, etc. for each chart), running under Canary:

Rickshaw: childCount: 2038, listenerCount: 0, renderTime: 0.26s
Vega: childCount: 4063, listenerCount: 2, renderTime: 1.12s
NVD3: childCount: 39092, listenerCount: 0, renderTime: 4.96s
Highcharts: childCount: 4055, listenerCount: 1, renderTime: 10.25s
C3: childCount: 18354, listenerCount: 3220, renderTime: 26.22s

I don't know if you can't have all the nice interactivity and transition features without the current architecture, but seems like its worth a try. Seems like if I turn off the interactive features, all event listeners should be disabled.

Let me know if you'd like to see my test code.

[masayuki0812]

Hi, Thank you for looking into this. I think you're right and there must be some room to optimise. We're going to refactor the architecture, so I'll work on this issue in it too.
Could you show us the test code? It would be really helpful if we can see it. Thanks!

[lblb]

This is interesting, I'm curious to see your testing code as well. It's strange that Rickshaw produced only 2038 child nodes, are you sure that "line smoothing" (interpolation) methods were not active there? This will reduce number of elements actually drawn.

Also please make sure that you disable C3 onresize listeners. By default C3 does a redraw if dimensions of bindto container change. For such a large chart dynamic size scale is definitely a bad idea. I believe all the libraries you compared to has onresize feature disabled (or don't have at all).