Create a new hook to enable data decimation (#8255)

* Create a new hook to enable data decimation

The `beforeElementUpdate` hook can be used to decimate data. The chart
elements will not be created until after this hook has fired ensuring that
if decimation occurs, only the needed elements will be created.

* Address code review feedback

* Rename hook to beforeElementsUpdate

* Simplify parsing logic

* Add decimation plugin to the core

* Allow a dataset to specify a different data key

* Decimation plugin uses the dataKey feature

* Refactor the decimation plugin to support configurable algorithms

* Lint the plugin changes

* Tests for the dataKey feature

* Convert test files to tabs

* Standardize on tabs in ts files

* Remove the dataKey feature

* Replace dataKey usage in decimation plugin

We define a new descriptor for the `data` key allowing the
plugin to be simpler.

* Disable decimation when indexAxis is Y

* Simplify the decimation width approximation

* Resolve the indexAxis correctly in all cases

* Initial documentation

* Reverse check

* Update TS definitions for new plugin options

* Move defineProperty after bailouts

* Add destroy hook

Author: etimberg

docs/docs/configuration/decimation.md

@@ -0,0 +1,33 @@
+---
+title: Data Decimation
+---
+
+The decimation plugin can be used with line charts to automatically decimate data at the start of the chart lifecycle. Before enabling this plugin, review the [requirements](#requirements) to ensure that it will work with the chart you want to create.
+
+## Configuration Options
+
+The decimation plugin configuration is passed into the `options.plugins.decimation` namespace. The global options for the plugin are defined in `Chart.defaults.plugins.decimation`.
+
+| Name | Type | Default | Description
+| ---- | ---- | ------- | -----------
+| `enabled` | `boolean` | `true` | Is decimation enabled?
+| `algorithm` | `string` | `'min-max'` | Decimation algorithm to use. See the [more...](#decimation-algorithms)
+
+## Decimation Algorithms
+
+Decimation algorithm to use for data. Options are:
+
+* `'min-max'`
+
+### Min/Max Decimation
+
+[Min/max](https://digital.ni.com/public.nsf/allkb/F694FFEEA0ACF282862576020075F784) decimation will preserve peaks in your data but could require up to 4 points for each pixel. This type of decimation would work well for a very noisy signal where you need to see data peaks.
+
+## Requirements
+
+To use the decimation plugin, the following requirements must be met:
+
+1. The dataset must have an `indexAxis` of `'x'`
+2. The dataset must be a line
+3. The X axis for the dataset must be either a `'linear'` or `'time'` type axis
+4. The dataset object must be mutable. The plugin stores the original data as `dataset._data` and then defines a new `data` property on the dataset.

docs/docs/general/performance.md

@@ -18,7 +18,7 @@ Chart.js is fastest if you provide data with indices that are unique, sorted, an
 
 Decimating your data will achieve the best results. When there is a lot of data to display on the graph, it doesn't make sense to show tens of thousands of data points on a graph that is only a few hundred pixels wide.
 
-There are many approaches to data decimation and selection of an algorithm will depend on your data and the results you want to achieve. For instance, [min/max](https://digital.ni.com/public.nsf/allkb/F694FFEEA0ACF282862576020075F784) decimation will preserve peaks in your data but could require up to 4 points for each pixel. This type of decimation would work well for a very noisy signal where you need to see data peaks.
+The [decimation plugin](./configuration/decimation.md) can be used with line charts to decimate data before the chart is rendered. This will provide the best performance since it will reduce the memory needed to render the chart.
 
 Line charts are able to do [automatic data decimation during draw](#automatic-data-decimation-during-draw), when certain conditions are met. You should still consider decimating data yourself before passing it in for maximum performance since the automatic decimation occurs late in the chart life cycle.
 

docs/sidebars.js

@@ -30,7 +30,8 @@ module.exports = {
 			'configuration/legend',
 			'configuration/title',
 			'configuration/tooltip',
-			'configuration/elements'
+			'configuration/elements',
+			'configuration/decimation'
 		],
 		'Chart Types': [
 			'charts/line',

src/core/core.controller.js

@@ -466,9 +466,15 @@ class Chart {
 		// Make sure dataset controllers are updated and new controllers are reset
 		const newControllers = me.buildOrUpdateControllers();
 
+		me.notifyPlugins('beforeElementsUpdate');
+
 		// Make sure all dataset controllers have correct meta data counts
 		for (i = 0, ilen = me.data.datasets.length; i < ilen; i++) {
-			me.getDatasetMeta(i).controller.buildOrUpdateElements();
+			const {controller} = me.getDatasetMeta(i);
+			const reset = !animsDisabled && newControllers.indexOf(controller) === -1;
+			// New controllers will be reset after the layout pass, so we only want to modify
+			// elements added to new datasets
+			controller.buildOrUpdateElements(reset);
 		}
 
 		me._updateLayout();

src/core/core.datasetController.js

@@ -349,19 +349,12 @@ export default class DatasetController {
 
 		me._dataCheck();
 
-		const data = me._data;
-		const metaData = meta.data = new Array(data.length);
-
-		for (let i = 0, ilen = data.length; i < ilen; ++i) {
-			metaData[i] = new me.dataElementType();
-		}
-
 		if (me.datasetElementType) {
 			meta.dataset = new me.datasetElementType();
 		}
 	}
 
-	buildOrUpdateElements() {
+	buildOrUpdateElements(resetNewElements) {
 		const me = this;
 		const meta = me._cachedMeta;
 		const dataset = me.getDataset();
@@ -382,7 +375,7 @@ export default class DatasetController {
 
 		// Re-sync meta data in case the user replaced the data array or if we missed
 		// any updates and so make sure that we handle number of datapoints changing.
-		me._resyncElements();
+		me._resyncElements(resetNewElements);
 
 		// if stack changed, update stack values for the whole dataset
 		if (stackChanged) {
@@ -402,7 +395,10 @@ export default class DatasetController {
 			me.getDataset(),
 		], {
 			merger(key, target, source) {
-				if (key !== 'data') {
+				// Cloning the data is expensive and unnecessary.
+				// Additionally, plugins may add dataset level fields that should
+				// not be cloned. We identify those via an underscore prefix
+				if (key !== 'data' && key.charAt(0) !== '_') {
 					_merger(key, target, source);
 				}
 			}
@@ -419,13 +415,10 @@ export default class DatasetController {
 		const {_cachedMeta: meta, _data: data} = me;
 		const {iScale, _stacked} = meta;
 		const iAxis = iScale.axis;
-		let sorted = true;
-		let i, parsed, cur, prev;
 
-		if (start > 0) {
-			sorted = meta._sorted;
-			prev = meta._parsed[start - 1];
-		}
+		let sorted = start === 0 && count === data.length ? true : meta._sorted;
+		let prev = start > 0 && meta._parsed[start - 1];
+		let i, cur, parsed;
 
 		if (me._parsing === false) {
 			meta._parsed = data;
@@ -971,13 +964,13 @@ export default class DatasetController {
 	/**
 	 * @private
 	 */
-	_resyncElements() {
+	_resyncElements(resetNewElements) {
 		const me = this;
 		const numMeta = me._cachedMeta.data.length;
 		const numData = me._data.length;
 
 		if (numData > numMeta) {
-			me._insertElements(numMeta, numData - numMeta);
+			me._insertElements(numMeta, numData - numMeta, resetNewElements);
 		} else if (numData < numMeta) {
 			me._removeElements(numData, numMeta - numData);
 		}
@@ -988,7 +981,7 @@ export default class DatasetController {
 	/**
 	 * @private
 	 */
-	_insertElements(start, count) {
+	_insertElements(start, count, resetNewElements = true) {
 		const me = this;
 		const elements = new Array(count);
 		const meta = me._cachedMeta;
@@ -1005,7 +998,9 @@ export default class DatasetController {
 		}
 		me.parse(start, count);
 
-		me.updateElements(data, start, count, 'reset');
+		if (resetNewElements) {
+			me.updateElements(data, start, count, 'reset');
+		}
 	}
 
 	updateElements(element, start, count, mode) {} // eslint-disable-line no-unused-vars

src/plugins/index.js

@@ -1,3 +1,4 @@
+export {default as Decimation} from './plugin.decimation';
 export {default as Filler} from './plugin.filler';
 export {default as Legend} from './plugin.legend';
 export {default as Title} from './plugin.title';

src/plugins/plugin.decimation.js

@@ -0,0 +1,135 @@
+import {isNullOrUndef, resolve} from '../helpers';
+
+function minMaxDecimation(data, availableWidth) {
+	let i, point, x, y, prevX, minIndex, maxIndex, minY, maxY;
+	const decimated = [];
+
+	const xMin = data[0].x;
+	const xMax = data[data.length - 1].x;
+	const dx = xMax - xMin;
+
+	for (i = 0; i < data.length; ++i) {
+		point = data[i];
+		x = (point.x - xMin) / dx * availableWidth;
+		y = point.y;
+		const truncX = x | 0;
+
+		if (truncX === prevX) {
+			// Determine `minY` / `maxY` and `avgX` while we stay within same x-position
+			if (y < minY) {
+				minY = y;
+				minIndex = i;
+			} else if (y > maxY) {
+				maxY = y;
+				maxIndex = i;
+			}
+		} else {
+			// Push up to 4 points, 3 for the last interval and the first point for this interval
+			if (minIndex && maxIndex) {
+				decimated.push(data[minIndex], data[maxIndex]);
+			}
+			if (i > 0) {
+				// Last point in the previous interval
+				decimated.push(data[i - 1]);
+			}
+			decimated.push(point);
+			prevX = truncX;
+			minY = maxY = y;
+			minIndex = maxIndex = i;
+		}
+	}
+
+	return decimated;
+}
+
+export default {
+	id: 'decimation',
+
+	defaults: {
+		algorithm: 'min-max',
+		enabled: false,
+	},
+
+	beforeElementsUpdate: (chart, args, options) => {
+		if (!options.enabled) {
+			return;
+		}
+
+		// Assume the entire chart is available to show a few more points than needed
+		const availableWidth = chart.width;
+
+		chart.data.datasets.forEach((dataset, datasetIndex) => {
+			const {_data, indexAxis} = dataset;
+			const meta = chart.getDatasetMeta(datasetIndex);
+			const data = _data || dataset.data;
+
+			if (resolve([indexAxis, chart.options.indexAxis]) === 'y') {
+				// Decimation is only supported for lines that have an X indexAxis
+				return;
+			}
+
+			if (meta.type !== 'line') {
+				// Only line datasets are supported
+				return;
+			}
+
+			const xAxis = chart.scales[meta.xAxisID];
+			if (xAxis.type !== 'linear' && xAxis.type !== 'time') {
+				// Only linear interpolation is supported
+				return;
+			}
+
+			if (chart.options.parsing) {
+				// Plugin only supports data that does not need parsing
+				return;
+			}
+
+			if (data.length <= 4 * availableWidth) {
+				// No decimation is required until we are above this threshold
+				return;
+			}
+
+			if (isNullOrUndef(_data)) {
+				// First time we are seeing this dataset
+				// We override the 'data' property with a setter that stores the
+				// raw data in _data, but reads the decimated data from _decimated
+				// TODO: Undo this on chart destruction
+				dataset._data = data;
+				delete dataset.data;
+				Object.defineProperty(dataset, 'data', {
+					configurable: true,
+					enumerable: true,
+					get: function() {
+						return this._decimated;
+					},
+					set: function(d) {
+						this._data = d;
+					}
+				});
+			}
+
+			// Point the chart to the decimated data
+			let decimated;
+			switch (options.algorithm) {
+			case 'min-max':
+				decimated = minMaxDecimation(data, availableWidth);
+				break;
+			default:
+				throw new Error(`Unsupported decimation algorithm '${options.algorithm}'`);
+			}
+
+			dataset._decimated = decimated;
+		});
+	},
+
+	destroy(chart) {
+		chart.data.datasets.forEach((dataset) => {
+			if (dataset._decimated) {
+				const data = dataset._data;
+				delete dataset._decimated;
+				delete dataset._data;
+				Object.defineProperty(dataset, 'data', {value: data});
+			}
+		});
+	}
+};

types/index.esm.d.ts

@@ -539,7 +539,7 @@ export class DatasetController<TElement extends Element = Element, TDatasetEleme
 	configure(): void;
 	initialize(): void;
 	addElements(): void;
-	buildOrUpdateElements(): void;
+	buildOrUpdateElements(resetNewElements?: boolean): void;
 
 	getStyle(index: number, active: boolean): any;
 	protected resolveDatasetElementOptions(active: boolean): any;
@@ -789,6 +789,14 @@ export interface Plugin<O = {}> extends ExtendedPlugin {
 	 * @param {object} options - The plugin options.
 	 */
 	afterUpdate?(chart: Chart, args: { mode: UpdateMode }, options: O): void;
+	/**
+	 * @desc Called during the update process, before any chart elements have been created.
+	 * This can be used for data decimation by changing the data array inside a dataset.
+	 * @param {Chart} chart - The chart instance.
+	 * @param {object} args - The call arguments.
+	 * @param {object} options - The plugin options.
+	 */
+	beforeElementsUpdate?(chart: Chart, args: {}, options: O): void;
 	/**
 	 * @desc Called during chart reset
 	 * @param {Chart} chart - The chart instance.
@@ -1902,8 +1910,16 @@ export class BasePlatform {
 export class BasicPlatform extends BasePlatform {}
 export class DomPlatform extends BasePlatform {}
 
-export const Filler: Plugin;
+export declare enum DecimationAlgorithm {
+	minmax = 'min-max',
+}
 
+export interface DecimationOptions {
+	enabled: boolean;
+	algorithm: DecimationAlgorithm;
+}
+
+export const Filler: Plugin;
 export interface FillerOptions {
 	propagate: boolean;
 }
@@ -2477,6 +2493,7 @@ export interface TooltipItem {
 }
 
 export interface PluginOptionsByType {
+	decimation: DecimationOptions;
 	filler: FillerOptions;
 	legend: LegendOptions;
 	title: TitleOptions;