better docs and string file

README.md

@@ -130,13 +130,13 @@ Running `gulp watch` will build the project and start all the automated watchers
 
 **Recent improvements:**
 
+- Configuration docs: https://github.com/implydata/pivot/blob/master/docs/configuration.md
+- Support for numeric dimensions
 - Configurable attribute overrides
 - Support for `absolute`, `power`, `sqrt`, and `fallback` actions
 - Relative filters
 - All new introspection code
 - Better comparison behavior and legend interaction
-- Support for query time lookups (ex. `$language.lookup('wikipedia-language-lookup')`)
-- Support for the extract function (ex. `resourceVersion: $resource.extract("(\d+\.\d+\.\d+)")`)
 
 **We will be working on:**
 

docs/configuration.md

@@ -0,0 +1,208 @@
+# Configuring Pivot
+
+It is easy to start using Pivot by pointing it at your Druid cluster like so: `pivot --druid druid.broker.host:8082`
+but to make the most of Pivot you will want to configure it.
+
+Pivot can be configured with a *config* YAML file. While you could write one from scratch it is recommended to let
+Pivot give you a head start by using it to generate a config file for you using the default introspection.
+
+Run:
+
+```bash
+pivot --druid druid.broker.host:8082 --print-config --with-comments > config.yaml
+```
+
+This will cause Pivot to go through its normal startup and introspection routine and then dump the internally generated
+config (complete with comments) into the provided file.
+
+You can now run `pivot --config config.yaml` to run Pivot with your config.
+
+The next step is to open the generated config file in your favourite text editor and configure Pivot to your liking.
+Below we will go through a typical configuration flow. At any point you can save the config and re-launch Pivot to load
+that config in.
+
+
+## Configuring the DataSources
+
+In your Pivot config you will see a top level `dataSources:` key that holds the data sources that will be loaded into
+Pivot. The order of the data sources in the config will define the ordering seen in the UI.
+
+
+### Basic data source properties
+
+Described here are only the properties which you might want to change.
+
+**name** (string)
+
+The name of the data source as used internally in Pivot and used in the URLs. This should be a URL safe string.
+Changing this property for a given data source will break any URLs that someone might have generated for that data
+source in the past.
+
+**title**: (string)
+
+The user visible name that will be used to describe this data source in the UI. It is always safe to change this.
+
+**defaultSortMeasure**: (string)
+
+The name of the measure that will be used for default sorting. It is commonly set to the measure that represents the
+count of events.
+
+**defaultPinnedDimensions** (string[])
+
+The names of the dimensions (in order) that will appear *pinned* by default on the right panel.
+
+
+### Attribute Overrides
+
+While Pivot tries to learn as much as it can from your data source from Druid directly.
+It can not (yet) do a perfect job. The `attributeOverrides:` section of the data source is there for you to fix that.
+
+**name** (string)
+
+The name of the attribute (column) in Druid. This must match the Druid name.
+
+Here are some common scenarios where you should add an attribute override:
+
+#### You have a HyperLogLog metric column but Pivot is not detecting it
+
+If you have a HyperLogLog metric (say: `unique_things`) it is possible that Druid introspection (Druid <= 0.8.3) will
+not describe it correctly.
+In that case it will be assumed to be a regular numeric column and will ether return incorrect results (or error out)
+when included in queries.
+
+You should add:
+
+```yaml
+         - name: unique_things
+           special: unique
+```
+
+To the `attributeOverrides` to tell Pivot that this is indeed a special (hyperUnique) column.
+
+You should also ensure that wherever it is used in the measures it is aggregated with `countDistinct($unique_things)`.
+
+#### You have a numeric dimension
+
+Pivot can not corretly detect numeric dimensions as Druid reports all dimensions to be strings.
+When a numeric dimension is incorrectly classified as a string its soring will appear wrong in the UI.
+If you have a dimension with numeric values (say: `age`).
+
+You should add:
+
+```yaml
+         - name: age
+           type: NUMBER
+```
+
+To the `attributeOverrides` to tell Pivot that this is numeric.
+
+You can now use `$age` in numeric expressions. For example you could create a dimension with the expression 
+`$age / 2 + 7`.
+
+
+### Dimensions
+
+In this section you can define the dimensions that users can *split* and *filter* on in the UI.
+The order of the dimension list in the top of the left panel is determined by the order of the dimensions definitions
+in this section.
+
+**name** (string)
+
+The name of the dimension.
+This does not have to correspond to the attribute name (but the auto generated dimensions do).
+This should be a URL safe string.
+Changing this property will break any URLs that someone might have generated that include this dimension.
+
+**title** (string)
+
+The title for this dimension in the UI. Can be anything and is safe to change at any time.
+
+**expression** (plywood expression)
+
+The expression for this dimension. By default it is `$name` where *name* is the name of the dimension.
+
+You can create derived dimensions by using non-trivial expressions. Here are some common use cases for derived dimensions:
+
+
+#### Lookups
+
+If you have a dimension that represents an ID that is a key into some other table. You may have set up a
+[Druid Query Time Lookup](http://druid.io/docs/latest/querying/lookups.html) in which case you could 
+
+```yaml
+      - name: correctValue
+        expression: $lookupKey.lookup('my_awesome_lookup')
+```
+
+Which would apply the lookup.
+
+You can also apply the `.fallback()` action as ether:
+
+- `$lookupKey.lookup('my_awesome_lookup').fallback($lookupKey)` to keep values that were not found as they are.
+- `$lookupKey.lookup('my_awesome_lookup').fallback('missing')` to map missing values to the word 'missing'.
+
+#### Extraction
+
+Imagine you have an attribute like `resourceName` which has values like:
+
+```json
+['druid-0.8.2', 'druid-0.8.1', 'druid-0.7.0', 'index.html']
+```
+
+You could apply, for example, the `.extract` function by creating a dimension like so:
+
+```yaml
+      - name: resourceVersion
+        expression: $resourceName.extract('(\d+\.\d+\.\d+)')
+```
+
+Which would have values like: 
+
+```json
+['0.8.2', '0.8.1', '0.7.0', null]
+```
+
+
+### Measures
+
+In this section you can define the measures that users can *aggregate* on (*apply*) on in the UI.
+The order of the measure list in the bottom of the left panel is determined by the order of the measure definitions
+in this section.
+
+**name** (string)
+
+The name of the measure.
+This should be a URL safe string.
+Changing this property will break any URLs that someone might have generated that include this measure.
+
+**title** (string)
+
+The title for this measure in the UI. Can be anything and is safe to change at any time.
+
+**expression** (plywood expression)
+
+The expression for this dimension. By default it is `$main.sum($name)` where *name* is the name of the measure.
+
+The `$main` part of the measure expressions serves as a place holder for the table name.
+In Plywood every aggregate is a function that acts on the segment group.
+
+You can create derived measures by using non-trivial expressions. Here are some common use cases for derived dimensions:
+
+
+#### Dividing to compute ratios
+
+```yaml
+      - name: ecpm
+        title: eCPM
+        expression: $main.sum($revenue) / $main.sum($impressions) * 1000
+```
+
+
+#### Filtering aggregations
+
+```yaml
+      - name: usa_revenue
+        title: USA Revenue
+        expression: $main.filter($country == 'United States').sum($revenue)
+```
+

package.json

@@ -29,7 +29,7 @@
     "body-parser": "1.15.0",
     "chronoshift": "0.3.7",
     "compression": "1.6.1",
-    "d3": "3.5.14",
+    "d3": "3.5.15",
     "debug": "2.2.0",
     "express": "4.13.4",
     "express-handlebars": "3.0.0",
@@ -41,7 +41,7 @@
     "morgan": "1.6.1",
     "nopt": "3.0.6",
     "numeral": "1.5.3",
-    "plywood": "0.8.9",
+    "plywood": "0.8.10",
     "plywood-druid-requester": "1.2.7",
     "plywood-mysql-requester": "1.1.5",
     "q": "1.4.1",

src/client/components/dimension-list-tile/dimension-list-tile.tsx

@@ -6,7 +6,7 @@ import * as ReactDOM from 'react-dom';
 import { SvgIcon } from '../svg-icon/svg-icon';
 import { List } from 'immutable';
 import { $, Expression, Executor, Dataset } from 'plywood';
-import { TITLE_HEIGHT, DIMENSION_HEIGHT } from '../../config/constants';
+import { STRINGS, TITLE_HEIGHT, DIMENSION_HEIGHT } from '../../config/constants';
 import { moveInList } from '../../../common/utils/general/general';
 import { DragManager } from '../../utils/drag-manager/drag-manager';
 import { findParentWithClass, setDragGhost, transformStyle } from '../../utils/dom/dom';
@@ -245,7 +245,7 @@ export class DimensionListTile extends React.Component<DimensionListTileProps, D
       onDrop={this.drop.bind(this)}
       style={style}
     >
-      <div className="title">Dimensions</div>
+      <div className="title">{STRINGS.dimensions}</div>
       <div className="items" ref="items">
         {dimensionItems}
       </div>

src/client/components/filter-tile/filter-tile.tsx

@@ -6,7 +6,7 @@ import * as ReactDOM from 'react-dom';
 import * as Q from 'q';
 import { Timezone, Duration, hour, day, week } from 'chronoshift';
 import { $, Expression, InAction, Executor, Dataset } from 'plywood';
-import { BAR_TITLE_WIDTH, CORE_ITEM_WIDTH, CORE_ITEM_GAP } from '../../config/constants';
+import { STRINGS, BAR_TITLE_WIDTH, CORE_ITEM_WIDTH, CORE_ITEM_GAP } from '../../config/constants';
 import { Stage, Clicker, Essence, DataSource, Filter, FilterClause, Dimension, Measure} from '../../../common/models/index';
 import { calculateDragPosition, DragPosition } from '../../../common/utils/general/general';
 import { formatTimeRange, DisplayYear } from '../../utils/date/date';
@@ -619,7 +619,7 @@ export class FilterTile extends React.Component<FilterTileProps, FilterTileState
       onDragLeave={this.dragLeave.bind(this)}
       onDrop={this.drop.bind(this)}
     >
-      <div className="title">Filter</div>
+      <div className="title">{STRINGS.filter}</div>
       <div className="items" ref="items">
         {filterItems}
       </div>

src/client/components/measures-tile/measures-tile.tsx

@@ -5,7 +5,7 @@ import { List, OrderedSet } from 'immutable';
 import * as React from 'react';
 import * as ReactDOM from 'react-dom';
 import { $, Expression, Executor, Dataset } from 'plywood';
-import { PIN_TITLE_HEIGHT, SEARCH_BOX_HEIGHT, MEASURE_HEIGHT, PIN_PADDING_BOTTOM } from '../../config/constants';
+import { STRINGS, PIN_TITLE_HEIGHT, SEARCH_BOX_HEIGHT, MEASURE_HEIGHT, PIN_PADDING_BOTTOM } from '../../config/constants';
 import { hasOwnProperty } from '../../../common/utils/general/general';
 import { Clicker, Essence, DataSource, Filter, Dimension, Measure } from '../../../common/models/index';
 import { TileHeader } from '../tile-header/tile-header';
@@ -72,7 +72,7 @@ export class MeasuresTile extends React.Component<MeasuresTileProps, MeasuresTil
     };
 
     return <div className="measures-tile" style={style}>
-      <div className="title">Measures</div>
+      <div className="title">{STRINGS.measures}</div>
       <div className="rows">{rows}</div>
     </div>;
   }

src/client/components/pinboard-panel/pinboard-panel.tsx

@@ -5,6 +5,7 @@ import * as React from 'react';
 import * as ReactDOM from 'react-dom';
 import { List, OrderedSet } from 'immutable';
 import { $, Expression, Executor, Dataset, RefExpression, SortAction } from 'plywood';
+import { STRINGS } from '../../config/constants';
 import { SvgIcon } from '../svg-icon/svg-icon';
 import { Clicker, Essence, DataSource, Filter, Dimension, Measure, SortOn, VisStrategy, Colors } from '../../../common/models/index';
 import { DragManager } from '../../utils/drag-manager/drag-manager';
@@ -197,7 +198,7 @@ export class PinboardPanel extends React.Component<PinboardPanelProps, PinboardP
     if (!dragOver && !dimensionTiles.length) {
       placeholder = <div className="placeholder">
         <SvgIcon svg={require('../../icons/preview-pin.svg')}/>
-        <div className="placeholder-message">Click or drag dimensions to pin them</div>
+        <div className="placeholder-message">{STRINGS.pinboardPlaceholder}</div>
       </div>;
     }
 
@@ -212,7 +213,7 @@ export class PinboardPanel extends React.Component<PinboardPanelProps, PinboardP
       {legendDimensionTile}
       <PinboardMeasureTile
         essence={essence}
-        title="Pinboard"
+        title={STRINGS.pinboard}
         sortOn={pinnedSortSortOn}
         onSelect={this.onPinboardSortOnSelect.bind(this)}
       />

src/client/components/preview-menu/preview-menu.tsx

@@ -6,6 +6,7 @@ import * as React from 'react';
 import * as ReactDOM from 'react-dom';
 import { SvgIcon } from '../svg-icon/svg-icon';
 import { $, Expression, Executor, Dataset } from 'plywood';
+import { STRINGS } from '../../config/constants';
 import { Stage, Clicker, Essence, VisStrategy, DataSource, Filter, Dimension, Measure, SplitCombine } from '../../../common/models/index';
 import { BubbleMenu } from '../bubble-menu/bubble-menu';
 //import { MenuHeader } from '../menu-header/menu-header';
@@ -85,19 +86,19 @@ export class PreviewMenu extends React.Component<PreviewMenuProps, PreviewMenuSt
     >
       <div className="filter action" onClick={this.onFilter.bind(this)}>
         <SvgIcon svg={require('../../icons/preview-filter.svg')}/>
-        <div className="action-label">Filter</div>
+        <div className="action-label">{STRINGS.filter}</div>
       </div>
       <div className="pin action" onClick={this.onPin.bind(this)}>
         <SvgIcon svg={require('../../icons/preview-pin.svg')}/>
-        <div className="action-label">Pin</div>
+        <div className="action-label">{STRINGS.pin}</div>
       </div>
       <div className="split action" onClick={this.onSplit.bind(this)}>
         <SvgIcon svg={require('../../icons/preview-split.svg')}/>
-        <div className="action-label">Split</div>
+        <div className="action-label">{STRINGS.split}</div>
       </div>
       <div className="subsplit action" onClick={this.onSubsplit.bind(this)}>
         <SvgIcon svg={require('../../icons/preview-subsplit.svg')}/>
-        <div className="action-label">Subsplit</div>
+        <div className="action-label">{STRINGS.subsplit}</div>
       </div>
     </BubbleMenu>;
   }

src/client/components/query-error/query-error.tsx

@@ -4,7 +4,7 @@ require('./query-error.css');
 import { List } from 'immutable';
 import * as React from 'react';
 import * as ReactDOM from 'react-dom';
-// import { ... } from '../../config/constants';
+import { STRINGS } from '../../config/constants';
 
 export interface QueryErrorProps extends React.Props<any> {
   error: any;
@@ -26,7 +26,7 @@ export class QueryError extends React.Component<QueryErrorProps, QueryErrorState
     return <div className="query-error">
       <div className="whiteout"></div>
       <div className="container">
-        <div className="error">Query Error</div>
+        <div className="error">{STRINGS.queryError}</div>
         <div className="message">{error.message}</div>
       </div>
     </div>;