Merge remote-tracking branch 'upstream/master' into i18n

CONTRIBUTING.md

@@ -85,7 +85,7 @@ Before you start changing the docs, you'll want to
 [fork the Superset project on Github](https://help.github.com/articles/fork-a-repo/).
 Once that new repository has been created, clone it on your local machine:
 
-    git clone git@github.com:your_username/superset.git
+    git clone git@github.com:your_username/incubator-superset.git
 
 At this point, you may also want to create a
 [Python virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
@@ -97,7 +97,7 @@ to manage the Python packages you're about to install:
 Finally, to make changes to the rst files and build the docs using Sphinx, 
 you'll need to install a handful of dependencies from the repo you cloned:
 
-    cd superset
+    cd incubator-superset
     pip install -r dev-reqs-for-docs.txt
 
 To get the feel for how to edit and build the docs, let's edit a file, build

README.md

@@ -182,6 +182,7 @@ the world know they are using Superset. Join our growing community!
  - [Konfío](http://konfio.mx)
  - [Maieutical Labs](https://cloudschooling.it)
  - [Qunar](https://www.qunar.com/)
+ - [Shopee](https://shopee.sg)
  - [Shopkick](https://www.shopkick.com)
  - [Tails.com](https://tails.com)
  - [Tobii](http://www.tobii.com/)

dev-reqs.txt

@@ -7,7 +7,6 @@ mysqlclient
 nose
 psycopg2
 pylint
-pythrifthiveapi
 pyyaml
 redis
 statsd

docs/installation.rst

@@ -8,6 +8,32 @@ Superset is tested against Python ``2.7`` and Python ``3.4``.
 Airbnb currently uses 2.7.* in production. We do not plan on supporting
 Python ``2.6``.
 
+Cloud-native!
+-------------
+
+Superset is designed to be highly available. It is
+"cloud-native" as it has been designed scale out in large,
+distributed environments, and works well inside containers.
+While you can easily
+test drive Superset on a modest setup or simply on your laptop,
+there's virtually no limit around scaling out the platform.
+Superset is also cloud-native in the sense that it is
+flexible and lets you choose your web server (Gunicorn, Nginx, Apache),
+your metadata database engine (MySQL, Postgres, MariaDB, ...),
+your message queue (Redis, RabbitMQ, SQS, ...),
+your results backend (S3, Redis, Memcached, ...), your caching layer
+(memcached, Redis, ...), works well with services like NewRelic, StatsD and
+DataDog, and has the ability to run analytic workloads against
+most popular database technologies.
+
+Superset is battle tested in large environments with hundreds
+of concurrent users. Airbnb's production environment runs inside
+Kubernetes and serves 600+ daily active users viewing over 100K charts a
+day.
+
+The Superset web server and the Superset Celery workers (optional)
+are stateless, so you can scale out by running on as many servers
+as needed.
 
 OS dependencies
 ---------------
@@ -107,10 +133,40 @@ the credential you entered while creating the admin account, and navigate to
 your datasources for Superset to be aware of, and they should show up in
 `Menu -> Datasources`, from where you can start playing with your data!
 
-Please note that *gunicorn*, Superset default application server, does not
-work on Windows so you need to use the development web server.
-The development web server though is not intended to be used on production systems
-so better use a supported platform that can run *gunicorn*.
+A proper WSGI HTTP Server
+-------------------------
+
+While you can setup Superset to run on Nginx or Apache, many use
+Gunicorn, preferably in **async mode**, which allows for impressive
+concurrency even and is fairly easy to install and configure. Please
+refer to the
+documentation of your preferred technology to set up this Flask WSGI
+application in a way that works well in your environment.
+
+While the `superset runserver` command act as an quick wrapper
+around `gunicorn`, it doesn't expose all the options you may need,
+so you'll want to craft your own `gunicorn` command in your production
+environment. Here's an **async** setup known to work well: ::
+
+	gunicorn \
+		-w 10 \
+		-k gevent \
+		--timeout 120 \
+		-b  0.0.0.0:6666 \
+		--limit-request-line 0 \
+		--limit-request-field_size 0 \
+		--statsd-host localhost:8125 \
+		superset:app
+
+Refer to the
+[Gunicorn documentation](http://docs.gunicorn.org/en/stable/design.html)
+for more information.
+
+Note that *gunicorn* does not
+work on Windows so the `superser runserver` command is not expected to work
+in that context. Also note that the development web
+server (`superset runserver -d`) is not intended for production use.
+
 
 Configuration behind a load balancer
 ------------------------------------
@@ -157,6 +213,8 @@ of the parameters you can copy / paste in that configuration module: ::
 
     # Flask-WTF flag for CSRF
     WTF_CSRF_ENABLED = True
+    # Add endpoints that need to be exempt from CSRF protection
+    WTF_CSRF_EXEMPT_LIST = []
 
     # Set this API key to enable Mapbox visualizations
     MAPBOX_API_KEY = ''
@@ -172,6 +230,11 @@ Please make sure to change:
 * *SQLALCHEMY_DATABASE_URI*, by default it is stored at *~/.superset/superset.db*
 * *SECRET_KEY*, to a long random string
 
+In case you need to exempt endpoints from CSRF, e.g. you are running a custom
+auth postback endpoint, you can add them to *WTF_CSRF_EXEMPT_LIST*
+
+     WTF_CSRF_EXEMPT_LIST = ['']
+
 Database dependencies
 ---------------------
 
@@ -223,10 +286,6 @@ database you want to connect to should get you to the right place.
 (AWS) Athena
 ------------
 
-This currently relies on an unreleased future version of `PyAthenaJDBC <https://github.com/laughingman7743/PyAthenaJDBC>`_. If you're adventurous or simply impatient, you can install directly from git: ::
-
-    pip install git+https://github.com/laughingman7743/PyAthenaJDBC@support_sqlalchemy
-
 The connection string for Athena looks like this ::
 
     awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...
@@ -284,6 +343,24 @@ on top of the **database**. For Superset to connect to a specific schema,
 there's a **schema** parameter you can set in the table form.
 
 
+External Password store for SQLAlchemy connections
+--------------------------------------------------
+It is possible to use an external store for you database passwords. This is
+useful if you a running a custom secret distribution framework and do not wish
+to store secrets in Superset's meta database.
+
+Example:
+Write a function that takes a single argument of type ``sqla.engine.url`` and returns
+the password for the given connection string. Then set ``SQLALCHEMY_CUSTOM_PASSWORD_STORE``
+in your config file to point to that function. ::
+
+    def example_lookup_password(url):
+        secret = <<get password from external framework>>
+        return 'secret'
+
+    SQLALCHEMY_CUSTOM_PASSWORD_STORE = example_lookup_password
+
+
 SSL Access to databases
 -----------------------
 This example worked with a MySQL database that requires SSL. The configuration

docs/security.rst

@@ -36,7 +36,7 @@ own. Alpha users can add and alter data sources.
 Gamma
 """""
 Gamma have limited access. They can only consume data coming from data sources
-they have been giving access to through another complementary role.
+they have been given access to through another complementary role.
 They only have access to view the slices and
 dashboards made from data sources that they have access to. Currently Gamma
 users are not able to alter or add data sources. We assume that they are

superset/__init__.py

@@ -20,14 +20,12 @@
 from superset.connectors.connector_registry import ConnectorRegistry
 from superset import utils, config  # noqa
 
-
 APP_DIR = os.path.dirname(__file__)
 CONFIG_MODULE = os.environ.get('SUPERSET_CONFIG', 'superset.config')
 
 with open(APP_DIR + '/static/assets/backendSync.json', 'r') as f:
     frontend_config = json.load(f)
 
-
 app = Flask(__name__)
 app.config.from_object(CONFIG_MODULE)
 conf = app.config
@@ -53,14 +51,16 @@ def get_manifest_file(filename):
         parse_manifest_json()
     return '/static/assets/dist/' + manifest.get(filename, '')
 
+
 parse_manifest_json()
 
+
 @app.context_processor
 def get_js_manifest():
     return dict(js_manifest=get_manifest_file)
 
-#################################################################
 
+#################################################################
 
 for bp in conf.get('BLUEPRINTS'):
     try:
@@ -83,6 +83,9 @@ def get_js_manifest():
 
 if conf.get('WTF_CSRF_ENABLED'):
     csrf = CSRFProtect(app)
+    csrf_exempt_list = conf.get('WTF_CSRF_EXEMPT_LIST', [])
+    for ex in csrf_exempt_list:
+        csrf.exempt(ex)
 
 utils.pessimistic_connection_handling(db.engine)
 
@@ -97,10 +100,11 @@ def get_js_manifest():
 
 if app.config.get('ENABLE_TIME_ROTATE'):
     logging.getLogger().setLevel(app.config.get('TIME_ROTATE_LOG_LEVEL'))
-    handler = TimedRotatingFileHandler(app.config.get('FILENAME'),
-                                       when=app.config.get('ROLLOVER'),
-                                       interval=app.config.get('INTERVAL'),
-                                       backupCount=app.config.get('BACKUP_COUNT'))
+    handler = TimedRotatingFileHandler(
+        app.config.get('FILENAME'),
+        when=app.config.get('ROLLOVER'),
+        interval=app.config.get('INTERVAL'),
+        backupCount=app.config.get('BACKUP_COUNT'))
     logging.getLogger().addHandler(handler)
 
 if app.config.get('ENABLE_CORS'):
@@ -111,17 +115,18 @@ def get_js_manifest():
     app.wsgi_app = ProxyFix(app.wsgi_app)
 
 if app.config.get('ENABLE_CHUNK_ENCODING'):
-    class ChunkedEncodingFix(object):
 
+    class ChunkedEncodingFix(object):
         def __init__(self, app):
             self.app = app
 
         def __call__(self, environ, start_response):
             # Setting wsgi.input_terminated tells werkzeug.wsgi to ignore
             # content-length and read the stream till the end.
-            if 'chunked' == environ.get('HTTP_TRANSFER_ENCODING', '').lower():
+            if environ.get('HTTP_TRANSFER_ENCODING', '').lower() == u'chunked':
                 environ['wsgi.input_terminated'] = True
             return self.app(environ, start_response)
+
     app.wsgi_app = ChunkedEncodingFix(app.wsgi_app)
 
 if app.config.get('UPLOAD_FOLDER'):
@@ -139,8 +144,10 @@ class MyIndexView(IndexView):
     def index(self):
         return redirect('/superset/welcome')
 
+
 appbuilder = AppBuilder(
-    app, db.session,
+    app,
+    db.session,
     base_template='superset/base.html',
     indexview=MyIndexView,
     security_manager_class=app.config.get("CUSTOM_SECURITY_MANAGER"))

superset/assets/javascripts/SqlLab/components/SqlEditor.jsx

@@ -77,6 +77,9 @@ class SqlEditor extends React.PureComponent {
     this.props.actions.queryEditorSetSql(this.props.queryEditor, sql);
   }
   runQuery(runAsync = false) {
+    if (!this.props.queryEditor.sql) {
+      return;
+    }
     let effectiveRunAsync = runAsync;
     if (!this.props.database.allow_run_sync) {
       effectiveRunAsync = true;

superset/assets/javascripts/dashboard/components/SliceCell.jsx

@@ -20,7 +20,7 @@ function SliceCell({ expandedSlices, removeSlice, slice }) {
             <span>{slice.slice_name}</span>
           </div>
           <div className="col-md-12 chart-controls">
-            <div className="pull-right">
+            <div id={'controls_' + slice.slice_id} className="pull-right">
               <a title={t('Move chart')} data-toggle="tooltip">
                 <i className="fa fa-arrows drag" />
               </a>
@@ -43,10 +43,20 @@ function SliceCell({ expandedSlices, removeSlice, slice }) {
               >
                 <i className="fa fa-pencil" />
               </a>
-              <a href={getExploreUrl(slice.form_data, 'csv')} title={t('Export CSV')} data-toggle="tooltip">
+              <a
+                className="exportCSV"
+                href={getExploreUrl(slice.form_data, 'csv')}
+                title={t('Export CSV')}
+                data-toggle="tooltip"
+              >
                 <i className="fa fa-table" />
               </a>
-              <a href={getExploreUrl(slice.form_data)} title={t('Explore chart')} data-toggle="tooltip">
+              <a
+                className="exploreChart"
+                href={getExploreUrl(slice.form_data)}
+                title={t('Explore chart')}
+                data-toggle="tooltip"
+              >
                 <i className="fa fa-share" />
               </a>
               <a

superset/assets/javascripts/explore/components/ControlPanelsContainer.jsx

@@ -65,10 +65,11 @@ class ControlPanelsContainer extends React.Component {
             </Alert>
           }
           {this.sectionsToRender().map((section) => {
-            const hasErrors = section.controlSetRows.some(rows => rows.some((s) => {
-              const errors = ctrls[s].validationErrors;
-              return errors && (errors.length > 0);
-            }));
+            const hasErrors = section.controlSetRows.some(rows => rows.some(s => (
+                ctrls[s] &&
+                ctrls[s].validationErrors &&
+                (ctrls[s].validationErrors.length > 0)
+            )));
             return (
               <ControlPanelSection
                 key={section.label}