Ensure proper JSON serialization of numpy.ndarray

[robdiciuccio]

CATEGORY

Choose one

• Bug Fix
• Enhancement (new features, refinement)
• Refactor
• Add tests
• Build / Development Environment
• Documentation

SUMMARY

When querying a Presto datasource with a column of type ARRAY asynchronously, the following error would be raised:

TypeError: Unserializable object [] of type <class 'numpy.ndarray'>

This fix ensures proper conversion of the numpy.ndarray to a list for JSON serialization.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Ensure queries against a Presto table containing nested data type columns work correctly. Repeat for other supported DBs having nested data type columns.

This fix was tested against Presto 0.227 in both async and sync queries.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

REVIEWERS

@graceguo-supercat @craig-rueda @dpgaspar @villebro

[craig-rueda]

LGTM

[villebro]

LGTM 👍

[codecov-io]

Codecov Report

Merging #8946 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #8946   +/-   ##
=======================================
  Coverage   58.97%   58.97%           
=======================================
  Files         359      359           
  Lines       11333    11333           
  Branches     2787     2787           
=======================================
  Hits         6684     6684           
  Misses       4471     4471           
  Partials      178      178

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3bedee7...8926e22. Read the comment docs.

[graceguo-supercat]

Hi @robdiciuccio Thanks very much for the fix. I run a few queries in different tables. Here is another error message also related to pyarrow:

  File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/dist-packages/newrelic/hooks/framework_flask.py", line 108, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/dist-packages/newrelic/hooks/framework_flask.py", line 45, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/flask_appbuilder/security/decorators.py", line 168, in wraps
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/superset/utils/log.py", line 59, in wrapper
    value = f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 2291, in results
    return self.results_exec(key)
  File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 2331, in results_exec
    payload, query, cast(bool, results_backend_use_msgpack)
  File "/usr/local/lib/python3.6/dist-packages/superset/views/core.py", line 233, in _deserialize_results_payload
    df = result_set.SupersetResultSet.convert_table_to_df(pa_table)
  File "/usr/local/lib/python3.6/dist-packages/superset/result_set.py", line 129, in convert_table_to_df
    return table.to_pandas(integer_object_nulls=True)
  File "pyarrow/array.pxi", line 468, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1238, in pyarrow.lib.Table._to_pandas
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 704, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories)
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 976, in _table_to_blocks
    result = pa.lib.table_to_blocks(options, block_table, categories)
  File "pyarrow/table.pxi", line 780, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 86, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for list in DataFrameBlock: struct<author_type: string, locale: string>

[robdiciuccio]

Thanks @graceguo-supercat. Is this in Presto? What are the data types of the columns being queried?

[graceguo-supercat]

Presto version=0.188.
I feel the problem might be caused by MAP<STRING,BOOLEAN> type or ARRAY<MAP<STRING,STRING>>

[robdiciuccio]

@graceguo-supercat I tested with the following table and data in our Presto (Hive) cluster, and was unable to replicate the error.

CREATE TABLE default.superset_test (
  id bigint,
  name varchar,
  list_ints array(bigint),
  list_maps array(map(varchar, varchar)),
  map1 map(varchar, varchar),
  map2 map(varchar, boolean),
  ds date
);

INSERT INTO default.superset_test VALUES (1, 'San Francisco', ARRAY[1,2,3], ARRAY[MAP(ARRAY['foo'], ARRAY['bar']), MAP(ARRAY['fad'], ARRAY['baz'])], MAP(ARRAY['foo'], ARRAY['bar']), MAP(ARRAY['foo'], ARRAY[True]), date '2019-01-09');

select * from default.superset_test;

[graceguo-supercat]

@robdiciuccio Do you test this PR with Hive Struct data type? otherwise it may break extra cases.

cc @john-bodley