[db-engine-spec] Aligning Hive/Presto partition logic #7007
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
john-bodley
force-pushed
the
john-bodley--latest-partition
branch
from
bf70267
to
f1d2c1e
Compare
Codecov Report
@@ Coverage Diff @@
## master #7007 +/- ##
==========================================
- Coverage 64.46% 64.45% -0.01%
==========================================
Files 421 421
Lines 20521 20523 +2
Branches 2245 2245
==========================================
Hits 13229 13229
- Misses 7165 7167 +2
Partials 127 127
Continue to review full report at Codecov.
|
graceguo-supercat
pushed a commit
to graceguo-supercat/superset
that referenced
this pull request
Mar 18, 2019
(cherry picked from commit 05be866)
xtinec
pushed a commit
that referenced
this pull request
Mar 21, 2019
(cherry picked from commit 05be866)
xtinec
pushed a commit
that referenced
this pull request
Apr 2, 2019
* Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * [bugfix] SQL Lab 'Filter Results' doesn't stick (#7104) When using a "Search Results" criteria, the subset of rows that match the criteria get displayed. While this the filter is applied, if another query is run, the filter is still active, but not displayed in the input text box. After this change, the state of the input box sticks after subsequent queries. (cherry picked from commit d5e8d66) * Injectable statsd client (#7138) * Add ability to inject statsd client; some py test/reqs updates - Updated the metrics logger to allow construction with an existing statsd client, so that it can be configured by external systems or libs. - added requirements to requirements-dev.txt which are needed to run tests-eg coverage, nose - removed dependency on mock lib, it is in python stdlib now - updated tox.ini to remove the now-superfluous deps * add license to test file, and remove blank line at EOF (cherry picked from commit ba19a62) * [Lyft-GA] Enable color consistency in a dashboard (#7135) * Enable color consistency in a dashboard Moved actions, minor UI, allowed dashboard copy Fix linting errors Undo unintentional change Updated and added unit tests Fail quietly if package has not been updated Fail quietly on dashboard copy if package is old * Update packages * Remove unnecessary code * Addressed Grace's comments * Small fix for item key * Reset chart's color during exploration * Do not reset chart form data when exploring chart * Fix double scroll bars when content of sql result table overflows horizontally (#7168) The PR substracts the scrollbar height from the height of the container of the react virtualized table so we don't see double scrollbars. (cherry picked from commit 7ffcabd) * Change number format default * Use smart formatter instead * fix merge issues * Use SMART_NUMBER
xtinec
pushed a commit
that referenced
this pull request
Apr 2, 2019
* Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * [bugfix] SQL Lab 'Filter Results' doesn't stick (#7104) When using a "Search Results" criteria, the subset of rows that match the criteria get displayed. While this the filter is applied, if another query is run, the filter is still active, but not displayed in the input text box. After this change, the state of the input box sticks after subsequent queries. (cherry picked from commit d5e8d66) * Injectable statsd client (#7138) * Add ability to inject statsd client; some py test/reqs updates - Updated the metrics logger to allow construction with an existing statsd client, so that it can be configured by external systems or libs. - added requirements to requirements-dev.txt which are needed to run tests-eg coverage, nose - removed dependency on mock lib, it is in python stdlib now - updated tox.ini to remove the now-superfluous deps * add license to test file, and remove blank line at EOF (cherry picked from commit ba19a62) * [Lyft-GA] Enable color consistency in a dashboard (#7135) * Enable color consistency in a dashboard Moved actions, minor UI, allowed dashboard copy Fix linting errors Undo unintentional change Updated and added unit tests Fail quietly if package has not been updated Fail quietly on dashboard copy if package is old * Update packages * Remove unnecessary code * Addressed Grace's comments * Small fix for item key * Reset chart's color during exploration * Do not reset chart form data when exploring chart * Fix double scroll bars when content of sql result table overflows horizontally (#7168) The PR substracts the scrollbar height from the height of the container of the react virtualized table so we don't see double scrollbars. (cherry picked from commit 7ffcabd) * Change number format default * Use smart formatter instead * fix merge issues * Use SMART_NUMBER
betodealmeida
added a commit
that referenced
this pull request
Apr 3, 2019
…equests (#7032) * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * Add decorator for etag cache * Fetch charts with GET * Small fixes * Fix typo * Compute correct cache key; fix logging * Check perms on cached response * Revert change * If perms fail, return naked response * Fix lint * Compute cache key from all form data * Pass extra_filters in GET request * Fix pylint * Fix flake8 * Use ETags even if no cache is set * Handle adhoc filters * Raise in debug mode * Rename actions * Fix integration tests * Do POST request on new charts * Set extra/adhoc filters only in GET requests * Raise if check_perms fails * Refactor auth * Fix flake8 * Fix js unit tests * Fix js unit tests that fail in lyftga * Fix js * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * Add decorator for etag cache * Fetch charts with GET * Small fixes * Fix typo * Compute correct cache key; fix logging * Check perms on cached response * Revert change * If perms fail, return naked response * Fix lint * Compute cache key from all form data * Pass extra_filters in GET request * Fix pylint * Fix flake8 * Use ETags even if no cache is set * Handle adhoc filters * Raise in debug mode * Rename actions * Fix integration tests * Do POST request on new charts * Set extra/adhoc filters only in GET requests * Raise if check_perms fails * Refactor auth * Fix flake8 * Fix js unit tests * Fix js unit tests that fail in lyftga * Fix js * Fix bad merge * Use far future when max_age=0
xtinec
pushed a commit
that referenced
this pull request
Apr 3, 2019
* Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Fix the white background shown in SQL editor on drag (#7021) This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends. * Show tooltip with time frame (#6979) * Fix time filter control (#6978) * Enhancement of query context and object. (#6962) * added more functionalities for query context and object. * fixed cache logic * added default value for groupby * updated comments and removed print (cherry picked from commit d5b9795) * [fix] /superset/slice/id url is too long (#6989) (cherry picked from commit 6a4d507) * [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027) (cherry picked from commit cc58f0e) * feat: add ability to change font size in big number (#7003) * Add ability to change font sizes in Big Number * rename big number to header * Add comment to clarify font size values * Allow LIMIT to be specified in parameters (#7052) * [fix] Cursor jumping when editing chart and dashboard titles (#7038) (cherry picked from commit fc1770f) * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * [db-engine-spec] Aligning Hive/Presto partition logic (#7007) (cherry picked from commit 05be866) * [fix] explore chart from dashboard missed slice title (#7046) (cherry picked from commit a6d48d4) * fix inaccurate data calculation with adata rolling and contribution (#7035) (cherry picked from commit 0782e83) * Adding warning message for sqllab save query (#7028) (cherry picked from commit ead3d48) * [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037) * Update datasource.py * Update datasource.py (cherry picked from commit c771625) * [csv-upload] Fixing message encoding (#6971) (cherry picked from commit 48431ab) * [sql-parse] Fixing LIMIT exceptions (#6963) (cherry picked from commit 3e076cb) * Adding custom control overrides (#6956) * Adding extraOverrides to line chart * Updating extraOverrides to fit with more cases * Moving extraOverrides to index.js * Removing webpack-merge in package.json * Fixing metrics control clearing metric (cherry picked from commit e619405) * [sqlparse] Fixing table name extraction for ill-defined query (#7029) (cherry picked from commit 07c340c) * [missing values] Removing replacing missing values (#4905) (cherry picked from commit 61add60) * [SQL Lab] Improved query and results tabs rendering reliability (#7082) closes #7080 (cherry picked from commit 9b58e9f) * Fix filter_box migration PR #6523 (#7066) * Fix filter_box migration PR #6523 * Fix druid-related bug (cherry picked from commit b210742) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * [forms] Fix handling of NULLs (cherry picked from commit e83a07d) * handle null column_name in sqla and druid models (cherry picked from commit 2ff721a) * Use metric name instead of metric in filter box (#7106) (cherry picked from commit 003364e) * Bump python lib croniter to an existing version (#7132) Package maintainers should really never delete packages, but it appears this happened with croniter and resulted in breaking our builds. This PR bumps to a more recent existing version of the library (cherry picked from commit 215ed39) * Revert PR #6933 (#7162) * Celery worker for warming up cache * Remove testing changes * Add documentation * Fix lint * WIP dashboard filters * Use new cache so it works with dashboards * Add more unit tests, fix old ones * Fix flake8 and docs * Sparkline dates aren't formatting in Time Series Table (#6976) * Exclude venv for python linter to ignore * Fix NaN error * Changing time table viz to pass formatTime a date (#7020) (cherry picked from commit 7f3c145) * SQL editor layout makeover (#7102) This PR includes the following layout and css tweaks: - Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox - Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane - Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars - css tweaks on the basic.html page so the loading animation appears in the center of the page across the board (cherry picked from commit 71f1bbd) * Celery worker for warming up cache * Remove testing changes * Add documentation * Fix lint * WIP dashboard filters * Use new cache so it works with dashboards * Add more unit tests, fix old ones * Fix flake8 and docs * Fix bad merge and pylint
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR remedies a couple of issues related to Hive/Presto partitions. Specifically:
For Hive there was no check to see whether the Pandas Dataframe was empty and thus an exception was raised when trying to obtain the
_latest_partition_from_df. For context this same issue was resolved for Presto in [sql lab] fix partitionned table has no partitions #2740.Added a check that if the partition value is explicitly
Noneto ensure that the query isn't mutated to include a non-validWHEREclause. The methods simply returnFalse(which is somewhat of a strange return value) which is consistent with other invalid scenarios.Note for (1) this wasn't an issue for
where_latest_partitionas the exception was caught, however it was problematic for theextra_table_metadatawhich is used in SQL Lab for fetching the preview.to: @michellethomas @mistercrunch