Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support mulitple temporal filters in AdhocFilter and move the Time Section away #21767

Merged
merged 30 commits into from
Nov 2, 2022

Conversation

zhaoyongjie
Copy link
Member

@zhaoyongjie zhaoyongjie commented Oct 11, 2022

Background

The Time Section is in the Superset Explore now. The original design is for selecting Druid time partition and time grains.
Over time, Superset has supported more databases. The time filter should be a kind of Adhoc Filters.

Summary

This PR will move functionality of the Time Section to the Adhoc Filters when GENERIC_CHART_AXES is enabled, and then Adhoc Filters should support multiple time columns.

The new operator in the Adhoc filter that is TEMPORAL_BETWEEN has been added, so the new queryObject will support like this

{
  ...
  filters: {
    "col": "Order Date",
    "op": "TEMPORAL_RANGE",
    "val": "DATEADD(DATETIME(\"2022-10-10T00:00:00\"), -20, year) : 2022-10-10T00:00:00"
  }
  ...
}

the filter will generate a SQL snippet

...
WHERE "Order Date" >= '2002-10-10 00:00:00.000000'
  AND "Order Date" < '2022-10-10 00:00:00.000000'
...

Unsupported Charts

  • Legacy Rose
  • Legacy Pivot table
  • Legacy Area
  • Legacy Bar
  • Legacy Compare
  • Legacy DualLine
  • Legacy Line
  • Legacy TimePivot
  • TimeTable

image

multiple.temporal.columns.mov

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Oct 11, 2022

Codecov Report

Merging #21767 (a10d259) into master (102909e) will increase coverage by 0.11%.
The diff coverage is 73.13%.

❗ Current head a10d259 differs from pull request most recent head 059fb2f. Consider uploading reports for the commit 059fb2f to get more accurate results

@@            Coverage Diff             @@
##           master   #21767      +/-   ##
==========================================
+ Coverage   66.86%   66.97%   +0.11%     
==========================================
  Files        1807     1811       +4     
  Lines       69218    69341     +123     
  Branches     7402     7442      +40     
==========================================
+ Hits        46280    46441     +161     
+ Misses      21033    20981      -52     
- Partials     1905     1919      +14     
Flag Coverage Δ
hive 52.90% <55.81%> (?)
javascript 53.47% <61.53%> (+0.08%) ⬆️
mysql ?
postgres 78.44% <95.34%> (+0.01%) ⬆️
presto 52.80% <55.81%> (-0.03%) ⬇️
python 81.45% <97.67%> (+0.17%) ⬆️
sqlite 76.92% <97.67%> (+<0.01%) ⬆️
unit 51.12% <76.74%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ackages/superset-ui-chart-controls/src/fixtures.ts 100.00% <ø> (ø)
...d/packages/superset-ui-chart-controls/src/index.ts 100.00% <ø> (ø)
...perset-ui-chart-controls/src/sections/sections.tsx 71.42% <0.00%> (-16.08%) ⬇️
...chart-controls/src/shared-controls/dndControls.tsx 58.33% <ø> (ø)
...d/packages/superset-ui-chart-controls/src/types.ts 100.00% <ø> (ø)
...kages/superset-ui-core/src/query/types/Operator.ts 100.00% <ø> (ø)
...packages/superset-ui-core/src/query/types/Query.ts 100.00% <ø> (ø)
...lugin-chart-handlebars/src/plugin/controlPanel.tsx 50.00% <ø> (ø)
superset-frontend/src/components/Modal/Modal.tsx 89.18% <ø> (ø)
...set-frontend/src/explore/actions/hydrateExplore.ts 57.14% <ø> (ø)
... and 73 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@zhaoyongjie zhaoyongjie force-pushed the feat/time_column_in_filters branch 3 times, most recently from d05501d to 8157fec Compare October 12, 2022 12:12
@zhaoyongjie zhaoyongjie marked this pull request as ready for review October 13, 2022 11:14
@zhaoyongjie zhaoyongjie changed the title feat: support mulitple datetime filter in AdhocFilter and move away time section [WIP]feat: support mulitple datetime filter in AdhocFilter and move away time section Oct 13, 2022
@zhaoyongjie zhaoyongjie marked this pull request as draft October 13, 2022 14:48
@zhaoyongjie zhaoyongjie marked this pull request as ready for review October 14, 2022 11:02
@zhaoyongjie zhaoyongjie force-pushed the feat/time_column_in_filters branch 2 times, most recently from a00e71d to bba9277 Compare October 15, 2022 03:29
@zhaoyongjie zhaoyongjie changed the title [WIP]feat: support mulitple datetime filter in AdhocFilter and move away time section feat: support mulitple datetime filter in AdhocFilter and move away time section Oct 17, 2022
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass comments. Super stoked to finally see the time section being removed! Also, it would be really great if we could have some design eyes on the adhoc filter modal, as the time picker pill feels slightly sketchy in this particular context (even making the pill full width would be an improvement IMO)

Comment on lines 63 to 70
if (hasGenericChartAxes && query.time_range) {
// eslint-disable-next-line no-param-reassign
query.filters = ensureIsArray(query.filters).map(flt =>
flt?.op === 'DATETIME_BETWEEN'
? ({ ...flt, val: query.time_range } as BinaryQueryObjectFilterClause)
: flt,
);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is for applying native/cross filters to the adhoc time filters? I think it would be really important to be able to specify to which adhoc time filters native/cross filters apply. In the adhoc filter modal we could have a checkbox under the time picker that says "Overridable by native time filter" or similar. Then if that field is unchecked, the query.time_range would not apply to the filter.

Copy link
Member Author

@zhaoyongjie zhaoyongjie Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's for applying native filters to the ad-hoc filter on the Dashboard, this part will override all ad-hoc time filters in the charts of the Dashboard.

In the original design is that the chart only has a single time filter so the Time Filter in the Native Filter was designed only to pass time_range to every chart rather than pass time column(granularity in the QueryObject) and time value(time_range in the QueryObject) to the chart.

To solve this issue, I think we might need to specify the time columns in the Native Filter so that the user knows which time column would apply.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's two distinct use cases here:

  1. Emitting a generic temporal value to be applied to a chart's own temporal column - this is the behavior of the current time filters (both dashboard native filters and filter box). This is handy when charts may have different temporal columns, for instance created_at or updated_at, and we don't want to change the temporal column, but rather just replace the value of the temporal filter.
  2. emitting a time filter for a specific temporal column. This would make it possible to have, for instance, two different time filters with different underlying columns, e.g. created_at and updated_at, and emit those filters along with the target column to all charts that are in scope.

I believe there's probably a need for both, and we may need to revisit the second option at some point, but for now let's at least make sure we don't break support for the first case.

superset-frontend/src/explore/constants.ts Outdated Show resolved Hide resolved
@zhaoyongjie zhaoyongjie changed the title feat: support mulitple datetime filter in AdhocFilter and move away time section feat: support mulitple datetime filter in AdhocFilter and move the Time Section away Oct 17, 2022
@zhaoyongjie zhaoyongjie changed the title feat: support mulitple datetime filter in AdhocFilter and move the Time Section away feat: support mulitple temporal filter in AdhocFilter and move the Time Section away Oct 17, 2022
@zhaoyongjie zhaoyongjie changed the title feat: support mulitple temporal filter in AdhocFilter and move the Time Section away feat: support mulitple temporal filters in AdhocFilter and move the Time Section away Oct 17, 2022
Comment on lines -80 to +84
if tt in (utils.TemporalType.TEXT, utils.TemporalType.DATETIME):
if tt in (
utils.TemporalType.TEXT,
utils.TemporalType.DATETIME,
utils.TemporalType.TIMESTAMP,
):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by catch: SQLite also has a timestamp type.

Copy link
Member

@codyml codyml left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not doing a full review because most I'm pretty out of my depth on most of the code changes, but I tested locally on several ECharts and it seems to work great! I did see a few issues, but they were also present on master so not sure if they're relevant here:

  • "Original Value" option in time grain is a little buggy for me, if you select it the selection disappears but if you select it again it appears.
  • For charts with "Temporal X-axis", you can't remove the default item with the little "x". If there are two options and you choose one that's not default, then click the "x", it'll just go back to the original one.
  • Some charts had a tooltip value that seemed off:

Screen Shot 2022-10-18 at 7 09 52 PM

My one other comment is that checking feature flags at module execution time has been responsible for some weird bugs in the past due to race conditions between when they're set and read (that's what #21315 was fixing). It would be great if we could only check feature flags during React rendering or other execution points that are guaranteed to happen after initial module execution. That honestly looks pretty hard to do in this PR, but just wanted to note that this might be a source of bugs in the future.

@apache apache deleted a comment from jinghua-qa Oct 31, 2022
@github-actions
Copy link
Contributor

@zhaoyongjie Ephemeral environment spinning up at http://54.188.24.204:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

@jinghua-qa
Copy link
Member

Thank you for the great work~
I have a question, it looks like when i use custom sql in filter with time column, time comparison did not support, is this expected?
Steps:
1, add a time column to filter
2, use time comparison to define start and end
3, go to time comparison and select compare time

Expect:
time comparison will work with time filter

Actual:
time comparison did not work with time filter

Screen.Recording.2022-11-01.at.8.36.28.AM.mov

@zhaoyongjie
Copy link
Member Author

Thank you for the great work~ I have a question, it looks like when i use custom sql in filter with time column, time comparison did not support, is this expected? Steps: 1, add a time column to filter 2, use time comparison to define start and end 3, go to time comparison and select compare time

Hi @jinghua-qa, the Time Comparision + Custom SQL doesn't been supported yet. Currently, the Time Comparision should with time range in the Adhoc filter rather than Custom SQL.

@jinghua-qa
Copy link
Member

Thank you for the great work~ I have a question, it looks like when i use custom sql in filter with time column, time comparison did not support, is this expected? Steps: 1, add a time column to filter 2, use time comparison to define start and end 3, go to time comparison and select compare time

Hi @jinghua-qa, the Time Comparision + Custom SQL doesn't been supported yet. Currently, the Time Comparision should with time range in the Adhoc filter rather than Custom SQL.

Are we going to support this feature in the future? it looks like when we move the time range to filter, user can use custom sql to support time column as filter, it will be great if we can support that with time comparison.

@zhaoyongjie
Copy link
Member Author

zhaoyongjie commented Nov 1, 2022

Thank you for the great work~ I have a question, it looks like when i use custom sql in filter with time column, time comparison did not support, is this expected? Steps: 1, add a time column to filter 2, use time comparison to define start and end 3, go to time comparison and select compare time

Hi @jinghua-qa, the Time Comparision + Custom SQL doesn't been supported yet. Currently, the Time Comparision should with time range in the Adhoc filter rather than Custom SQL.

Are we going to support this feature in the future? it looks like when we move the time range to filter, user can use custom sql to support time column as filter, it will be great if we can support that with time comparison.

If we want to support custom SQL + Time Comparison, there are some pre-requirement works, e.g. 1) parsing and validating it. 2) it should contain the start time and end time.

This part of the work is completely beyond the scope of this PR.

Copy link
Member

@jinghua-qa jinghua-qa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhaoyongjie zhaoyongjie merged commit a9b229d into apache:master Nov 2, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Ephemeral environment shutdown and build artifacts deleted.

@korpa
Copy link

korpa commented May 11, 2023

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns

  2. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

@bilalabdullah44000
Copy link

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns
  2. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

Any update on this ??

@zhaoyongjie
Copy link
Member Author

zhaoyongjie commented Jul 7, 2023

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns

What version of Superset do you use? Native filter in the Dashboard should append a new time filter, so I think it should be worked on Dashboard.

  1. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

Should support Jinja Template if not, it should be a regression bug.

@bilalabdullah44000
Copy link

bilalabdullah44000 commented Jul 7, 2023

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns

What version of Superset do you use? Native filter in the Dashboard should append a new time filter, so I think it should be worked on Dashboard.

  1. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

Should support Jinja Template if not, it should be a regression bug.

  1. In jinja, it is giving result in the form of
    {
    ...
    filters: {
    "col": "Date1",
    "op": "TEMPORAL_RANGE",
    "val": "DATEADD(DATETIME("2022-10-10T00:00:00"), -20, year) : 2022-10-10T00:00:00"
    },
    {
    "col": "Date2",
    "op": "TEMPORAL_RANGE",
    "val": "Last day : 2022-10-10T00:00:00"
    }

             }
    

ideally it should give result in the form of timestamp so it can be easily used in sql just like from_dttm and to_dttm. Can we parse above values to get timestamp from above format effectively ?

  1. Also at my end, if i create two dashboard filters i.e. created_at and updated_at , it is applying filter value on my default datetime/temporal column i.e. last_state_change_time

Screenshot from 2023-07-07 17-09-01

@zhaoyongjie
Copy link
Member Author

zhaoyongjie commented Jul 7, 2023

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns

What version of Superset do you use? Native filter in the Dashboard should append a new time filter, so I think it should be worked on Dashboard.

  1. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

Should support Jinja Template if not, it should be a regression bug.

  1. In jinja, it is giving result in the form of
    {
    ...
    filters: {
    "col": "Date1",
    "op": "TEMPORAL_RANGE",
    "val": "DATEADD(DATETIME("2022-10-10T00:00:00"), -20, year) : 2022-10-10T00:00:00"
    },
    {
    "col": "Date1",
    "op": "TEMPORAL_RANGE",
    "val": "Last day : 2022-10-10T00:00:00"
    }
             }
    

ideally it should give result in the form of timestamp so it can be easily used in sql just like from_dttm and to_dttm. Can we parse above values to get timestamp from above format effectively ?

The val in each time filter means that dynamically calculate "from date time" and "to date time". If you just want to use fixed timestamp, leave timestamp string in Custom, for instance:

image
  1. Also at my end, if i create two dashboard filters i.e. created_at and updated_at , it is applying filter values against these 2 filters on my default datetime/temporal column i.e. last_state_change_time

Screenshot from 2023-07-07 17-09-01

It might be a regression bug, I've revert this PR in my custom Superset. I hope it should help with you.

@zhaoyongjie
Copy link
Member Author

@bilalabdullah44000 posted an incorrect link, just updated

@bilalabdullah44000
Copy link

Two questions on this feature:

  1. How does this work in dashboards. Even if I add two time range filters in a dashboard, it is not possible to select the correct date columns

What version of Superset do you use? Native filter in the Dashboard should append a new time filter, so I think it should be worked on Dashboard.

  1. How does this feature work with jinja templates? Are there any other variables than {{ from_dttm }} and {{ to_dttm }} to support this feature?

Should support Jinja Template if not, it should be a regression bug.

  1. In jinja, it is giving result in the form of
    {
    ...
    filters: {
    "col": "Date1",
    "op": "TEMPORAL_RANGE",
    "val": "DATEADD(DATETIME("2022-10-10T00:00:00"), -20, year) : 2022-10-10T00:00:00"
    },
    {
    "col": "Date1",
    "op": "TEMPORAL_RANGE",
    "val": "Last day : 2022-10-10T00:00:00"
    }
             }
    

ideally it should give result in the form of timestamp so it can be easily used in sql just like from_dttm and to_dttm. Can we parse above values to get timestamp from above format effectively ?

The val in each time filter means that dynamically calculate "from date time" and "to date time". If you just want to use fixed timestamp, leave timestamp string in Custom, for instance:

image > 2. Also at my end, if i create two dashboard filters i.e. created_at and updated_at , it is applying filter values against these 2 filters on my default datetime/temporal column i.e. last_state_change_time > > ![Screenshot from 2023-07-07 17-09-01](https://user-images.githubusercontent.com/111341070/251736265-5c9cbb55-c986-42c6-ac8e-caf6d8515368.png)

It might be a regression bug, I've revert this PR in my custom Superset. I hope it should help with you.

I Understand. if i select timecolumn , then the time range filter will be applied on the column selected in the time column. but how do i apply multiple timerange filters i.e. filter on both created_by column and updated_by column at dashboard level ?

@bilalabdullah44000
Copy link

@zhaoyongjie Can we apply multiple time range filters at the same time on dashboard as well like slice explore ?

@zhaoyongjie
Copy link
Member Author

zhaoyongjie commented Jul 8, 2023

@zhaoyongjie Can we apply multiple time range filters at the same time on dashboard as well like slice explore ?

IMO, Dashboard must be applied multiple time range filter. If not, it's a bug.

@bilalabdullah44000
Copy link

@zhaoyongjie Can we apply multiple time range filters at the same time on dashboard as well like slice explore ?

IMO, Dashboard must be applied multiple time range filter. If not, it's a bug.

Here in attached screenshot, I have applied two time range filters i.e. created_at and updated_at.

As seen in the query, Only one filter is applied on default temporal column or on the column which is selected in time column field in dashboard filters section.

251736265-5c9cbb55-c986-42c6-ac8e-caf6d8515368

@zhaoyongjie
Copy link
Member Author

zhaoyongjie commented Jul 12, 2023

@bilalabdullah44000 You need to use Time Column filter and Time Range filter together to select multiple time filter, but I also found this feature was not correct work on the master(git hash:4881328fb), the Time Range replaced all of the value of time filters. I suggest that the first step revert the PR #23021 and then try to fix it in your own side.

Multiple time filters in the Explore page, one is Last Month, another is Last Year

image

Multiple time filters on Dashboard was not work

image

@zhaoyongjie
Copy link
Member Author

@bilalabdullah44000 You need to use Time Column filter and Time Range filter together to select multiple time filter, but I also found this feature was not correct work on the master, the Time Range replaced all of the value of time filters. I suggest that the first step revert the PR #23021 and then try to fix it in your side.

image

By this feature, I think we should redesign the Time Filter on the Dashboard, should merge the Time Column/Time Range/Time Grain together as a unique Time Filter.

Unfortunately, the new Time Filter almost didn't support multiple filters on the Dashboard, and some codebase was broken previous design, ---- the Native Filter should APPEND the new where clause rather than REPLACE value of filter of chart.

@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 and removed 🚢 2.1.3 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XXL 🚢 2.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants