Support filters for non-index pattern visualizations in dashboard

[joshuali925]

Is your feature request related to a problem? Please describe.

Currently filters in OSD dashboard are based on index patterns and by default they are applied to all visualizations regardless of the index pattern used for the visualization. This causes two issues:

1. Difficult to filter visualizations based on other data sources that do not use index patterns, e.g. PPL based visualizations
2. Filters are applied to all visualizations in the same dashboard
   1. If a dashboard contains two visualizations (V1, V2) from two index patterns (I1, I2), adding a filter on a field from I1 that's not in I2 would make V2 blank
   2. If user changes courier:ignoreFilterIfFieldNotInIndex to true, then V2 is not affected, but there is no visual indication of "V2 is not affected" or which filter is applied to which visualization

Describe the solution you'd like

1. Support filtering on fields that's not from an index pattern but from an index or user defined or defined by embeddable
2. Visually indicate which filter is applied to which visualization
3. Probably make courier:ignoreFilterIfFieldNotInIndex true by default, I didn't understand why this defaults to false

Describe alternatives you've considered

Make PPL based visualizations take DSL and DQL filters. But it doesn't address the missing visual indication issue and user has to type out the filter, which is harder to use than the dropdown selector.

Additional context

Need to discuss more on the solution.

[joshuali925]

@KrooshalUX @anirudha tracking issue for PPL visualization filters, feel free to add comments/questions, thanks

[kavilla]

This might be an epic since right now the architecture of OpenSearch Dashboards is built around index patterns and how this information is passed between apps. To support this feature might lead down a rabbit hole.

I'm 50/50 on this one as to me, if I was adding a visualization to a dashboard I would do so with the assumption that this information is relative and filters will be applied to all the visualizations on all of them. If the field is not present that's still insightful information.

I would suggest the alternative and pipe the filters and convert them. It provides a more cohesive experience because users can also pin global filters and this request wouldn't address global filters.

[joshuali925]

If the field is not present that's still insightful information.

I think it depends on what's in the dashboard. If data are all coming from similar index patterns then this is a valid point, but if we are adding other data sources like Prometheus metrics in dashboards, then it's expected for them to not have those fields.

I would suggest the alternative and pipe the filters and convert them.

Currently it's done by converting filters to PPL, but since filters are built around index patterns, it will not automatically populate fields from the PPL query. It would be more flexible if it takes fields offered by the embeddable implementation rather than only those defined in the index patterns that are associated with the embeddable.

But I definitely agree that this needs major changes on how OSD dashboard works and needs more discussion on how/if it should be done.

[joshuarrrr]

@joshuali925 This request has major implications for some of the fundamental technical assumptions that are implemented consistently throughout the OpenSearch Dashboards core.

Index Patterns are the primary abstraction for interacting with OpenSearch Data, and they're a concept unique to OpenSearch Dashboards (there is no equivalent concept in OpenSearch). The original developers easily could have built all the visualization and dashboarding functionality on raw indices instead, but chose not to. It may well be worth revisiting our fundamental data abstractions, but that's a product and UX decision as part of a long term plan from @dagneyb and @kgcreative which would have ripple effects across the entire platform, so shouldn't be taken lightly.

Similarly, one of the key design constraints of the Dashboard plugin is the expectation that queries and filters are applied globally. As you mention, it's common for dashboards to mix visualizations built on different index patterns, and the filtering UI provided by the data plugin already accounts for that. I agree that we need a unified design approach for global actions that are not supported by particular panel visualizations, as I called out here: #1242 (comment)

As part of #2840, a similar idea I had in mind was the concept of "capabilities" for visualization panels, where visualization configurations could declare which global interactions they support (or which they don't). That would make it much easier to implement whatever design treatment we come up with.

I think we're blocked on moving forward with either of these changes until we have more comprehensive product and UX guidance.

[KrooshalUX]

@joshuarrrr to piggy back on "As part of #2840, a similar idea I had in mind was the concept of "capabilities" for visualization panels, where visualization configurations could declare which global interactions they support (or which they don't). That would make it much easier to implement whatever design treatment we come up with."

I am currently working on a "Visualization Container" (because PANEL is a very loaded word in the context of both OUI and OpenSearch Dashboards & Observability plug-in) which is informed by the Date Range and the Anywhere projects, primarily (and to address your concern from #1242 : Clutter of query controls/displays on any number of panels)

In the near term, this Visualization Container can help us communicate to users which visualizations have or have not been filtered via the query bar, as the ideal end state (every visualization being filterable regardless of index pattern / query language) is a much larger project.

Draft of OUI issue opensearch-project/oui#736

I will follow up on this issue once we have a proposal within OUI for the visualization container, which I am targeting Mid-May in order to address the potential customer confusion of the current reality.