[PROPOSAL] Update OpenSearch Dashboards' Design Strategy

[ahopp]

What problem are you trying to solve?

Historically, delivering disparate features via plugin tabs was the only way developers could historically add features to OpenDistro given the limitations of the influencing an actively commercialized upstream. This approach not only solved the problem of the commercialized upstream and helped differentiate what was delivered from upstream solutions among others. It is therefore completely understandable that developers built OpenDistro the way they did. BUT after the fork, it has led to some unintended outcomes in design that no longer serve the best interests of the project or community. These include;

• Disparate, Duplicative Design Patterns: OpenDistro plugins were made to explicitly align to AWS patterns instead of upstream interaction patterns. This caused plugins (ex. Anomaly Detection) and features (ex. security interfaces) to be aligned to novel patterns rather than the native interfaces available in the upstream (now OpenSearch Dashboards). Rather than disparate, duplicative design patterns, we should reuses design patterns to assist in usability, user comprehension, and user adoption...and they should be agnostic of historic upstream products or downstream managed service patterns.
• Disparate, Duplicative Experiences: As OpenDistro did not have access to the core dashboards primitives, or a supportive upstream at the time, OpenDistro developers explicitly chose to implement duplicative experiences rather than evolving, contributing, or replacing core functionality. For example, I would argue that the Observability Log Explorer could have augmented Discover by adding Metrics and Trace correlations to the existing Log exploration capabilities in OpenSearch Dashboards core, but the upstream project made those contributions impossible. To deliver value to OpenDistro users, they made the only choice available at the time; prioritize feature delivery at the expense of interoperability and composition. Rather than disparate, duplicative experiences, we reuse features and processes that would otherwise overlap between dedicated plugins or user applications.
• Disparate, Duplicative Information Architecture: As OpenDistro and OpenDistro Plugins did not rely on common patterns, each plugin implemented distinct patterns at the time they were built, which were not revised holistically after the fork. Additionally, OpenDistro plugins each implemented crud patterns that did not align with existing patterns, resulting in different information architecture, navigation, and page structures then those available in native Kibana (and later OpenSearch Dashboards). Rather than disparate, duplicative information architecture, we should provide a consistent information architecture across the experience to support known-item seeking, exploratory seeking, re-finding, etc. (including the ongoing maintenance to support the continued evolution of the project).
• Disparate, Duplicative Object Store/API Schemas: As a result of upstream projects making breaking changes on minor and patch releases, OpenDistro developers did not have confidence that APIs would remain stable. As a result, plugins made the explicit choice to implement their own object storage and API schemas. Rather than relying on the Kibana saved objects API, plugin registration hooks, and other internal Kibana APIs and methods, each plugin relied on their own implementation and patterns. Examples of this include Alerting, Anomaly Detection, Reporting, and Index Management plugins. None of those natively use the save object APIs for managing dashboard objects, and as a result, they rely on inconsistent implementations and interactions with the Security Plugin, tenancy, and role management. Rather than disparate, duplicative object stories/API schemas, we should be as unified as possible while still supporting the distinct, ever-evolving needs of use-case specific applications of OpenSearch.

We need to improve the overall design approach for the current OpenSearch Dashboards model and specifically we should develop a more holistic design approach for the project in order to maintain and improve the long-term usability of the OpenSearch Dashboards experience inclusive of related project features (e.g., Observability, Alerting, etc.).

Why should we solve it?

While the UX goal is still be to delight users by delivering great features and experiences across the project, there is an opportunity to de-risk our long-term project aspirations by aligning to a more unified design strategy and allow the community to better stack feature value via an ever-expanding portfolio of features and capabilities. Some of the specific reason to solving it now include;

1. The set of constraints that existed with OpenDistro no longer apply to the OpenSearch Project. While the current model was designed to address the challenges of OpenDistro, it is now preventing the project from more holistic improvement of the user experience (ex. ) and it serves as obstacles to innovating (ex. adding functions and capabilities that elevate the entire Dashboards experience). Either way, these resulting usability and composability challenges need to be resolved.
2. These challenges will continue to grow as the feature diverge without a more paternalistic approach to the design and UX of OpenSearch Dashboards. The earlier we can provide tools and guidance to the community and project, the less technical and design debt we will create. Additionally, some of our tech debt has had a cooling effect on contributions to Dashboards (e.g., different charting libraries, different visualizations, etc.), which has led to further isolated experiences, challenges in integrating these features across the product surface, and inconsistent experiences.
3. Since some features can span all products (e.g., some observability features have a core component, an OpenSearch Dashboards component, and a Data Prepper component) while other features span a few products (e.g., the OpenSearch Dashboards ISM is a front-end feature that manages state management jobs that exist in OpenSearch), it is my opinion that building these features in an isolated way prevents OpenSearch Dashboards users from fully capitalizing on the benefits while increase the integration cost for isolated experiences. A more central approach allows the project to better fuel the OpenSearch platform flywheel.

How do you propose to solve it?

To provide OpenSearch Dashboards users with great out-of-the-box experiences for any/all of their use cases (current and future) while enabling seamless integration of capabilities across vertical solutions, we need to provide a platform that allows end-users and builders to create experiences that are purpose built for their use cases, that are easy to use and extend, and fit within a consistent UX framework. To support OpenSearch Dashboards user’s long-term needs, my recommendation is to provide a platform that allows users and builders to curate their experience and custom tailor them to their use case. All features should offer a consistent product look, feel, interaction affordances, interoperability, and more. New functionality should benefit all use cases wherever possible, and should be delivered in a consistent way. In this way, OpenSearch Dashboard can begin to serve as a platform for users and developers to compose use-cases for users as curated experiences within, or across, existing products rather than as separate products (e.g., plugins with different design affordances, UX patterns, etc.).

In order to do this, we should invest in flattening our information architecture, design composable primitives (e.g., Query Editor, Syntax highlighting, autocomplete, and a variety of widgets that can be used anywhere in the experience), and establish better extensibility among feature sets so that dashboards admins, feature builders, and end-users can build customized project views for their work that integrate elegantly. In order to achieve this more holistic vision, I have highlighted a few workstreams I believe we need to invest in;

• Enable Common and Consistent Design System: As a platform, the OpenSearch Dashboard team (i.e., contributors and community members) should provide guidance for how features and extensions work. The OpenSearch Dashboards team (inclusive of product, design, and tech contributors on GitHub), should provide direction on the look and feel of Dashboards via the OpenSearch Design system (inclusive of developer and designer ergonomics, documentation, pattern and component libraries, best practices, guidance, user interface guidelines, and use case examples). Additionally, it should be easy to add and self-serve functionality when it exists (e.g., publish components to NPM, make it easy to pull in components and widgets across features, with appropriate documentation of how and when it’s appropriate to do so).
• Enable Common and Consistent Navigation: While we should not dictate how features should work specifically, we should standardize the navigation patterns across all our feature areas. Today, observability, index management, alerting, reporting, anomaly detection, and other features, all have varying information architecture (IA) and navigation paradigms. From tabs, to stacked left navigation, to other affordances, each application area behaves slightly differently, and customers have to learn new patterns in order to do their work. We should unify and standardize our navigation, and publish guidance in our design system documentation.
• Enable Common and Consistent Patterns: Similar to Navigation, OpenSearch Dashboard various features followed multiple design patterns/design systems, and mix and matched them in order keep velocity. This means multiple variable ways of saving, loading, editing objects, as well as different ways of interacting with either a back-end feature, or front-end functionality. We should evaluate all these patterns today, and seek to standardize them and provide guidance in the form of best practices and common patterns and affordances in our design system documentation.
• Enable Common and Consistent Functionality: The current project strategy has been to focus on feature areas, rather than composable components. This largely has meant that each features has had to re-implement components and component improvements on their own, rather than calling core functionality to compose features together. There is a big opportunity in providing more composable components which would allow features to contribute to and extend the core component functionality. As an example, any time a feature needs to call a search bar, if any additional search functionality has been extended by a feature or extension, the feature that consumes that search bar should be able to get those updates without having to explicitly call those extensions. The same can be said of query editors with auto-complete, support for alternative languages, etc.
• Define curated experiences: Rather than building for each use case as disparate application with bespoke UX and design affordances, I think we should allow developers create and run applications (historically called plugins) that address the needs of their use case beyond what the platform allows out of the box that still operate within the same UX and design affordances. In order to help builders provide strong out-of-the-box experiences, we want to empower our community to be able to create applications that provide curated views that are tailored around specific vertical use cases, while staying consistent to familiar interaction patterns, and being able to bring their data and saved objects along. Extension points on the Dashboards platform should make it easy for developers to add new features and connect add-ons that enhance performance and usability. This should also include an explicit top-level container for saved objects and settings for a curated experience. The application/experience owner would then be in control of which features they want to turn on for a specific use case, which will facilitate the creation of curated views for specific purposes. We should disambiguate OpenSearch Dashboards settings from plugin/use case settings from user settings.

For builders, this means a platform with well-documented building blocks that offer a consistent product look, feel, interaction affordances, interoperability, and templates that accelerate the development of high-quality apps and extensions that are well integrated into our ecosystem. Builders will have access to composable primitives that are generalized to benefit most use cases, with standard interfaces that can be leveraged consistently. It will allow app and extension builders to compose experiences that leverage platform primitives, extensions, object sharing, task management, security features and other components that accelerate development and time to value. Apps will be easy to extend and integrate across the ecosystem. For users, this means that in addition to a powerful data ingestion, analytics and visualization platform, they should have access to a catalog of apps and extensions that allows them to quickly add new capabilities, the power to easily share and control who and how they view and share their data, and a centralized place to manage and operate their OpenSearch infrastructure, regardless of whether it’s a single cluster, or a fleet.

Finally, we should ensure decisions are rooted in usability, accessibility and research best practices. Without a grounding on user research, many have been flying largely blind, and relying primarily on heuristics and anecdotal evidence, lacking clear user-centered methodology to ground us in user needs. @kgcreative and his team are working to design, with the community, a new UX processes to ensure features and product plans and development are not only grounded on UX and Engineering best practices, but also grounded in an increased understanding of our customers, and their needs and pain points.

Next Steps

There’s a lot on this list in terms of outcome and it may realistically take multiple years to fully achieve them (i.e., duplicative experiences will coexist while a path to consolidation and composability is charted) but I think there are a number of immediate workstream we can and should kick off.

• Align Duplicative Experiences: In order to align the experiences in OpenSearch Dashboards, we should audit use cases across the application and plugins, and propose a path to consolidating these experiences with the goal of reducing cognitive load, and helping customers better interact with their data in more consistent ways. We can provide a single integrated suite of powerful, composable analytics primitives that customers can use to explore and analyze their data. This workstream should include creating an inventory of all duplicative experiences, building a burn down list of features to deduplicate in GitHub, and deduplicating components that solve the same problem. @apasun has stepped up to start looking into this support from @kgcreative.
• Build Common Design System (ex. EUI Fork) : Earlier this year we forked EUI into OUI which resolved both a number of CVEs (200+) as well as resolved any license issues. Next, we need to update the design system to better consider extension registration points (and any future Dashboard SDK), unify component and patterns used across dashboards and plugin projects (including iterative UX/UI, implementation and documentation changes). Eventually, we will need to have a fully refreshed version of OUI (across both UX and Tech) with complete documentation. @KrooshalUX is already hard at work understanding what is the current system and working to make a recommendation on what should come next.
• Migrate to a Common Charting Library and build a visualization abstraction layer (ex. Charting, Abstraction): Today, there are at five different charting libraries that are use to render different visualization types in OpenSearch Dashboards (OSD). This makes it difficult for developers to provide a consistent user experience across visualization types, because any vertical feature that targets all visualizations must be implemented separately in each. We will provide OpenSearch Dashboards users with a clean, unified charting experience that looks polished and consistent while providing OpenSearch Dashboards developers with unified interfaces to develop, enhance, and interact with charts. This will come in the form of a new abstraction layer that will unify and standardize our interfaces for interacting with the lower level charting primitives. @joshuarrrr is already driving this work (with support from @ashwin-pc, @seanneumann among others).
• Define and build the framework for curation (i.e., the use-case specific projects): A mechanism for curating experiences will need to be designed, which will allow feature owners, admins, and users to define project templates to make it easier to launch use-case specific projects that provide a curated lens and view into the data that end-users want to see. I'm hoping to help support workstream with support from the Dashboards maintainers like @kavilla and the design folks like @kgcreative.

In parallel, @kgcreative and the design folks across the Dashboards project (including @apasun, @KrooshalUX, @btzeng) will be supporting these efforts by;

• Researching and Reporting on Existing and Future User Personas: Researching and codifying personas will help OpenSearch understand current and future users’ needs, experiences, behaviors, and goals. In the short-term, clear personas make the design tasks at hand less complex, they help guide the project through ideation processes, and they can help the project to achieve the goal of creating a good user experience. In the long-term, they ensure OpenSearch is aligning feature development to customer and user needs.
• Researching and Reporting on Existing and Future User Journeys: User journey mapping is a useful activity for bringing projects together to create one shared project-wide vision for prioritizing design and user experience ideas (as well as what should ultimately be built). The team will initially focus on current-state journey maps to visualize the experience customers have when attempting to accomplish a goal with OpenSearch as it exists today which will inform both the information architecture refresh and aligning duplicative journeys. The team will then transition to future-state journey maps to help visualize the best case, ideal-state journey for an existing product or a journey for an aspiration OpenSearch future state.
• Normalizing the Information Architecture Across OpenSearch Project: While information architecture is not visible to end-users, it is the backbone for the design. The goal of this workstream would be to create an experience that allows the user to focus on their tasks, not on finding their way around. Once the user journeys, we can align the long-term information architecture with the user’s needs.
• And General Design Guidance and Support!

Finally, this design strategy needs to include a stronger prioritization of Accessibility and Internationalization. For OpenSearch Dashboards, accessibility will focus on identifying barriers that exist in the product and working to remove them. OpenSearch Dashboards should seek to be accessible to, and delightful for, all users, specifically including the ~15% of the world population that has a significant disability. This includes accessible color palettes, compatibility with screen readers, meaningful keyboard navigation, and adherence to visual design guidance. Specifically to Internationalization, OpenSearch Dashboards is already being used globally, in many countries and languages. The north star should be a localized experience in the customer’s language of preference with a design following the language conventions.

Note: I also think we need to invest in building a more unified management experience (including security, administration, etc.) as well as develop an updated model for more unified objects across OpenSearch Dashboards and associated features but I think these fit better in dedicated issues/proposals. I think @setiah has started thinking about the unified management / admin experience, but I'm not sure about the current progress for more unified objects across the repos.

What are remaining open questions?

While I tried to focus on the opportunities I've been seeing in the overall design strategy for the project, I know there are still a ton of opportunities to be highlighted and/or included here. Please poke holes, provide feedback, identify gaps, and make alternative proposals for @kgcreative and I to respond to! I'd like to get as much input as possible before working with @kgcreative to build out a more detailed design/roadmap for the design and product work (upstream to the technical work).

[kavilla]

@ahopp do we plan on migrating old experiences or keeping current models the same (like how alerting does stuff)? Because then we should mention security and permissions in the design strategy.

[ahopp]

@kavilla good question! I don't think this strategy necessarily dictates either option. Rather it means making design and development decisions informed not by isolated use cases but informed but the holistic user experience and an overall view of the user's needs. In some cases that might be slightly modifying current experiences and in other cases it might required some migrating. @kgcreative any thoughts to add here?

For security specifically, I think it makes sense to unify the current CX into a more cohesive security experience that is inclusive of current and future use cases, e.g., permission of multidata source, isolation of extensions, etc. But given that, what type of clarification would you like to see above?

[apasun]

Excited to work with you on this @ahopp!

[nandi-github]

In order to pursue the new direction in Ux/Visualization, is there any standardization or change required from the core side of the OpenSearch ?

[ahopp]

pursue the new direction in Ux/Visualization, is there any standardization or change required from the core side of the OpenSearch

@nandi-github Can you help me understand what you mean by "Ux/Visualization"? There's multiple approaches depending on what you want to do in said "new direction", but a flow through the process might look like this;

1. Open an issue speaking to the gap in the current direction
2. Make a proposal to resolve with new direction
3. Get alignment with community and maintainers
4. Submit a PR for the changes
5. Release in next applicable version

[ahopp]

Excited to work with you on this @ahopp!

Likewise!

[ahopp]

Hello all! I had some feedback come in to me directly and I'm adding it here (with my responses) for everyone's visibility.

Who is the customer/user we are trying to solve this for? Is this for managed service users only or all OpenSearch users?

I don't see this design approach as forcing a decision between any managed service, ISV, or any specific OpenSearch Dashboards user. Enabling common and consistent navigation, affordances/user patterns, functionality, etc. while also enabling a design system that is consistent across the project benefits all users, IMO. In fact, I've received feedback on the usability of the product on diverse set of users from many different use cases. Similarly, the challenges of usage and adoption, internationalization, accessibility, and charting are common for both customers.

Simply put, a unified design strategy/approach is Kevin and I's proposal for an improved flywheel for both MS and OSS OpenSearch users.

If the latter, what happens to plugins already developed by these users.

I'm not sure I understand this question. Nothing about the current design proposal is forcing developers or users away from current plugins. We're also not forcing developers or users to change any of their current approaches. We are simply recommending a design approach that we believe provides the best, long-term flywheel for all OpenSearch users. Our goal is to make this approach default by the sake of its merit, not by restricting the development of our community.

I also think Discover has bigger scope than just logs/metrics/traces ("Disparate, Duplicative Experiences" above). It is meant for any data analysis. Is there any reason why it should be the starting point for all log/metric/trace interactions?

I 100% agree that Discover has a bigger scope. This is core to why I believe that the enhancement of the core discover flow is a better path for long-term value then individual and isolated investments in parallel improvements.

As to which we should have one starting point, frankly, I can't think of a user-centric reason for forcing users into disparate, isolated experiences for the same core use case of data exploration and discovery - or at least not one that risks usability, adoption, ubiquity, etc. We can absolutely provide paternalistic UI so each use case is met in a bespoke manner (or allow the user to take distinct paths to complete their objective) but I believe a unified starting point is preferred. There are also a number of product maxims I've believe to be true (e.g., "it better to meet users where they are", "reduce users cognitive load as much as possible", "don't force users to learn more than absolutely required", "form should imply function where possible") that would suggest a unified starting point is best in the long-term.

[ashwin-pc]

@ahopp I wonder if we should add documentation of existing features, abstractions, libraries and processes as a requirement to the design strategy. If we want everyone to align on a unified design, we should also make it the path of least resistance. Today the biggest hurdle to implement a feature is simply identifying the right way to do it, causing a lot of shortcuts to be taken that diverge from the unified design. And before we blame the feature developer for the divergence, we often dont have good docs that can refer to to do the right thing. The time it takes to identify the right approach without this may far outweigh the time it takes to build the feature itself. And if each feature developer needs to do this each time, thats a lot of wasted time that can be avoided by simply having a good set of foundational docs and processes that a feature developer can follow.

For example we have many ways in which we can interact with the index. Each plugin has implemented custom strategies for this. Now with multiple datasources this problem is compounded. The documentation we have today for interacting with the datasource is very limited so each plugin build using the best reference abstraction that they can find instead of the right abstraction because we lack the docs that they can use to figure this out. This causes a divergence. And honestly I would not blame them for doing so because reverse engineering these abstractions is not trivial and very time consuming.

[joshuarrrr]

I think this proposal has been accepted and largely informs the way we work. @dagneyb @seanneumann Any reason to keep this open?

[kgcreative]

@joshuarrrr - I think this can be closed at this point.