[Feature Anywhere] Performance test the new feature anywhere integration across OSD and plugins.

[amitgalitz]

We should have metrics we look at to make sure the new feature anywhere doesn't bring any degradation to cluster or current OSD as well as that feature anywhere itself holds to reasonable performance standards. This means testing out the limitations of changing the default number of associated objects per visualizations and any other new UI or visualizations that is part of the feature anywhere project.

[ohltyler]

We should also test some bulk deletion scenarios. Ref: #4059 (comment)

[kavilla]

As called out: #3981

We will target this for v2.next for prioritization on Node upgrade.

[joshuarrrr]

@ohltyler @amitgalitz This is tagged for 2.9 release - is there any remaining work to be done here or updates on progress? Will there be any PRs to review?

[ohltyler]

This has been in progress - I'll post some comprehensive results here when completed. The initial testing showed some clear bottlenecks and led to 2 big improvements (one included in 2.9)

• Add resource ID filtering in fetch augment-vis obj queries #4608
• Optimize augment-vis saved obj searching by adding arg to saved obj client #4595

[ohltyler]

Results:

[ohltyler]

Conclusions:

• Overall association count has impact on loading dashboard. On 3-node cluster, 32s increase when processing 0 vs. 1000 associations
• Overall association count has impact on loading view events flyout. On 3-node cluster, 25s increase when processing 1 vs. 100 associations
• Loading time for view events flyout scales linearly. This make sense, since each visualization rendered in the flyout is executed sequentially, and rendered all at once when all of them have been created / render workflow completed. An improvement can be done to have these done asynchronously similar to a dashboard.
• No load issues in plugin flyouts regardless of overall association count
• Event count & time range had little to no effect. For example, rendering a vis with 500 events had the identical load times compared to rendering the same vis with 0 events. This shows that event fetching / event processing / bucketing / rendering of event datapoints scales fine. It makes sense since there is a fixed number of datapoints shown on the vega charts.
• Vis count did not have a huge effect. Since visualizations are rendered async on a dashboard, the overall count didn't have drastic impact. There was a noticeable effect however, as the vis count increased to something like 20+ visualizations. For example, it took roughly double the time for a dashboard with 20 visualizations (each with 50 associations) to load (39s), than for a dashboard with 1 visualization with 50 associations (18s).