[EPIC] Performance: SQLite backed cache

[richard-cox]

• This issues covers the integration of server-side pagination with the dashboard
• This will come in a number of iterations, each will be listed here when agreed on
• Performance: Server-side pagination - Alpha #9478
• Investigate places where pagination is not supported #9499
• Performance: Server-side pagination - First, limited release #9545
• Restrict / Remove usages of findAll for config maps, secrets and pods #9964
• Server-side Pagination: Rancher Backups: Credential Secret does not show values #11507
• Server-side Pagination: Rancher UI becomes unresponsive when the user navigates to the Performance page and then to the cluster dashboard while server-side pag enabled #11495
• Performance: Rancher UI becomes unresponsive when user navigates to Performance page and then to Nodes page #11496

2.9

• Performance: Server-side pagination setting is enabled when user hits cancel option #11497
  • Ensure Performance --> Pagination setting only enables post compatible check #11505

2.11 (WIP)

• Bugs
  • Server-side Pagination: When a user clicks on the Refresh button on one page and navigates to another button does not appear #11494
  • Server-side Pagination: Sorting: Cluster Dash: Newly created resource populates at bottom of list; sort order preference ignored #11498
  • Server-side Pagination: Pods list not updated after resource deleted #11499
  • Switching between server-side paginated lists incorrectly briefly shows 'no row's message before rows shown #12748
• General
  • Performance: Server-side pagination: Document regressions following Vai supported changes #12975
  • Performance: Server-side pagination: Developer Docs #11207
• Flux
  • Compare / Record filtering requirements from shell/components/SortableTable/filtering.js, discuss with backend
  • Investigate impact of displaying model's warped state but filtering/sorting via native metadata.state.name
  • TBD - Investigate / Validate new resource update process (on any resource change fetch current page). Determine
  • if we need a disable / updates option in lists that use this feature
  • Discuss/investigate dynamic application of server-side pagination #13044
• Features
  • Performance: Server-side pagination: home page cluster list #10283
    • Ensure home page cluster lists handles many many clusters #9494
    • Add server-side pagination to home page cluster list #11663
    • Update process to remove stale namespaces from namespace filter #12864
    • Ensure RKE1 clusters can be searched/filtered by name #12890
  • Wire in new pagination resources.change event #12734
  • Performance: Server-side pagination - All Cluster Explorer primary level lists #9546
    • Create new project/namespace experience given server-side pagination #11674
    • Add server-side pagination to cluster explorer cluster and project member lists #12563
  • Performance: Server-side pagination - Remaining cluster explorer lists #9547
  • Application of selectors locally scales badly #10417 (can be split up as per remove usages of findAlls)
  • Restrict / Remove usages of findAll for all cluster explorer related resources #9965
    • Use new pagination label select to find containers in Logging - Editing a Flow - Matches #12559
    • Remove usages of $fetchType for secondary types that support pagination (aka individual workload lists) #12555
    • Cluster Explorer: Restrict / Remove usages of findAll: traditional types #13045
      • Possible performance degradation at edit configmap #9453
  • Performance: Server-side pagination - Cluster management lists & detail pages #9548
  • Restrict / Remove usages of findAll for all cluster management related resources #9966
  • Performance: Server-side pagination - Remaining lists & detail pages #9549
  • Performance: Server-side pagination - Fully implemented #9550
  • Mark some obsolete performance improvement features as deprecated  #12873
• [EPIC] QA for SQLite backed cache / Vai / Server-side pagination #13043

2.12 (wip)

• Replace Workloads page / list with an overview page #11513
• Investigate how the catalog store can use server-side pagination to improve performance #12557
• Remove obsolete performance features #12556
  • Note quite pagination related, but might impact above, we should also pick a web worker (simple / advanced) OR remove entirely (if we're not processing 100s of messages a second we might not need to process in a separate thread)
• Fleet
• [ENHANCEMENT] Improve Harvester UI Extension Performance (placeholder) harvester/harvester#7342
• KubeWarden
• Elemental
• Other Rancher Extensions
• Implement more complex server-side pagination sorting #12815
  • Restrict / Remove usages of findAll for all cluster explorer related resources #9965
    • Cluster Explorer: Restrict / Remove usages of findAll: rbac types #13046
    • Cluster Explorer: Restrict / Remove usages of findAll: cluster management types #13047
    • Cluster Explorer: Restrict / Remove usages of findAll: norman #13048

Backend Issues (not complete list)

• [EPIC] SQLite backed cache in the Steve API - iteration 1 rancher#45635
• [EPIC] SQLite backed cache in the Steve API - iteration 2 rancher#46348
• SQLite backed cache in the Steve API - small things part 3 rancher#47825
• Specifically
  • sort/filter on all labels plus label selectors support - SQLite backed cache: support sorting and filtering on arbitrary labels rancher#46333
  • updates to lists given websockets - SQLite backed cache: smarter WebSockets updates for paginated resources rancher#40773
  • ad-hoc requests (as required) to sort/filter on resource properties
    • SQLite backed cache: Request for additional indexed fields rancher#46642
    • SQLite backed cache: Request for additional indexed fields #2 rancher#48011
    • SQLite backed cache: Request for additional indexed fields #3 rancher#48103
  • feature blocking, but not ui - SQLite backed cache: support listing of unwatchable resources rancher#46768
  • SQLite backed cache: Support sorting mgmt clusters on value in a specific condition rancher#48092

Related

• Side Menu Cluster List performance badly at scale #11993
• Cluster List filter performs badly at scale #11994
• Home Page Cluster List Scales Badly #11995

This issue requires a QA template.

[richard-cox]

Some of the backend changes have already been implemented in 2.7.2

• SURE-5387 - [RFE] Steve should support pagination with sorting and filtering - Step 1 rancher#38427
  • Some docs on the new UI are at https://github.com/rancher/steve#readme, https://github.com/rancher/rancher/tree/release/v2.7/tests/v2/integration/steveapi#readme

Anything marked as Development note are implementation details for this issue and are candidates to spin out

Some are incoming in 2.7 Q2

• SURE-5698 - [RFE] Add more advanced filtering options to steve rancher#40141
  • Adds required searching
    • Development note - only single search term is supported. searching by multiple terms locally seems broken anyway List search with multiple terms does not work as expected #8042
  • Filtering by one or more projects or namespaces
    • Development note - NOT is supported, aka filter by NOT list of system namespaces (i.e. user namespaces)
    • Development note - a maximum of 50 individual namespaces are expected. We need to handle this in UI

Some that we would like for 2.7 Q2 but are not blocking (in order of importance)

• SURE-5733 - [RFE] Steve pagination: aggregated fields rancher#40771
  • Filtering/Sorting by aggregated fields (for instance Deployment Pod restarts)
  • Without this adoption is limited (feature can only be used in lists that don't have aggregated fields
• SURE-5734 - SQLite backed cache: smarter WebSockets updates for paginated resources rancher#40773
  • Socket updates for paginated lists
  • Development note - See comments in 5734 for details on how we will mitigate it (receive all in socket but only persist to store the entries within the page)
• SURE-5735 / SURE-5897 - [RFE] Virtual Resource Type to contain Multiple Resources rancher#40772
  • Pagination of virtual resource (for instance Workloads)
  • Development note - If this is not delivered we would need to hide the Workloads list. We could have an overview page instead

In addition we need to consider more advanced ways to search

• Sort ip addresses in a more numerical way (need to also consider ip6)
  • e.g. xx.xx.17.196, xx.xx.17.2, xx.xx.17.204, xx.xx.17.24, xx.xx.17.243 and should be sorted numerically-octet-wise.
• Sort by duration
  • e.g. 7d4h, 6m56s, 23h should be 6m56s, 23h, 7d4h
• There's some more examples over at Wrong sorting of various views and columns in Rancher UI #9782, but the remaining ones are generally around properties that have been computed client side and therefore can't be supported using server side pagination

[richard-cox]

The backend will not support sorting/filtering on 'calculated' fields e.g. current model getters. For the first iteration we should concentrate on badly performing lists that don't need them, this does include all workload types and secrets.

Possible Flow

• user changes pagination based setting
• list persists them in the cluster store
• list shows loading / in progress indicator?
  • tracked in component or store? what happens if long running http request, user leaves, user returns?
• list triggers a cluster/request (via resource-fetch)
• cluster/request uses store settings to make http request (ignores values in vuex store)
  • this includes namespace/project filters
• list receives page
• list cancels loading / in progress indicator

Question

• Do we reset these when a user returns to a previous list? What happens if the page has changed and now has no entries?
• How do incremental loading and manual refresh tie in?
• How do we work with / around lack of restriction to subscriptions (i.e. we still need to subscribe for all resource change events)
  • What happens if a resources changes that would alter page?

[clayrisser]

I wanted to say this 2 years ago, but I decided to wait and give Rancher time to improve. I can't help but notice this issue is a year old. Still, in 2024 Rancher's performance makes the product completely unusable, while alternative tools are fast and snappy.

I quit using Rancher because it's soooo slow. It's hogging 4 GB memory and 4 CPUs on a brand new cluster. It's been doing this ever since rancher 2.6 and there seems to be no improvements.

I don't understand how alternative tools can be so fast, while the rancher dashboard just freezes all the time because the rancher server CPU and Memory get maxed out. Makes no sense to me.

1 user, 1 cluster, 4 gb ram, 4 cpus and it's still not enough, and it's been this way for years. It's the one issue that's forced me to look for alternatives.

The rancher product feature set is great, but I really wish I could say the same about its performance.

[richard-cox]

@clayrisser This is a year old, and I can empathise with the frustration you've experienced. We are very actively working on a solution though (see linked issues and their prs for UI progress). Unfortunately API side hit a road block which has delayed things a bit. I'm happy to say though that we are targeting a tech preview of the feature in 2.9.0.

In addition 2.9.0 contains many UI performance improvements that assist both memory and CPU usage. It's worth giving that a try when released.

[clayrisser]

I will try again with 2.9.0, but if it requires running a large server, (more than 4 CPU) it's just not worth it.

I'm going to assume rancher works well for very large deployments that have resources to throw at it, but small deployments can't justify too many resources for it.

[chad-barensfeld-exa]

Please add pagination to the "new" UI. It's unusable when importing hundreds of clusters. We have to use the old UI which still has pagination. /g/clusters

[richard-cox]

@chad-barensfeld-exa Server-side pagination is a large project that we are currently targeting in and enabled by default in 2.11. In parallel we have been making a number of other performance improvements which have received a lot of positive feedback from the community and SUSE customers, including those using Ranchers with a large number of clusters.

Unfortunately the old UI is not supported and will soon be unavailable. It also did not use server-side pagination (instead normally only showed a very restricted set of resources).

To help us with your specific performance issue would you be able to open a new github issue and provide as much detail as possible on what and where the performance is problematic, such as

• Does the blue spinner when reloading a page take very long time, or never stops? Does that happen always, or only some times?
• Does a page load slowly when the user first navigates to the UI?
• Does it take a long time to go from page to page?
• Does a list of resources take a long time to show any rows?
• Does the UI become unresponsive, clicks don't do anything?

Add concrete examples, eg.

• "Viewing a cluster's config from the Cluster Management page was just sitting on Loading"
• "Closing Rancher and opening a new Rancher instance in Chrome showed the blue spinning circle"
• "After reloading the page (CTRL+R) in about 3 minutes the UI would either eventually load or show a Fail Whale error (~50% of cases)"

If there's lots of places where performance isn't great, focus on one process they found most likely to see issues, or where the issues are worst.

Do all network requests take a long time to complete, or is there one or two in particular that take time?