Osctrl-admin UI: Performance Problems with Large Node Sets (10,000+ nodes)

[zhuoyuan-liu]

The current implementation has significant performance issues when dealing with large node sets. Two critical bottlenecks have been identified:

1. No Pagination in Node Retrieval
   • All nodes are retrieved in one database query
   • This loads the entire result set into memory at once
   • As the number of nodes grows, this single query becomes increasingly slow
   • No limit or offset mechanism exists for large result sets
2. Full Node Data Transfer
   • The entire node dataset is processed and sent to the client as a single JSON payload
   • For large environments, this creates massive JSON responses
   • Both server processing time and network transfer time increase linearly with node count
   • The client browser must then parse and render this large dataset

Suggested Solutions

1. Implement Server-side Pagination:
   • Modify GetByEnv to support pagination parameters (limit, offset)
   • Add pagination controls to the UI
   • Only retrieve and display a subset of nodes at any time
2. Add Server-side DataTables Processing:
   • Convert existing DataTables implementation to use server-side processing
   • Leverage DataTables' built-in AJAX pagination capabilities
   • Client requests only the data needed for the current page view

During our test, most of the time was spent on rendering the JSON in the browser. It takes around 500ms to download the JSON file and 10+ seconds to load the JSON.

[javuto]

I will improve the performance of the UI, probably with a mix of pagination and caching. Thanks for opening the issue and all the info!