[bug] [feature request] No results when database is huge - introduce results paging

[mgeeky]

Hi there Dirk!

Thank you for a splendid tool! :-)

I've just collected a rather big database (2,5GB) containing more than 100k users, 500k groups, 260k devices etc.
The gather phase went just fine, but when I attempted to review results from GUI - no records could be rendered.

Whenever frontend app launches Ajax request, the server is unable to serialize JSON in a timely manner, making frontend wait extremely long for the response. This results in GUI displaying no records until server responds.

In my situation, the server was unable to come up with a response in 30 minutes.

Maybe there could be a paging functionality added with frontend API calling out to specific results-pages as a workaround?

Leaving this as a food for thought! :-)

Regards,
Mariusz.

[dirkjanm]

Hey,

I've run into this as well on large tenants. The problem here is that for all tenants except those extremely large ones it works fine to just load all the data and use in-memory filtering.
If doing paging on the backend that would need to include searching and filtering as well. Which essentially means I have to rebuild a large part of the frontend and backend just to accomodate those larger tenants.
This may be something I will do at some point, but at the moment it's not a high priority.

[mgeeky]

Yes, I agree - these changes would entail substantial upgrades to both frontend and backend. Yet I believe they would offer worth ROI while working in larger tenants. Currently I'm pretty much unable to efficiently explore data collected - and I'm yet to work with more of that large tenants in upcoming engagements.

Anyway, I'm keeping my fingers crossed for you to find time to address this one! :-)

Regards,
M.

[rgmz]

If doing paging on the backend that would need to include searching and filtering as well. Which essentially means I have to rebuild a large part of the frontend and backend just to accomodate those larger tenants.
This may be something I will do at some point, but at the moment it's not a high priority.

@dirkjanm I've created a POC in #94 (only pagination, sorting, and filtering for /api/users). Have a look and let me know your thoughts.

[rgmz]

Hey @dirkjanm, let me know if there's interest in fleshing out my proposed fix. If not, I will close the PR.