Slow/freezing sometimes with large files

[zacaj]

Not sure how much this is a bug or not, but once the log files get to 10+MB the plugin starts struggling, and sometimes will just freeze completely. Not sure if it's even trying to do any sort of streaming/large large file support or rendering only visible table rows/other standard optimizations, or if it's just loading the whole thing, but it'd be cool if it worked with 20/50/100MB files; if it's not doing any of the low hanging fruit it may not be too bad to add in. Alternately, if it should work for arbitrarily large files, maybe there's actually just a bug I'm hitting somewhere, although idk how to diagnose that to give you any more info

[thedangander]

I've seen the same thing. It's because the extension attempts to parse the whole file in one go and generate the view. I'm planning on adding large file support with some virtualised paging or chunking. I'm working on this issue this week already, so should have something in place soon

[thedangander]

Really, I should be parsing into a proper datastore like sqlite first, then driving the view from that, but it's more work than I wanted to do right now. But it would be the right move - scales better, separates the UI and storage properly, and I can leverage proper indexes for performance, so I should do that really.

[zacaj]

Do you think sqlite would be fast enough to populate in real time? seems like it'd work good for performance once loaded, but...

[thedangander]

Well there's a few options I think. Virtual scrolling to only render what's in view is one. At the moment, parsing blocks the page render too, so I could move the file parsing to a web worker. A sqlite store would make query performance better, but could add to the startup times.
I'm going to try a couple of options and see what works. Virtual scrolling is probably the quickest win.

[zacaj]

Yeah the freeze for a while sometimes just when switching back to the tab is quite annoying