Performance issues with version 5.0 (example videos)

[AEgit]

JabRef 3.8.2
windows 10 10.0 amd64
Java 1.8.0_211

VS

JabRef 5.0-dev--snapshot--2019-06-20--master--6265e2460
Windows 10 10.0 amd64
Java 1.8.0_211

As announced previously (#4430 (comment)), here I add some videos that showcase the performance difference between JabRef 3.8.2 vs. JabRef 5.0 when working with a large database (> 18,000 entries) and several thousand static groups.

These issues have been reported previously, but I thought it would be better to create a new bug report instead of adding the videos to just one of these previous reports (they should apply to all of them, I reckon). Previous reports of issues with the performance include (among others), the following:

#4430
#4756
#4526

Note, that I had to run these tests on a virtual machine, since I do not have a full license for YourKit Java Profiler and had to use a test license (since the test license I had on my actual machine is no longer valid). So the performance on an actual machine should be slightly better (the virtual machine is running on a SSD, and I assigned two cores of a Core i7 Hexacore and 8 GB of 32 GB of RAM to the virtual machine), but the relative performance difference between version 3.8.2 and version 5.0 should be comparable. In fact, my impression is that this setup favours version 5.0 because 3.8.2 appears slower in the virtual machine than in a non-virtualized setting. Version 5.0 on the other hand, is equally slow in a virtual machine and when running on the actual hardware.

Note, that I use the same database when comparing the two versions. The only difference is, that the database in version 5.0 has been saved using version 5.0 (while the other one is still using a database in 3.8.2 format). This allows for better comparison, since version 5.0 stores groups differently.

1. Comparison of first start of JabRef:
   3.8.2:
   https://app.box.com/s/7agrhreojpk85itl9dvsz7mx8ueaydcl
   5.0:
   https://app.box.com/s/79dqowjer214qdi0wiceh03x9g4iuexi

Note, that version 5.0 sometimes seems to have crashed. Furthermore, I did not immediately click onto 3.8.2 when it was actually already ready.

2. Comparison of JabRef search feature:
   3.8.2:
   https://app.box.com/s/fgr84sgpsz6dscbnku91m5o2hhbf6rwt
   5.0:
   https://app.box.com/s/fxwcska8dm5rlilv6a9mqswelkrqkh0t

3. Comparison of working with JabRef groups:
   3.8.2:
   https://app.box.com/s/fc5ku29f39o9r7h86hj309tao4uifo8o
   5.0:
   https://app.box.com/s/cubdi2vvmzjnj5i7rtn57lfxp24kvzb9

Note, how JabRef freezes several times in version 5.0 when using certain groups features (e.g., assigning an item to a group, assigning a group as a new subgroup, ....). Sometimes it appears as if nothing is happening in the video, but it is actually just JabRef freezing from time to time.
Note, the big differences in the time that JabRef spents on some actions (see YourKit Java Profiler). For certain actions, JabRef 5.0 will take much much longer than 3.8.2 (which translates into higher CPU demands).

[buhtz]

#5156

[tobiasdiez]

@AEgit the most recent developement version now includes the latest performance improvements. Could you please check what still needs improvement. Thanks!

[AEgit]

@tobiasdiez Due to the issues with the current Windows installer (see #5580 (comment)) and since the snap package on Linux also does not seem to work yet (see #5417) I cannot test the current developer version, I am afraid.

[LyzardKing]

The snap can be installed from https://builds.jabref.org/master/
It won't auto update and you need to install with --dangerous (since it's not from the store)
But for testing it can be done.

[AEgit]

JabRef 5.0.0-dev--2019-11-17----151a216a2
Windows 10 10.0 amd64
Java 13.0.1

@tobiasdiez Tried the current dev version. This one is much improved in terms of performance. Switching between different groups and articles is now much smoother. Well done!

The following performance issues remain:

1. Assigning an item to a group has a heavy impact on the CPU. Now, if you assign just a single item, this appears to be no major isse for a user with a powerful machine. I have a 6-core i7 and while the assignment of a single item brings the CPU usage to nearly 50% for a few seconds, the item appears (?) to be assigned immediately to this group (this might already be a problem, though, for people with weaker machines).
   What is a problem, however, is when you try to assign multiple items simultaneously to a new group. The CPU usage shoots up and (depending on how many items you try to assign to the group) JabRef freezes for some time. The CPU usage also stays high for a protracted time period: I am not sure, whether the items have already been assigned to the group as indicated by the item counter since the CPU usage stays high even after that for quite some time.
   To recreate this problem, try the following:
   a) Select multiple items in the main table
   b) Drag'n'drop them onto a group they had not be assigned to
   c) Observe the mentioned issues
2. The same problem arises when you try to assign groups as subgroups of other groups (in fact, I think this feature might be bugged, but I have to test this more thoroughly). If you assign just a single group, things appear fine. But if you assign multiple groups simultaneously as subgroups to a new group (again, by using the drag'n'drop feature - but this time in the groups panel), you end up with a high CPU usage over a protracted period of time.
3. When you select multiple groups in the groups panel simultaneously (using left click on the starting group and left click + SHIFT on the ending group), there is a noticeable delay until all groups are selected and the CPU usage shoots up again.
4. There is a noticeable delay (a few seconds) when you select the "All entries" group until the respective items are shown in the main table. I reckon this is directly dependent on the number of entries assigned to a certain group, because this problem also appears to arise, when groups with a high entry count are selected (currently my "All entries" group contains nearly 20,000 items).

These are the performance issues I have observed so far. Nevertheless, again a big thanks to @tobiasdiez for the major performance improvements that the current version offers now - this is definitely a major step into the right direction.

[tobiasdiez]

Thanks for the feedback. I'll try to have a look at the remaining performance issues in the next days.

[AEgit]

JabRef 5.0.0-dev--2019-11-24----8780a5632
Windows 10 10.0 amd64
Java 13.0.1

Note also, that it appears, as if the search has become slower again. Not as bad as it was in the past (#2852), but it seems, that the latest versions have seen a performance decrease there as well. Entering a search term using a large database results in lagged input (similar to a low frame rate in a video game).