Attribute table loads features really slow when changing from "Show visible.." to "Show all"

[tudorbarascu]

Describe the bug

Changing the attribute table from Show Features Visible On Map to Show All Features is taking forever (approximately 40 features per second) for any layers (tested with Postgres and Shapefiles). If I open the attribute table directly to either Show All Features or Show Features Visible On Map it works instant. Only when switching from Visible to All the bug occurs.

How to Reproduce

1. Go to attribute table of a vector layer
2. Change to Show Features Visible On Map then to Show All Features
3. Stuck for eternity

QGIS and OS versions

Tested on 3.10.5 on Ubuntu Focal and Debian Bullseye but also with Master built from source on Debian Bullseye.

QGIS version	3.13.0-Master	QGIS code revision	45bead2
Compiled against Qt	5.12.5	Running against Qt	5.12.5
Compiled against GDAL/OGR	3.0.4	Running against GDAL/OGR	3.0.4
Compiled against GEOS	3.8.1-CAPI-1.13.3	Running against GEOS	3.8.1-CAPI-1.13.3
Compiled against SQLite	3.31.1	Running against SQLite	3.31.1
PostgreSQL Client Version	12.2 (Debian 12.2-4)	SpatiaLite Version	4.3.0a
QWT Version	6.1.4	QScintilla2 Version	2.11.2
Compiled against PROJ	7.0.0	Running against PROJ	Rel. 7.0.0, March 1st, 2020
OS Version	Debian GNU/Linux bullseye/sid
Active python plugins	pluginbuilder3; plugin_reloader; scriptrunner3; trackable_project_files; db_manager; processing

and

QGIS version	3.10.5-A Coruña	QGIS code revision	58395a2
Compiled against Qt	5.12.5	Running against Qt	5.12.5
Compiled against GDAL/OGR	3.0.4	Running against GDAL/OGR	3.0.4
Compiled against GEOS	3.8.1-CAPI-1.13.3	Running against GEOS	3.8.1-CAPI-1.13.3
Compiled against SQLite	3.31.1	Running against SQLite	3.31.1
PostgreSQL Client Version	12.2 (Debian 12.2-4)	SpatiaLite Version	4.3.0a
QWT Version	6.1.4	QScintilla2 Version	2.11.2
Compiled against PROJ	7.0.0	Running against PROJ	Rel. 7.0.0, March 1st, 2020
OS Version	Debian GNU/Linux bullseye/sid
Active python plugins	pluginbuilder3; qgepplugin; plugin_reloader; scriptrunner3; trackable_project_files; db_manager; processing

Additional context

[gioman]

How to Reproduce

1. Go to attribute table of a vector layer

2. Change to `Show Features Visible On Map` then to `Show All Features`

3. Stuck for eternity

@tudorbarascu confirmed here too (same setup). Seems a regression to me, agree?

[tudorbarascu]

That's for sure. Two weeks ago it wasn't like this for sure. I didn't had time to bisect it, will try to find.

[tudorbarascu]

I bisected up till here but I got stuck as it cannot build anymore at this commit, will investigate later:

web@thinkpad:~/dev/cpp/qgis$ git bisect log
git bisect start
# good: [b134c777f676292de4869e7306cb2be118678f6a] Merge pull request #35065 from elpaso/mssql-connections-api
git bisect good b134c777f676292de4869e7306cb2be118678f6a
# bad: [9c857ecb94dc35842234351c481896130b0551ba] Merge pull request #35503 from pblottiere/ut_type
git bisect bad 9c857ecb94dc35842234351c481896130b0551ba
# good: [afbe727c1a7637b99608b70e1d4449e537b9b933] Allow serialization of QgsProcessingFeatureSourceDefinition through QgsXmlUtils
git bisect good afbe727c1a7637b99608b70e1d4449e537b9b933
# bad: [2856e7d2a960c4bb00337540c9c7fe118bd79ac7] Somedays Travis is just calling out to be punched in the face...
git bisect bad 2856e7d2a960c4bb00337540c9c7fe118bd79ac7
# bad: [7248c433b39a86c2dba411c03a45197112fb2914] Merge pull request #35034 from signedav/attradd
git bisect bad 7248c433b39a86c2dba411c03a45197112fb2914
# good: [9b0db6e7905ac818316a4ec028a3ae2d83414bef] Merge pull request #35345 from m-kuhn/wrong_arg_count_in_server_error_messages
git bisect good 9b0db6e7905ac818316a4ec028a3ae2d83414bef

[tudorbarascu]

I've gone a commit up and realised git bisect is smarter than I thought.
So I've caught the bad commit. It's 00492ca

[gioman]

So I've caught the bad commit. It's 00492ca

@signedav ping

[signedav]

Thanks for pinging. That's not good. I'll have a look.

[gioman]

Thanks for pinging. That's not good. I'll have a look.

@signedav thanks much appreciated.

[signedav]

Hi @tudorbarascu
I have troubles to reproduce it. I agree that since this change it takes longer to load the features in the attribute list when switching back to Show All Features than before. But it's never stuck for eternity. Did you test it with small data sets as well? Can you see this progress bar?

I'll have a look on what makes it to take more time. But it would be helpful when I know if it's just stuck at your place because it took too long or if it really stuck for eternity (what means there's another problem).

[tudorbarascu]

@signedav Sorry! I meant is loading really slow, few features per second... and if you have thousands of features, you wait a really really long time (minutes/hours) for the attribute table to load. Retested now and for a table with a little over 3000 features it takes approximately 30 seconds (but it's a local shapefile). If I load the full attribute table normally it's instant. So the difference in performance is 30 seconds - instant. I can see the progress bar.

[signedav]

Okay thanks. That's what I can reproduce. I'll have a look.

[tudorbarascu]

Thank you!

[signedav]

Hi @tudorbarascu
Could you test my PRs for master and (different) backport branches fixing this issue?

[tudorbarascu]

@signedav Onto it, will comment in the pull requests as soon as I've built and tested. Thank you!