You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I found that get_full_data was unusably slow for datasets >1m rows.
I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.
I think a better implementation would be to use a server-side cursor and use fetchmany to fetch size number of rows at a time. See Psycopg docs for more.
The text was updated successfully, but these errors were encountered:
I found that
get_full_data
was unusably slow for datasets >1m rows.I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.
I think a better implementation would be to use a server-side cursor and use
fetchmany
to fetchsize
number of rows at a time.See Psycopg docs for more.
The text was updated successfully, but these errors were encountered: