Skip to content

[5.1] Fix pagination for group by and having statements #10632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[5.1] Fix pagination for group by and having statements #10632

wants to merge 1 commit into from

Conversation

acasar
Copy link
Contributor

@acasar acasar commented Oct 16, 2015

There are two problems that this PR is trying to address:

1.) length-aware pagination with "group by" statements is currently slow since counting of the results is done in PHP instead of in the query
2.) and it's currently not possible to paginate queries containing having statements

In these two cases, we can use a sub query to get the total number of rows:

select count(*) as aggregate from (<original query>) AS aggregate_table

By doing this, we can now paginate any query and it's all done in SQL so it should be a lot faster. If no "groups" or "havings" are defined, then we still use the normal pagination without a subquery. And since we don't need the "hack" for group by statements any more, I was able to clean up some related code as well and defer to the built-in count function in both cases.

This was tested in SQLite and MySQL. Any feedback about other supported grammars is welcome.

@GrahamCampbell GrahamCampbell changed the title Fix pagination for group by and having statements [5.1] Fix pagination for group by and having statements Oct 16, 2015
@GrahamCampbell
Copy link
Member

@taylorotwell
Copy link
Member

Problem with this. Say you have a 1,000,000 record user table. Instead of just running a query like select count(*) from users... now you are running select count(*) from (select * from users) as ...

You are selecting all 1,000,000 rows and loading them up, then counting them. That's not going to be performant.

@acasar
Copy link
Contributor Author

acasar commented Oct 16, 2015

@taylorotwell Well, until now we were doing the same, just loading them all in PHP and counting them there. I think this is still much better.

And once again - it doesn't change anything if you don't have "group by" or "having" statements - in these cases a normal select count(*) from users... is executed. This PR just takes care of some edge cases which didn't work before (having statements) and improves performance for "group by" statements.

@taylorotwell
Copy link
Member

No, that's not what we're doing right now. Here are the two queries generated from User::paginate() out of the box:

"select count(*) as aggregate from `users`"
"select * from `users` limit 15 offset 0"

@taylorotwell
Copy link
Member

You can see that we're just doing a count, we're not actually pulling back the entire users table.

@acasar
Copy link
Contributor Author

acasar commented Oct 16, 2015

And that still remains the same. Try adding a group by and see what happens now vs. after.

@acasar
Copy link
Contributor Author

acasar commented Oct 16, 2015

Again - for the case above, nothing has changed.

@acasar
Copy link
Contributor Author

acasar commented Oct 16, 2015

@taylorotwell I made some benchmarks on a table with 100,000 rows and I ran the following query 100 times:

EloquentTestUser::groupBy('email')->getQuery()->getCountForPagination()

The results show that with this PR it's about 20% faster.

Without PR: 12.6s, 12.9s
With PR: 10,7s, 10,6s

@tylercrompton
Copy link
Contributor

Which database did you use? Did you try others. The results are possibly very implementation-specific.

@coveralls
Copy link

Coverage Status

Changes Unknown when pulling 85f3352 on acasar:fix-pagination into ** on laravel:5.1**.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants