Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[12.x] Fix chunked queries not honoring user-defined limits and offsets #52093

Merged
merged 17 commits into from
Aug 1, 2024

Conversation

tonysm
Copy link
Contributor

@tonysm tonysm commented Jul 11, 2024

Changelog

  • ADDED: Adds new getter methods for offset and limit to the query builder (getOffset() and getLimit())
  • FIXED: Honor user-defined limits and offsets in chunk and chunkById queries instead of silently overriding it
  • FIXED: Any user-defined offset would cause the chunkById to silently skip the offset on every iteration (since it doesn't override it)

Currently, the chunk method ignores any user-defined limits and offsets. The chunkById method ignores user-defined limits, but gets tripped up by user-defined offsets (it skips the offset amount of entries on every batch iteration).

This PR ensures that user-defined limits and offsets are honored by chunked methods. For instance:

$survey->emails()
  ->orderBy('id', 'asc')
  ->offset(2)
  ->limit(3)
  ->chunk(2, function (Collection $users, $page) {
    // ...
  });

Given there are 7 entries in the database, this query would result in the following chunks distribution:

Entry ID Status Chunk Page
1 skipped --
2 skipped --
3 included 1
4 included 1
5 included 2
6 skipped --
7 skipped --

Warning

Anyone currently using chunk or chunkById with limits and offsets (and not testing the behavior, I guess) could be affected by this fix.

Context

I noticed chunked queries were ignoring user-defined limits while I was working on a legacy application that had a job doing something like this:

$survey->emails()
  ->orderBy('id', 'desc')
  ->limit($plan->remainingSurveysFor($survey))
  ->each(function ($email) {
    dispatch(new SendSurvey($email));
  });

As I was covering this with a test like this:

/** @test */
public function only_sends_plan_limit()
{
  Queue::fake();

  $survey = Survey::factory()
    ->has(Email::factory()->times(10))
    ->create();

  (new BroadcastSurveys($survey))->handle((new FakePlan)->withRemainingSurveys($limit = 5));

  Queue::assertPushed(SendSurvey::class, $limit);
}

I noticed the test was failing saying that 10 jobs were pushed instead of the expected 5.

I decided to dig in and found out the each method calls chunk behind the scenes (here), which calls the forPage() helper, which overrides offset and limits for the query.

After some searching, it looks like this was a known issue:

A fix hasn't been attempted in a while, so I figured it was about time for another one.

I've tested this patch in the application where I discovered it, and it works.

As I was looking for references in other frameworks, I found out that Rails' ActiveRecord also honors limits on their batched queries:

Limits are honored, and if present there is no requirement for the batch size: it can be less than, equal to, or greater than the limit.

*
* @return mixed
*/
public function getLimit()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not

/**
 * Get the current "limit" value from the query or null if not set.
 */
public function getLimit() ?int

analog for all NEW function declarations. Dockblocks are not needed if you write the function definition with return type and typed params.

Copy link
Contributor Author

@tonysm tonysm Jul 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just matching the same style that is there right now

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not in compliance with what laravel stated that starting with laravel 10, that they will implement typed declarations...
I see many new functions merged with old style codding...

@taylorotwell taylorotwell merged commit c1a21ab into laravel:master Aug 1, 2024
29 checks passed
@taylorotwell
Copy link
Member

Thanks!

@tonysm tonysm deleted the tm/limited-chunks branch August 2, 2024 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants