Skip to content

Commit

Permalink
Simplify the keyset API:
Browse files Browse the repository at this point in the history 
- Deprecate the :after_latest variable in favour of :filter_newest
- Add the keyset argument to the :filter_newest lambda
- Rename protected after_latest_query > filter_newest_query
  • Loading branch information
@ddnexus
ddnexus committed Nov 5, 2024
1 parent 5818b57 commit ff1c1cf
Show file tree
Hide file tree
Showing 7 changed files with 40 additions and 36 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ If you upgrade from version `< 9.0.0` see the following:

## Deprecations

- None
- `:after_latest` keyset variable: use `:filter_newest`
<hr>

## Version 9.1.1
Expand Down
36 changes: 18 additions & 18 deletions docs/api/keyset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@ less convenient for UIs.

### Glossary

| Term | Description |
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Term | Description |
|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `offset pagination` | Technique to fetch each page by changing the `offset` from the collection start.<br/>It requires two queries per page (or one if [countless](/docs/api/countless.md)): it's slow toward the end of big tables.<br/>It can be used for a rich frontend: it's the regular pagy pagination. |
| `keyset pagination` | Technique to fetch the next page starting from the `latest` fetched record in an `uniquely ordered` collection.<br/>It requires only one query per page: it's very fast regardless the table size and position (if properly indexed). It has a very limited usage in frontend. |
| `uniquely ordered` | When the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primary `key` for the ordered table, but dynamically based on the `keyset` columns. |
| `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. |
| `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. |
| `latest` | The hash of `keyset` attributes of the `latest` fetched record (from the latest page). Pagy decodes it from the `:page` variable and uses it to filter out the records already fetched. |
| `next` | The next `page`, i.e. the encoded reference to the last record of the **current page**. |
| `page` | The current `page`, i.e. the encoded reference to the `latest` record of the **latest page**. |
| `keyset pagination` | Technique to fetch the next page starting from the `latest` fetched record in an `uniquely ordered` collection.<br/>It requires only one query per page: it's very fast regardless the table size and position (if properly indexed). It has a very limited usage in frontend. |
| `uniquely ordered` | When the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primary `key` for the ordered table, but dynamically based on the `keyset` columns. |
| `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. |
| `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. |
| `latest` | The hash of `keyset` attributes of the `latest` fetched record (from the latest page). Pagy decodes it from the `:page` variable and uses it to filter the newest records. |
| `next` | The next `page`, i.e. the encoded reference to the last record of the **current page**. |
| `page` | The current `page`, i.e. the encoded reference to the `latest` record of the **latest page**. |

### Keyset or Offset pagination?

Expand Down Expand Up @@ -116,8 +116,8 @@ If you need a specific order:
- You pass an `uniquely ordered` `set` and `Pagy::Keyset` queries the page of records.
- It keeps track of the `latest` fetched record by encoding its `keyset` attributes into the `page` query string param of the
`next` URL.
- At each request, the `:page` is decoded and used to prepare a `when` clause that filters out the records already fetched, and
the `:limit` of requested records is pulled.
- At each request, the `:page` is decoded and used to prepare a `when` clause that filters the newest records, and
the `:limit` of records is pulled.
- You know that you reached the end of the collection when `pagy.next.nil?`.

## ORMs
Expand Down Expand Up @@ -166,19 +166,19 @@ Boolean variable that enables the tuple comparison e.g. `(brand, id) > (:brand,
order, hence it's ignored for mixed order. Check how your DB supports it (your `keyset` should include only `NOT NULL` columns).
Default `nil`.

==- `:after_latest`
==- `:filter_newest`

**Use this for DB-specific extra optimizations, if you know what you are doing.**

If the `:after_latest` variable is set to a lambda, pagy will call it with the `set` and `latest` arguments instead of
using its auto-generated query to filter out the records after the `latest`. It must return the filtered set. For example:
If the `:filter_newest` variable is set to a lambda, pagy will call it with the `set`, `latest` and `keyset` arguments instead of
using its auto-generated query to filter the newest records (after the `latest`). It must return the filtered set. For example:

```ruby
after_latest = lambda do |set, latest|
set.where(my_optimized_query, **latest)
filter_newest = lambda do |set, latest, keyset|
set.where(my_optimized_query(keyset), **latest)
end

Pagy::Keyset(set, after_latest:)
Pagy::Keyset(set, filter_newest:)
```

==- `:typecast_latest`
Expand Down Expand Up @@ -243,7 +243,7 @@ They may have been stored as strings formatted differently than the default form
or similar tool to confirm.
- Consider using the same direction order, enabling the `:tuple_comparison`, and changing type of index (different DBs may behave
differently)
- Consider using your custom optimized `when` query with the [:after_latest](#after-latest) variable
- Consider using your custom optimized `when` query with the [:filter_newest](#filter-newest) variable
!!!

===
14 changes: 9 additions & 5 deletions gem/lib/pagy/keyset.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,15 @@ def next
@next ||= B64.urlsafe_encode(latest_from(@records.last).to_json)
end

# Retrieve the array of records for the current page
# Fetch the array of records for the current page
def records
@records ||= begin
@set = apply_select if select?
@set = @vars[:after_latest]&.(@set, @latest) || after_latest if @latest
@set = apply_select if select?
if @latest
@set = @vars[:after_latest]&.(@set, @latest) || # deprecated
@vars[:filter_newest]&.(@set, @latest, @keyset) ||
filter_newest
end
records = @set.limit(@limit + 1).to_a
@more = records.size > @limit && !records.pop.nil?
records
Expand All @@ -69,8 +73,8 @@ def records

protected

# Prepare the literal query to filter out the already fetched records
def after_latest_query
# Prepare the literal query to filter the newest records
def filter_newest_query
operator = { asc: '>', desc: '<' }
directions = @keyset.values
if @vars[:tuple_comparison] && (directions.all?(:asc) || directions.all?(:desc))
Expand Down
4 changes: 2 additions & 2 deletions gem/lib/pagy/keyset/active_record.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ def extract_keyset
end
end

# Filter out the already retrieved records
def after_latest = @set.where(after_latest_query, **@latest)
# Filter the newest records
def filter_newest = @set.where(filter_newest_query, **@latest)

# Append the missing keyset keys if the set is restricted by select
def apply_select
Expand Down
4 changes: 2 additions & 2 deletions gem/lib/pagy/keyset/sequel.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ def extract_keyset
end
end

# Filter out the already retrieved records
def after_latest = @set.where(::Sequel.lit(after_latest_query, **@latest))
# Filter the newest records
def filter_newest = @set.where(::Sequel.lit(filter_newest_query, **@latest))

# Append the missing keyset keys if the set is restricted by select
def apply_select
Expand Down
2 changes: 1 addition & 1 deletion test/pagy/countless_test.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@
_(pagy.prev).must_equal 2
_(pagy.next).must_equal 1
end
it 'raises exception with no retrieved records and page > 1' do
it 'raises exception with no fetched records and page > 1' do
_ { Pagy::Countless.new(page: 2, overflow: :exception).finalize(0) }.must_raise Pagy::OverflowError
end
end
Expand Down
14 changes: 7 additions & 7 deletions test/pagy/keyset_test.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,16 @@
_ = pagy.records
_(pagy.latest).must_equal({id: 10})
end
it 'uses :after_latest' do
after_latest = if model == Pet
->(set, latest) { set.where('id > :id', **latest) }
else
->(set, latest) { set.where(Sequel.lit('id > :id', **latest)) }
end
it 'uses :filter_newest' do
filter_newest = if model == Pet
->(set, latest, _keyset) { set.where('id > :id', **latest) }
else
->(set, latest, _keyset) { set.where(Sequel.lit('id > :id', **latest)) }
end
pagy = Pagy::Keyset.new(model.order(:id),
page: "eyJpZCI6MTB9",
limit: 10,
after_latest: after_latest)
filter_newest:)
records = pagy.records
_(records.first.id).must_equal(11)
end
Expand Down

0 comments on commit ff1c1cf

Please sign in to comment.