From ff1c1cf9b15e7e1e5b9201ac9892ff8b3ede4ca2 Mon Sep 17 00:00:00 2001 From: Domizio Demichelis Date: Mon, 4 Nov 2024 14:53:50 +0700 Subject: [PATCH] Simplify the keyset API: - Deprecate the :after_latest variable in favour of :filter_newest - Add the keyset argument to the :filter_newest lambda - Rename protected after_latest_query > filter_newest_query --- CHANGELOG.md | 2 +- docs/api/keyset.md | 36 ++++++++++++++-------------- gem/lib/pagy/keyset.rb | 14 +++++++---- gem/lib/pagy/keyset/active_record.rb | 4 ++-- gem/lib/pagy/keyset/sequel.rb | 4 ++-- test/pagy/countless_test.rb | 2 +- test/pagy/keyset_test.rb | 14 +++++------ 7 files changed, 40 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34e8fd517..2b5a5ff35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ If you upgrade from version `< 9.0.0` see the following: ## Deprecations -- None +- `:after_latest` keyset variable: use `:filter_newest`
## Version 9.1.1 diff --git a/docs/api/keyset.md b/docs/api/keyset.md index f3a3efcff..36d721289 100644 --- a/docs/api/keyset.md +++ b/docs/api/keyset.md @@ -34,16 +34,16 @@ less convenient for UIs. ### Glossary -| Term | Description | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Term | Description | +|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `offset pagination` | Technique to fetch each page by changing the `offset` from the collection start.
It requires two queries per page (or one if [countless](/docs/api/countless.md)): it's slow toward the end of big tables.
It can be used for a rich frontend: it's the regular pagy pagination. | -| `keyset pagination` | Technique to fetch the next page starting from the `latest` fetched record in an `uniquely ordered` collection.
It requires only one query per page: it's very fast regardless the table size and position (if properly indexed). It has a very limited usage in frontend. | -| `uniquely ordered` | When the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primary `key` for the ordered table, but dynamically based on the `keyset` columns. | -| `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. | -| `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. | -| `latest` | The hash of `keyset` attributes of the `latest` fetched record (from the latest page). Pagy decodes it from the `:page` variable and uses it to filter out the records already fetched. | -| `next` | The next `page`, i.e. the encoded reference to the last record of the **current page**. | -| `page` | The current `page`, i.e. the encoded reference to the `latest` record of the **latest page**. | +| `keyset pagination` | Technique to fetch the next page starting from the `latest` fetched record in an `uniquely ordered` collection.
It requires only one query per page: it's very fast regardless the table size and position (if properly indexed). It has a very limited usage in frontend. | +| `uniquely ordered` | When the concatenation of the values of the ordered columns is unique for each record. It is similar to a composite primary `key` for the ordered table, but dynamically based on the `keyset` columns. | +| `set` | The `uniquely ordered` `ActiveRecord::Relation` or `Sequel::Dataset` collection to paginate. | +| `keyset` | The hash of column/direction pairs. Pagy extracts it from the order of the `set`. | +| `latest` | The hash of `keyset` attributes of the `latest` fetched record (from the latest page). Pagy decodes it from the `:page` variable and uses it to filter the newest records. | +| `next` | The next `page`, i.e. the encoded reference to the last record of the **current page**. | +| `page` | The current `page`, i.e. the encoded reference to the `latest` record of the **latest page**. | ### Keyset or Offset pagination? @@ -116,8 +116,8 @@ If you need a specific order: - You pass an `uniquely ordered` `set` and `Pagy::Keyset` queries the page of records. - It keeps track of the `latest` fetched record by encoding its `keyset` attributes into the `page` query string param of the `next` URL. -- At each request, the `:page` is decoded and used to prepare a `when` clause that filters out the records already fetched, and - the `:limit` of requested records is pulled. +- At each request, the `:page` is decoded and used to prepare a `when` clause that filters the newest records, and + the `:limit` of records is pulled. - You know that you reached the end of the collection when `pagy.next.nil?`. ## ORMs @@ -166,19 +166,19 @@ Boolean variable that enables the tuple comparison e.g. `(brand, id) > (:brand, order, hence it's ignored for mixed order. Check how your DB supports it (your `keyset` should include only `NOT NULL` columns). Default `nil`. -==- `:after_latest` +==- `:filter_newest` **Use this for DB-specific extra optimizations, if you know what you are doing.** -If the `:after_latest` variable is set to a lambda, pagy will call it with the `set` and `latest` arguments instead of -using its auto-generated query to filter out the records after the `latest`. It must return the filtered set. For example: +If the `:filter_newest` variable is set to a lambda, pagy will call it with the `set`, `latest` and `keyset` arguments instead of +using its auto-generated query to filter the newest records (after the `latest`). It must return the filtered set. For example: ```ruby -after_latest = lambda do |set, latest| - set.where(my_optimized_query, **latest) +filter_newest = lambda do |set, latest, keyset| + set.where(my_optimized_query(keyset), **latest) end -Pagy::Keyset(set, after_latest:) +Pagy::Keyset(set, filter_newest:) ``` ==- `:typecast_latest` @@ -243,7 +243,7 @@ They may have been stored as strings formatted differently than the default form or similar tool to confirm. - Consider using the same direction order, enabling the `:tuple_comparison`, and changing type of index (different DBs may behave differently) -- Consider using your custom optimized `when` query with the [:after_latest](#after-latest) variable +- Consider using your custom optimized `when` query with the [:filter_newest](#filter-newest) variable !!! === diff --git a/gem/lib/pagy/keyset.rb b/gem/lib/pagy/keyset.rb index 4b26ca0b9..0b5c230ce 100644 --- a/gem/lib/pagy/keyset.rb +++ b/gem/lib/pagy/keyset.rb @@ -56,11 +56,15 @@ def next @next ||= B64.urlsafe_encode(latest_from(@records.last).to_json) end - # Retrieve the array of records for the current page + # Fetch the array of records for the current page def records @records ||= begin - @set = apply_select if select? - @set = @vars[:after_latest]&.(@set, @latest) || after_latest if @latest + @set = apply_select if select? + if @latest + @set = @vars[:after_latest]&.(@set, @latest) || # deprecated + @vars[:filter_newest]&.(@set, @latest, @keyset) || + filter_newest + end records = @set.limit(@limit + 1).to_a @more = records.size > @limit && !records.pop.nil? records @@ -69,8 +73,8 @@ def records protected - # Prepare the literal query to filter out the already fetched records - def after_latest_query + # Prepare the literal query to filter the newest records + def filter_newest_query operator = { asc: '>', desc: '<' } directions = @keyset.values if @vars[:tuple_comparison] && (directions.all?(:asc) || directions.all?(:desc)) diff --git a/gem/lib/pagy/keyset/active_record.rb b/gem/lib/pagy/keyset/active_record.rb index 8a2024d90..ad4a49195 100644 --- a/gem/lib/pagy/keyset/active_record.rb +++ b/gem/lib/pagy/keyset/active_record.rb @@ -17,8 +17,8 @@ def extract_keyset end end - # Filter out the already retrieved records - def after_latest = @set.where(after_latest_query, **@latest) + # Filter the newest records + def filter_newest = @set.where(filter_newest_query, **@latest) # Append the missing keyset keys if the set is restricted by select def apply_select diff --git a/gem/lib/pagy/keyset/sequel.rb b/gem/lib/pagy/keyset/sequel.rb index 479cf67ad..e828d9dad 100644 --- a/gem/lib/pagy/keyset/sequel.rb +++ b/gem/lib/pagy/keyset/sequel.rb @@ -26,8 +26,8 @@ def extract_keyset end end - # Filter out the already retrieved records - def after_latest = @set.where(::Sequel.lit(after_latest_query, **@latest)) + # Filter the newest records + def filter_newest = @set.where(::Sequel.lit(filter_newest_query, **@latest)) # Append the missing keyset keys if the set is restricted by select def apply_select diff --git a/test/pagy/countless_test.rb b/test/pagy/countless_test.rb index fbc863bb3..2325b7644 100644 --- a/test/pagy/countless_test.rb +++ b/test/pagy/countless_test.rb @@ -79,7 +79,7 @@ _(pagy.prev).must_equal 2 _(pagy.next).must_equal 1 end - it 'raises exception with no retrieved records and page > 1' do + it 'raises exception with no fetched records and page > 1' do _ { Pagy::Countless.new(page: 2, overflow: :exception).finalize(0) }.must_raise Pagy::OverflowError end end diff --git a/test/pagy/keyset_test.rb b/test/pagy/keyset_test.rb index a9d3220bf..e9fa1b86c 100644 --- a/test/pagy/keyset_test.rb +++ b/test/pagy/keyset_test.rb @@ -49,16 +49,16 @@ _ = pagy.records _(pagy.latest).must_equal({id: 10}) end - it 'uses :after_latest' do - after_latest = if model == Pet - ->(set, latest) { set.where('id > :id', **latest) } - else - ->(set, latest) { set.where(Sequel.lit('id > :id', **latest)) } - end + it 'uses :filter_newest' do + filter_newest = if model == Pet + ->(set, latest, _keyset) { set.where('id > :id', **latest) } + else + ->(set, latest, _keyset) { set.where(Sequel.lit('id > :id', **latest)) } + end pagy = Pagy::Keyset.new(model.order(:id), page: "eyJpZCI6MTB9", limit: 10, - after_latest: after_latest) + filter_newest:) records = pagy.records _(records.first.id).must_equal(11) end