Pagy with Chewy gem built on top of elasticsearch-rails?

[dlegr250]

I see Pagy has an implementation for elasticsearch-rails gem itself, where it's getting the response from Elasticsearch and generating the pagination for it.

However, a project I'm on uses chewy, which is built on top of elasticsearch-rails and provides useful features not found in the base gem itself. The chewy gem autoloads either the kaminari or will_paginate gems for pagination, if they are present, which does cause me some frustration.

It looks like the only way I can get Pagy to work with chewy is to ensure that neither kaminari or will_paginate are included in the project, and then I have to use the raw Pagy object itself to get the results.

chewy_query = SomeIndex.query(...)
@pagy, @records = Pagy.new(chewy_query.objects, page: 1, items: 50, count: chewy_query.total)

In short, chewy wraps the elasticsearch-rails results and the #objects method returns the ActiveRecord objects themselves, and chewy implements a #limit and #offset method that is used for pagination. Reading the Pagy docs, it looks like Pagy uses those methods automatically for limit/offset pagination.

I did some basic benchmarking using benchmark-memory for this setup and Pagy does indeed have about a 40-50% memory reduction, but the difficulty was I had to add/remove gems and run the benchmarks separately because I could not have both Pagy and will_paginate running together in the project because chewy would automatically load and use will_paginate and Pagy was then doubling the objects used because the pagination was being done twice.

I'm going to reach out to the chewy repo separately and ask about how they handle optional pagination or if there's a better way to turn off automatic loading of pagination, but I also wanted to see if anyone on the Pagy discussions could let me know if the approach I'm using above is an appropriate use of Pagy for this?

I'm used to using Pagy with the standard ActiveRecord setup, so I've never really had to dive deep into the inner workings of it.

[ddnexus]

If you have an ElasticsearchRails object available, I guess you could just create a pagy object out of it by using the pagy extra in passive mode.

And I guess that you can use that pagy object to paginate the AR objects coming out of chewy, because the number of results should match 1 to 1 the AR records.

So it might be a lot simpler than what it looks.

[benkoshy]

https://ddnexus.github.io/pagy/docs/extras/elasticsearch_rails/#passive-mode

[ddnexus]

Why this is marked as closed but it's unanswered? Reopening.

[benkoshy]

Why this is marked as closed but it's unanswered? Reopening.

I do not know why / how it is marked as closed. Github says it was marked as resolved 9 hours ago. I think it might be a github UI design issue?

[ddnexus]

@dlegr250 any feedback about my answer?

[dlegr250]

@ddnexus I built a working version built on top of Chewy without touching the ES-rails object, but I am also going to try getting the ES-rails client from Chewy directly as you mentioned and try that out. I'll try that tomorrow at my work project and post my results to this thread.

[dlegr250]

The version that works on my project using Chewy and Pagy is:

chewy_query = SomeIndex.query(...)
@pagy, @records = Pagy.new(chewy_query, page: 1, items: 50, count: chewy_query.count)

The chewy_record.objects are called internally when processed, and we only have to pass in the count value to Pagy in order to work.

I didn't get a good version working connecting to the Elasticsearch-rails object internally held by Chewy, because Chewy adds additional functionality that I needed, so trying to retrieve records from ES object itself wouldn't work in my situation. But the above code seems to work just fine without having to make any further additions to Pagy or Chewy!

[ddnexus]

@dlegr250 thank you for the feedback

trying to retrieve records from ES object itself wouldn't work in my situation

Indeed I wasn't suggesting to use the raw ES objects, but the AR objects provided by chewy exactly as you currently do. The only difference would be extracting the pagy object instead of manually creating it.

chewy_query = SomeIndex.query(...)
# you can even put the next line directly in the view instead of pre-building the object, if you want to delay it
@pagy = Pagy.new_from_elasticsearch_rails(chewy_query.whatever_is_the_metod_to_get_the_elasticsearch_response)

That would be the less app-invasive way to use pagy: i.e. leveraging the native/built-in ES pagination. Pagy would merely adapt to whatever pagination has already been performed by chewy or ES or whatever does it.

[ddnexus]

A bit off topic, but worth to mention.

Retrieving the results from ES and query the DB just to show them (or creating AR objects-like out of the ES result, even without querying) is EXTREMELY inefficient and slow, beside being a big waste of CPU and memory.

If you want performance, you just don't do that. Just index the fields that you need to search AND the fields that you need to show, and retrieve only the ones that you need to show. All the data you need is in your ES result, nicely available in a simple JSON structure. You should use it directly in your view, or if you want to relieve your server, just serve it and render it in the client with a few lines of javascript.