Performance issue when using 'count' after upgrading to Mongoid 7.5.4

[sosolidkk]

After upgrading our application to Mongoid version 7.5.4, we observed a significant performance issue, with some screens taking several minutes to load paginated data.

The issue seems related to the use of the count method. According to the Mongoid 7.2.0 release notes, there was a notable change in how the count method works, and the recommended approach now is to use estimated_count where possible.

In our application, two pages in particular have become noticeably slower. These pages are backed by collections containing 3,350,998 and 110,339,992 records, respectively. The page with 3 million records takes between 15 seconds and 3 minutes to load, while the one with 110 million records can take up to 18 minutes.

We believe the issue is located in the following file:

• kaminari-mongoid/mongoid_criteria_methods.rb

Specifically, the problem seems to be with the use of the count method on line 11, and in the total_count function on line 22.

Steps we’ve taken so far:

• Rollback to a previous version:
  We downgraded to Mongoid version 7.1.11, and the issue did not occur.

• Monkey patching:
  We applied a monkey patch to replace the count method with the appropriate method call, which resolved the performance issue. Here is the code for the monkey patch currently running in production:

module MongoidCriteriaMethods
  def entry_name(options={})
    model_name.human(options.reverse_merge(default: model_name.human.pluralize(options[:count_registers])))
  end

  # Overrides the `total_count` method to use `estimated_count` instead of
  # `count` or `size` when possible. This change is crucial for large datasets because
  # `count` and `size` do not utilize MongoDB indexes, causing slow queries
  # in scenarios like paginating or fetching activity histories.
  def total_count
    @total_count ||= if options[:max_scan] && (options[:max_scan] < count_registers)
      options[:max_scan]
    elsif max_pages && (max_pages * limit_value < count_registers)
      max_pages * limit_value
    else
      unpage.count_registers
    end
  end

  def count_registers
    estimated_count
  rescue ::Mongoid::Errors::InvalidEstimatedCountCriteria
    size
  end
end

System Information:

• Ruby: 3.0.7
• Mongoid: 7.5.4
• Rails: 6.1.7.8

For reference, here is the release note highlighting the change in the count method:

• Mongoid 7.2.0 Release Notes

Model#count is now implemented using the aggregation pipeline $count stage. Model#estimated_count method was added to provide the estimated count that Model#count used to provide. This change should be transparent to most applications but may increase runtime for applications that call #count, especially with complex conditions (with the benefit being of #count producing accurate results). See this upgrading guide section for further details.

We would appreciate any guidance or suggestions on addressing this issue more permanently.

Thank you!