Elasticsearch::Persistence find_in_batches causes BadRequest reason unrecognized token

[phlegx]

Hi!
I use:

• Elastisearch 5.x
• elasticsearch-rails 0.1.9
• elasticsearch-persistence 0.1.9
• elasticsearch-ruby 5.0.3

and if I want to find_in_batches I get this error:

Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":
{"type":"illegal_argument_exception","reason":"Failed to parse request body"}],"type":"
illegal_argument_exception","reason":"Failed to parse request body","caused_by":
"type":"json_parse_exception","reason":"Unrecognized token 'DnF1ZX...': was expecting
('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput
@2e5b1e0d; line: 1, column: 257]"}},"status":400}

The problem comes from this line in persistence/model/find.rb#L135

response = gateway.client.scroll( { scroll_id: response['_scroll_id'], scroll: scroll } )

Any idea what can be the problem?

[SOLUTION] Bug is solved in master repository of elasticsearch-ruby but not in version 5.0.3.

[jogaco]

Same problem for find_each here

[jogaco]

Wonder if this should remain opened, as the Gemfile dependencies refer to a possibly incompatible 6.0.0 version in the future:

  specs:
    elasticsearch (6.0.0.pre)
      elasticsearch-api (= 6.0.0.pre)
      elasticsearch-transport (= 6.0.0.pre)
    elasticsearch-api (6.0.0.pre)
      multi_json
    elasticsearch-transport (6.0.0.pre)
      faraday
      multi_json

[karmi]

@phlegx I've released couple of new versions for both -rails and -ruby, can you check it with them, please?

@jogaco, in which Gemfile is the 6.0.0.pre version referenced?

[phlegx]

I have write this hook using the versions mentioned in this issue. If this changes are in the new pre versions than all works fine:

# config/initializers/hook_elasticsearch.rb
#
# Bug not solved in version 5.0.3 but in master
# https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-api/lib/elasticsearch/api/actions/scroll.rb
module Elasticsearch
  module API
    module Actions

      def scroll(arguments={})
        method = HTTP_GET
        path   = "_search/scroll"
        valid_params = [
          :scroll,
          :scroll_id ]

        params = Utils.__validate_and_extract_params arguments, valid_params
        body   = arguments[:body]

        perform_request(method, path, params, body).body
      end

    end
  end
end


# Bug not solved in version 0.1.9
# https://github.com/elastic/elasticsearch-rails/commit/e81fc5b5d0648280fdf054566823e647352b45eb
module Elasticsearch
  module Persistence
    module Model
      module Find
        module ClassMethods

          def find_in_batches(options={}, &block)
            return to_enum(:find_in_batches, options) unless block_given?

            search_params = options.extract!(
              :index,
              :type,
              :scroll,
              :size,
              :explain,
              :ignore_indices,
              :ignore_unavailable,
              :allow_no_indices,
              :expand_wildcards,
              :preference,
              :q,
              :routing,
              :source,
              :_source,
              :_source_include,
              :_source_exclude,
              :stats,
              :timeout)

            scroll = search_params.delete(:scroll) || '5m'

            body = options

            # Get the initial batch of documents and the scroll_id
            #
            response = gateway.client.search( { index: gateway.index_name,
                                         type:  gateway.document_type,
                                         scroll: scroll,
                                         sort:   ['_doc'],
                                         size:   20,
                                         body:   body }.merge(search_params) )


            # Scroll the search object and break when receiving an empty array of hits
            #
            while response['hits']['hits'].any? do
              yield Repository::Response::Results.new(gateway, response)

              response = gateway.client.scroll( { scroll_id: response['_scroll_id'].to_s, scroll: scroll } )
            end

            return response['_scroll_id']
          end

        end
      end
    end
  end
end

[karmi]

I've updated the dependencies in the .gemspec, and fixed the failing integration tests, which are now passing for me locally.

[jogaco]

@karmi The working version was in master.
Here the relevant portion from my Gemfile.lock after bundle
Will check again when I find the time.
Thanks for the fix.

  remote: git://github.com/elastic/elasticsearch-ruby.git
  revision: 24d361223a5de52e6fd76d66662215deeb845caf
  specs:
    elasticsearch (6.0.0.pre)
      elasticsearch-api (= 6.0.0.pre)
      elasticsearch-transport (= 6.0.0.pre)
    elasticsearch-api (6.0.0.pre)
      multi_json
    elasticsearch-transport (6.0.0.pre)
      faraday
      multi_json

[estolfo]

Closing as this is no longer relevant.