TypeError: index() got an unexpected keyword argument 'headers' with sample code

[wpm]

I have written a pytest version of the sample code.

import elasticsearch

class FooService:

    def __init__(self):
        self.es = elasticsearch.Elasticsearch(hosts=[{'host': 'localhost', 'port': 9200}])

    def create(self, index, body):
        es_object = self.es.index(index, body)
        return es_object.get('_id')

    def read(self, index, id):
        es_object = self.es.get(index, id)
        return es_object.get('_source')

import pytest
from elasticmock import elasticmock

@elasticmock
def test_create_and_read_object():
    # Variables used to test
    index = 'test-index'
    expected_document = {
        'foo': 'bar'
    }

    # Instantiate service
    service = FooService()

    # Index document on ElasticSearch
    id = service.create(index, expected_document)
    assert id is not None

    # Retrieve document from ElasticSearch
    document = service.read(index, id)
    assert expected_document == document

When I run it I get the following error

$ pytest em.py 
========================================== test session starts ===========================================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/wmcneill/Documents/Misc
plugins: celery-4.4.2
collected 1 item                                                                                         

em.py F                                                                                            [100%]

================================================ FAILURES ================================================
______________________________________ test_create_and_read_object _______________________________________

    @elasticmock
    def test_create_and_read_object():
        # Variables used to test
        index = 'test-index'
        expected_document = {
            'foo': 'bar'
        }
    
        # Instantiate service
        service = FooService()
    
        # Index document on ElasticSearch
>       id = service.create(index, expected_document)

em.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
em.py:9: in create
    es_object = self.es.index(index, body)
../../greenfield/document_store/venv/lib/python3.7/site-packages/elasticmock/behaviour/server_failure.py:27: in decorated
    response = f(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<elasticmock.fake_elasticsearch.FakeElasticsearch object at 0x1051cc110>, 'test-index', {'foo': 'bar'})
kwargs = {}, params = {}, headers = {}, p = 'request_timeout'

    @wraps(func)
    def _wrapped(*args, **kwargs):
        params = {}
        headers = {}
        if "params" in kwargs:
            params = kwargs.pop("params").copy()
        if "headers" in kwargs:
            headers = {
                k.lower(): v for k, v in (kwargs.pop("headers") or {}).items()
            }
        if "opaque_id" in kwargs:
            headers["x-opaque-id"] = kwargs.pop("opaque_id")
    
        for p in es_query_params + GLOBAL_PARAMS:
            if p in kwargs:
                v = kwargs.pop(p)
                if v is not None:
                    params[p] = _escape(v)
    
        # don't treat ignore, request_timeout, and opaque_id as other params to avoid escaping
        for p in ("ignore", "request_timeout"):
            if p in kwargs:
                params[p] = kwargs.pop(p)
>       return func(*args, params=params, headers=headers, **kwargs)
E       TypeError: index() got an unexpected keyword argument 'headers'

../../greenfield/document_store/venv/lib/python3.7/site-packages/elasticsearch/client/utils.py:92: TypeError
======================================== short test summary info =========================================
FAILED em.py::test_create_and_read_object - TypeError: index() got an unexpected keyword argument 'head...
=========================================== 1 failed in 0.28s ============================================

This is ElasticMock version 1.5.0 and elasticsearch version 7.6.0.

[vrcmarcos]

I run a similar experiment in my environment and it worked. Here's my step-by-step:

1. Create foo_service.py:

import elasticsearch


class FooService:

    def __init__(self):
        self.es = elasticsearch.Elasticsearch(hosts=[{'host': 'localhost', 'port': 9200}])

    def create(self, index, body):
        es_object = self.es.index(index, body)
        return es_object.get('_id')

    def read(self, index, id):
        es_object = self.es.get(index, id)
        return es_object.get('_source')

2. Create foo_service_test.py:

from unittest import TestCase
from elasticmock import elasticmock
from elasticmock.foo_service import FooService


class FooServiceTest(TestCase):

    @elasticmock
    def test_create_and_read_object(self):
        # Variables used to test
        index = 'test-index'
        expected_document = {
            'foo': 'bar'
        }

        # Instantiate service
        service = FooService()

        # Index document on ElasticSearch
        id = service.create(index, expected_document)
        assert id is not None

        # Retrieve document from ElasticSearch
        document = service.read(index, id)
        assert expected_document == document

3. Test output:

Testing started at 16:50 ...
C:\Users\Marcos\py_venv\elasticmock\Scripts\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2019.3.3\plugins\python-ce\helpers\pycharm\_jb_unittest_runner.py" --target foo_service_test.FooServiceTest
Launching unittests with arguments python -m unittest foo_service_test.FooServiceTest in C:\Users\Marcos\workspace\elasticmock\tests



Ran 1 test in 0.003s

OK

Process finished with exit code 0

Could you try it running your tests in a class OR using unittest instead of py.test? Don't know what's going on in your side, but let's figure out!

[stbe]

Hey! I happened to stumble across the same issue. I believe the reason is that some methods have changed between elasticsearch-py 6.x and 7.x and now also accept an optional headers argument. For example the ping:

In 6.x:

def ping(self, params=None):

In 7.x:

def ping(self, params=None, headers=None):

So the methods in elasticmock/fake_elasticsearch.py do not account for that parameter that gets applied by the query_params decorator.

[chesstrian]

Based on the error shared by @wpm, which I could reproduce, the test passes when you add headers=None to index and get methods.

[vrcmarcos]

Awesome that you dove deep and found a workaround! Thanks for sharing that!

I'll try to add more checks in the library to make syre it'll properly work even if you do not pass the additional parameters.

[jcezarms]

@vrcmarcos is it plausible to add the headers=None attribute in FakeElasticsearch's methods to prevent this issue?

I mean, I could do it in a PR, but I don't see any change other than adding that parameter to all operations.

[DementeTech2]

If it worth I have the same issue with info() and because I'm working with elasticsearch 6.8 I just fix the library version:

elasticsearch>=6.0.0,<7.0.0

That solve my issue