Skip to content

responses multipart_matcher does not consume the passed iterator; how to mock?  #713

Open

Description

Describe the bug

Hi, I am developing some python code utilizing mulipart/form-data headers for uploading a file. When I try to mock the endpoint response with the following code, then it hangs.

@responses.activate
def test_multipart_upload(client, test_file_definition):
    responses.add(
        responses.POST,
        url=gen_url(<some_endpoint>),
        json=1,
    )
    client.multipart_upload(...)

Responses: 0.25.0
Python: 3.11

Additional context

Once I will add a multipart matcher, I can get an error stating that the iterator does not match the data (as the generator is just an object in the memory).

~#@❯ pytest -x --no-cov -k test_multipart_upload
Test session starts (platform: win32, Python 3.11.8, pytest 7.4.0, pytest-sugar 1.0.0)
rootdir: C:\Users\<REDACTED>
configfile: pytest.ini
testpaths: tests
plugins: anyio-4.3.0, nbmake-1.5.3, cov-4.1.0, icdiff-0.9, sugar-1.0.0, timeout-2.3.1


―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_create_new_version ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 

client = Client(dummy@unknown(http://test:54422))>
tmp_path = WindowsPath('C:/Users/<REDACTED>/pytest-442/test_multipart_upload0')

    @responses.activate
    def test_multipart_upload(client, tmp_path):
        file_path = tmp_path / "test_data.txt"
        write_file(file_path)  # 1 kB

        responses.add(
            responses.POST,
            url=gen_url(<some_endpoint>),
            match=[
                responses.matchers.multipart_matcher({"file_name": b""})
                ],
            json=1,
        )
>       client.multipart_upload(...)

C:\Users\<REDACTED>\tests\client\test_multipart_upload.py
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1173: in send
    return self._on_request(adapter, request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <responses.RequestsMock object at 0x000002AAF71E2B10>, adapter = <requests.adapters.HTTPAdapter object at 0x000002AAF743D210>, request = <PreparedRequest [POST]>, retries = None
kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}, match = None
match_failed_reasons = ['multipart/form-data doesn\'t match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0...: form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'']
resp_callback = None
error_msg = 'Connection refused by Responses - the call doesn\'t match any registered mock.\n\nRequest: \n- POST http://test:54422... form-data; name="file_name"; filename="file_name"\\r\\n\\r\\n\\r\\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\\r\\n\'\n'
i = 0, m = <Response(url='http://test:54422/<some_endpoint>' status=200 content_type='application/json' headers='null')>

    def _on_request(
        self,
        adapter: "HTTPAdapter",
        request: "PreparedRequest",
        *,
        retries: Optional["_Retry"] = None,
        **kwargs: Any,
    ) -> "models.Response":
        # add attributes params and req_kwargs to 'request' object for further match comparison
        # original request object does not have these attributes
        request.params = self._parse_request_params(request.path_url)  # type: ignore[attr-defined]
        request.req_kwargs = kwargs  # type: ignore[attr-defined]
        request_url = str(request.url)

        match, match_failed_reasons = self._find_match(request)
        resp_callback = self.response_callback

        if match is None:
            if any(
                [
                    p.match(request_url)
                    if isinstance(p, Pattern)
                    else request_url.startswith(p)
                    for p in self.passthru_prefixes
                ]
            ):
                logger.info("request.allowed-passthru", extra={"url": request_url})
                return self._real_send(adapter, request, **kwargs)  # type: ignore

            error_msg = (
                "Connection refused by Responses - the call doesn't "
                "match any registered mock.\n\n"
                "Request: \n"
                f"- {request.method} {request_url}\n\n"
                "Available matches:\n"
            )
            for i, m in enumerate(self.registered()):
                error_msg += "- {} {} {}\n".format(
                    m.method, m.url, match_failed_reasons[i]
                )

            if self.passthru_prefixes:
                error_msg += "Passthru prefixes:\n"
                for p in self.passthru_prefixes:
                    error_msg += f"- {p}\n"

            response = ConnectionError(error_msg)
            response.request = request

            self._calls.add(request, response)
>           raise response
E           requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.
E
E           Request: 
E           - POST http://test:54422/<some_endpoint>?complete=False&encoding=utf-16
E
E           Available matches:
E           - POST http://test:54422/<some_endpoint> multipart/form-data doesn't match. Request body differs. <generator object chunk_file_for_upload at 0x000002AAF73C0DC0> aren't equal b'--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1\r\nContent-Disposition: form-data; name="file_name"; filename="file_name"\r\n\r\n\r\n--a5d476d4-c8cc-43cc-bf88-6b8246dadfa1--\r\n'

C:\Users\<REDACTED>\envs\mylib311\Lib\site-packages\responses\__init__.py:1100: ConnectionError

Version of responses

0.25.0

Steps to Reproduce

Add an generator to be consumed by the request.

Expected Result

Code using responses does not hang anymore.

Actual Result

Tests using responses hang.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions