How to upload binary data files?

[mikeholler]

Hello there,

I considered posting this to Gitter, but it does not seem very active so I wanted to post here instead.

I'm evaluating using uplink for our API clients, and so far it looks like a wonderful tool, checking all of the boxes I'd expect. However, we have a few routes that accept binary data (think simple document store), and I don't see a way to upload unstructured binary data using uplink. Here's what I tried:

from uplink import Consumer, Body, post

class SampleApi(Consumer):
    @post("post")
    def create(self, body: Body):
        """Post some data to httpbin's /post method"""

Then I start httpbin locally with docker run --rm -p 80:80 kennethreitz/httpbin, and fire up an interpreter and tried a few things (which you'll see below). I've combed the docs and cannot find any examples of arbitrary data being uploaded, and when I search binary, bytes, file, or bytearray in this project I didn't see much. I'd like to know how to do this, and I'm thinking it should probably be added to the project's documentation as well.

>>> api = Sample(base_url="http://localhost")
>>> api.create(body=b'1234')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/builder.py", line 99, in __call__
    self._request_definition.define_request(request_builder, args, kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/commands.py", line 267, in define_request
    self._argument_handler.handle_call(
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 153, in handle_call
    self.handle_call_args(request_builder, call_args)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 158, in handle_call_args
    annotation.modify_request(request_builder, call_args[name])
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 182, in modify_request
    self._modify_request(request_builder, converter(value))
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 19, in convert
    return self._converter(value)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 30, in convert
    dumped = json.dumps(value, default=self._default_json_dumper)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 25, in _default_json_dumper
    return obj.__dict__  # pragma: no cover
AttributeError: 'bytes' object has no attribute '__dict__'

>>> api.create(body=bytearray([1, 2, 3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/builder.py", line 99, in __call__
    self._request_definition.define_request(request_builder, args, kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/commands.py", line 267, in define_request
    self._argument_handler.handle_call(
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 153, in handle_call
    self.handle_call_args(request_builder, call_args)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 158, in handle_call_args
    annotation.modify_request(request_builder, call_args[name])
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/arguments.py", line 182, in modify_request
    self._modify_request(request_builder, converter(value))
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 19, in convert
    return self._converter(value)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 30, in convert
    dumped = json.dumps(value, default=self._default_json_dumper)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/mjholler/.pyenv/versions/3.8.0/lib/python3.8/site-packages/uplink/converters/standard.py", line 25, in _default_json_dumper
    return obj.__dict__  # pragma: no cover
AttributeError: 'bytearray' object has no attribute '__dict__'

Then, when this didn't work, I just wanted to try something with plain requests, which of course did work:

>>> import requests
>>> r = requests.post("http://localhost/post", data=b'1234')
>>> r.status_code
200
>>> r.json()
{'args': {}, 'data': '1234', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Content-Length': '4', 'Host': 'localhost', 'User-Agent': 'python-requests/2.22.0'}, 'json': 1234, 'origin': '172.17.0.1', 'url': 'http://localhost/post'}
>>> 

Note the 'json' field has 1234 which I posted is binary, but it's just coincidentally rendering it as valid JSON (since it is). I don't know if httpbin has something for binary files, so this was what I thought I could use.

[mikeholler]

Update: httpbin has a POST /anything method which doesn't do the weird json interpretation that my requests example shown above does, but results in the same error when I tried to use uplink in a similar way to above with that path.

[prkumar]

Hi @mikeholler,

Thanks for opening this issue! This looks related to #180, in that it is an issue with the library's default serialization strategy: before data is passed through requests.post, Uplink applies the StandardConverter, which passes the data through the json serializer. Frankly, this default logic is outright wrong, and the library should just pass the data to requests without any transformation. I'll prioritize this work for the next minor release.

A workaround for now is to monkey-patch the StandardConverter's create_request_body_converter at the beginning of your module, like so:

from uplink.converters import StandardConverter

def pass_through_request_body_converter(self, type_, *args, **kwargs):
        return lambda value: value

StandardConverter.create_request_body_converter = pass_through_request_body_converter

[mikeholler]

@prkumar thank you for the timely response! The code snippet is great, too. Any idea, with this workaround, how I might adapt it to have one route that interprets the body as JSON and one that just passes it through as bytes in the same class?

I love that you're prioritizing this work, too. It does seem like this is pretty fundamental behavior and I'm glad to see that you think so as well :)

[prkumar]

@mikeholler - Sure thing! If you need to accept both JSON and binary data, I would recommend you define two separate methods, one decorated with @json . Here's an (untested) example that illustrates this approach:

from uplink import Consumer, Body, post, json

class SampleApi(Consumer):
    @post("post")
    def create_binary(self, body: Body):
        """Post some binary data to httpbin's /post method"""

    @json
    @create
    def create_json(self, body: Body):
        """Post some JSON data to httpbin's /post method"""

(This example demonstrates the ability to extend consumer methods.)

You could take this further and expose a third method to joins the behavior and picks the correct request based on the type of the given body:

from uplink import Consumer, Body, post, json

class SampleApi(Consumer):
    def create(self, body):
        if isinstance(body, (bytes, bytearray)):
            return self.create_binary(body)
        return self.create_json(body)

    @post("post")
    def create_binary(self, body: Body):
        """Post some binary data to httpbin's /post method"""

    @json
    @create
    def create_json(self, body: Body):
        """Post some JSON data to httpbin's /post method"""

[mikeholler]

@prkumar that's a great workaround. I think you've misunderstood my use-case a little bit (I only want to except binary OR json in a method, mutually exclusive, but I do want to have a Consumer that has methods that do both). However, I am clear on exactly how to solve my problem now. Thank you for the help and I look forward to an update making this into master in the future.