Querying Clipper with an image

[kakru]

Please provide an example of querying Clipper with an image.
I assume that it should be already possible with an image encoded to JSON and passed as a string:

{
"input" := [double] | [int] | [byte] | [float] | string
}

A basic example could return image size tuple done with a Python Closure model.

Are there any plans on extending accepted input format types?

[kakru]

I managed to run a simple query sending an image as (b64encoded) string in JSON:

{"input": base64.b64encode(open(filename, "rb").read())}

and using the following python closure in Clipper:

def jpgsize(img):
  import base64, io, PIL.Image, tempfile, os
  tmp = tempfile.NamedTemporaryFile('w', delete=False, suffix='.jpg')
  tmp.write(io.BytesIO(base64.b64decode(img)).getvalue())
  tmp.close()
  size = PIL.Image.open(tmp.name, 'r').size
  os.unlink(tmp.name)
  return [size]

But it doesn't seem like an optimal solution.

Is there a better way?

[dcrankshaw]

You can actually register a model with type "bytes" and then send your image as a base64 encoded string to Clipper in the same way. Clipper will decode the string back into a byte array and provide your Python closure a byte array directly. This is a little more efficient because the RPC message from Clipper to the model container will be the bytes directly, rather than the base64 encoding.

We don't have any plans to extend the set of possible input type to include something like "image".

To further improve performance (because sending base64 encoded images isn't particularly efficient), we will be adding support for an RPC-based query interface (see #238) where you could send the image as bytes directly.

[kakru]

Great, thank you!

I have registered the model with type "bytes" as suggested

from clipper_admin import ClipperConnection, DockerContainerManager
from clipper_admin.deployers import python as python_deployer
clipper_conn = ClipperConnection(DockerContainerManager())
clipper_conn.connect()
clipper_conn.register_application(name="jpgsize", input_type="bytes", default_output="-1.0", slo_micros=1000000)
python_deployer.deploy_python_closure(clipper_conn, name="jpgsize", version=1, input_type="bytes", func=jpgsize)
clipper_conn.link_model_to_app(app_name="jpgsize", model_name="jpgsize")

and modified the python closure to the following:

def jpgsize(img):
  import io, PIL.Image, tempfile, os
  tmp = tempfile.NamedTemporaryFile('w', delete=False, suffix='.jpg')
  tmp.write(io.BytesIO(img).getvalue())
  tmp.close()
  size = PIL.Image.open(tmp.name, 'r').size
  os.unlink(tmp.name)
  return [size]

It works well.

If you find it useful to add an example like that to the guide/tutorial - I can prepare a PR for docs.
This would open a possibility of extending the tutorial with TensorFlow supported object detection (with already trained network) and other interesting examples when TF support is merged into the master.

[dcrankshaw]

Yeah that would be a great contribution! The guides are part of the Clipper website repo (https://github.com/ucbrise/clipper-website/tree/master/content/tutorials). You can either add a section to the "Basic Concepts" guide, or create a new tutorial for image detection (if you're feeling really ambitious).

By the way, Tensorflow support was added in #319. It just hasn't made it out into a release yet. And #322 will add support for PyTorch models, which have their own model zoo of image models.

[Corey-Zumar]

@kakru Reviewing #325 now, sorry for the delay

[rkooo567]

Closed with #504