[docs] A complete example of a basic string processing model and of calling it of invoking it via `curl`

[vadimkantorov]

I have the following echo-like models/modelA/1/model.py.

How can I call it from a command-line using curl?

curl -i -X POST localhost:8000/api/infer/modelA/1 -H "Content-Type: application/octet-stream" -H 'NV-InferRequest:batch_size: 1 input { name: "INPUT0" } output { name: "OUTPUT0" }' --data 'hello' gives HTTP 400.

I think I am not formatting the --data argument correctly. But it should not be very difficult for UTF-8 encoding, right?

Model code:

import json
import numpy as np
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
    @staticmethod
    def auto_complete_config(auto_complete_model_config):
        auto_complete_model_config.add_input( {"name": "INPUT0",  "data_type": "TYPE_UINT8", "dims": [-1]})
        auto_complete_model_config.add_output({"name": "OUTPUT0", "data_type": "TYPE_UINT8", "dims": [-1]})
        auto_complete_model_config.set_max_batch_size(0)
        return auto_complete_model_config

    def execute(self, requests):
        responses = []
        for request in requests:
            in_numpy = pb_utils.get_input_tensor_by_name(request, 'INPUT0').as_numpy()
            in_str = str(bytes(in_numpy), 'utf8')
            
            out_str = 'modelA:' + in_str
            out_numpy = np.frombuffer(bytes(out_str, 'utf8'), dtype = np.uint8)
            out_pb = pb_utils.Tensor('OUTPUT0', out_numpy)

            responses.append(pb_utils.InferenceResponse(output_tensors = [out_pb]))
        return responses

Currently I can successfully call it using:

import numpy as np
import tritonclient.http as httpclient

triton_client = httpclient.InferenceServerClient("localhost:8000")
model_name = 'modelA'

input_arr = np.frombuffer(bytes('hello', 'utf8'), dtype = np.uint8)
inputs = [httpclient.InferInput("INPUT0", input_arr.shape, "UINT8")]
inputs[0].set_data_from_numpy(input_arr, binary_data=True)

res = triton_client.infer(model_name=model_name, inputs=inputs)

output_arr = res.as_numpy('OUTPUT0')
output_str = str(bytes(output_arr), 'utf8')

print(output_str)

but I would like to use curl as it's simpler for some demonstration purposes and might be simpler for debugging string-processing models

[vadimkantorov]

I somehow solved it: https://github.com/vadimkantorov/tritoninfererenceserverstringprocprimer, but it would be nice to include something like this (the simplest possible curl-calling of a string-processing model) as an example in the README

[dyastremsky]

Thanks for solving this and sharing your code! We just released a generate endpoint (documentation here) that should hopefully make this exact use case. I think you can use any input instead of "text_input".

We are looking at options to make sending simple requests to Triton easier, so we will definitely be looking at this in the near future. If we do not do that soon, we can look at adding this as an example in our documentation.

[vadimkantorov]

@dyastremsky Does this generate endpoint support batch-inference? E.g. providing a batch of text inputs and providing a batch of text outputs back?

It would also be nice to have natively supported JSON inputs (including a batch/array of JSON inputs). I think this is very common in post-processing. So it would be nice to have a native support without having to manually serialize/deserialize from bytes/string

It's also important to have some extremely-simple format for calling by HTTP. Currently it must contain some prefix specifying the body length (required by grpc I guess?), but it's not very elegant. It would be nice to support a mode without this - for prototyping and kicking off requests from command line without having to calculate the length of the string used as input to the model. It would also be nice to support a mode when this meta-info is specified in some HTTP headers - there exist native, nice ways of providing these extra HTTP headers in curl

[vadimkantorov]

@dyastremsky Could you please hint on an example of a model code that knows how to support such a generate method? Is there anything to be done in Triton command line options or build-from-source options required to enable the generate extension? Should I pass --enable-generate to ./build.py?

Thank you :)

[dyastremsky]

Hi Vadim! Great questions. As far as getting the generate extension working, I think you just need to build off main or wait for the 23.10 release. CC: @GuanLuo

The generate endpoint is meant to simplify basic LLM inference and not meant to be run as a highly performant solution in production. So things like batching inputs on the client side are not supported and I don't expect we'll be adding more complicated use cases.

We do have in our queue to look into JSON support and easier client interaction. We also always encourage and appreciate contributions to the project.

[vadimkantorov]

Supporting batches may sometimes be useful for the coding/UX/uniformity standpoint, even if it doesn't increase perf for se. This can also be useful for some string postproc

[dyastremsky]

Thanks, Vadim! I received a clarification from the team that this should theoretically support batching if an array of inputs is provided for each batched input. However, we do not specifically test or document it. We have created a ticket to test and document batching with the generate endpoint to ensure it works and is available.

Ticket reference number: DLIS-5717.

[vadimkantorov]

Regarding the generate endpoint, it would also be great to have a model code example :) seems missing from the docs PR you linked above

[dyastremsky]

Noted, thanks for the suggestion Vadim!

[GuanLuo]

@vadimkantorov Have you looked at the example in inference protocol documentation? This is the example to send an inference request via curl. Your example is trying to use binary data extension which requires additional header, Inference-Header-Content-Length, for the server to parse the request.

If your model expects string/byte type instead of uint8, you don't need to put the integer representation of the string in the data field, but the string itself directly, i.e. ... "data" : ["hello"] . And with inference protocol, you can send batched request by properly setting the shape and tensor data, , i.e. "... shape" : [2, 1]", data" : [["hello"], ["another text"]] .

As @dyastremsky mentioned, you may send batched request via generate endpoint if you orchestrate your model properly, all I/Os' batch dimension is the only dynamic dimension so the server will map JSON entries to model inputs in batch. Inference protocol gives you more control when your use case is complex. The generate protocol is meant to provide a simple way to execute the model without needing to specify the tensor representation, which imposes restrictions and makes it difficult to be used for other use case.

[vadimkantorov]

For now, I managed to make it work via this binary data extension - so I am not blocked.

Have you looked at the example in inference protocol documentation?

I did look at this, but I did not understand how to pass a string in the data field and how to read it in the string form in the model code (I think there was no complete code example)

As @dyastremsky mentioned, you may send batched request via generate endpoint

For string postproc, this would be perfect, but again I could not find a complete model code example (implementing this generate endpoint) for now.

Then again what'd be nice is an example of generate endpoint supporting batches of text inputs and an endpoint supporting batches of arbitrary JSON inputs/outputs (maybe alread supported via this Predict Protocol - Version 2? a complete example of model code + calling the endpoint with curl would be perfect)

[vadimkantorov]

E.g. the generate endpoint docs stress on using text_input as field name, but the code in https://github.com/triton-inference-server/server/blob/9da513528e34a9b91a216cd7e1b668fee9fbbc92/qa/L0_http/generate_models/mock_llm/1/model.py uses some other PROMPT field name. Are these field names standardized wrt generate endpoint or are fully custom?

Also, in the code example, the strings stay encoded as numpy tensors, but what would be great is getting a regular python string out of them, and it would demonstrate how to do proper decoding (regarding utf-8/utf-16 etc)