starting.md

Starting DJL Serving

Use the following command to start model server locally:

djl-serving

The model server will be listening on port 8080. You can also load a model for serving on start up:

djl-serving -m "https://resources.djl.ai/demo/mxnet/resnet18_v1.zip"

Open another terminal, and type the following command to test the inference REST API:

curl -O https://resources.djl.ai/images/kitten.jpg
curl -X POST http://localhost:8080/predictions/resnet18_v1 -T kitten.jpg

or:

curl -X POST http://localhost:8080/predictions/resnet18_v1 -F "data=@kitten.jpg"

[
  {
    "className": "n02123045 tabby, tabby cat",
    "probability": 0.4838452935218811
  },
  {
    "className": "n02123159 tiger cat",
    "probability": 0.20599420368671417
  },
  {
    "className": "n02124075 Egyptian cat",
    "probability": 0.18810515105724335
  },
  {
    "className": "n02123394 Persian cat",
    "probability": 0.06411745399236679
  },
  {
    "className": "n02127052 lynx, catamount",
    "probability": 0.010215568356215954
  }
]

Examples for loading models

# Load models from the DJL model zoo on startup
djl-serving -m "djl://ai.djl.pytorch/resnet"

# Load version v1 of a PyTorch model on GPU(0) from the local file system
djl-serving -m "resnet:v1:PyTorch:0=file:$HOME/models/pytorch/resnet18/"

# Load a TensorFlow model from TFHub
djl-serving -m "resnet=https://tfhub.dev/tensorflow/resnet_50/classification/1"

Examples for customizing data processing

# Use the default data processing for a well-known application
djl-serving -m "file:/resnet?application=CV/image_classification"

# Specify a custom data processing with a Translator
djl-serving -m "file:/resnet?translatorFactory=MyFactory"

## Pass parameters for data processing
djl-serving -m "djl://ai.djl.pytorch/resnet?applySoftmax=false"

Using DJL Extensions

# Load a model from an AWS S3 Bucket
djl-serving -m "s3://djl-ai/demo/resnet/resnet18.zip"

# Load a model from HDFS
djl-serving -m "hdfs://localhost:50070/models/pytorch/resnet18/"

# Use a HuggingFace tokenizer
djl-serving -m "file:/bertqa?transaltorFactory=ai.djl.huggingface.BertQATranslator"