Cradle For Algorithm Docker

Python 3.5 cradle for algorithm dockers, based on Sanic, so asynchronous and blazing fast.

Get started

Prerequisite

To build and use CADoR you will need (see requirements) :

• Python >= 3.5
• Sanic==0.7.0
• sanic-openapi
• requests-file
• jsonschema
• aiokafka==0.4.0
• apache-libcloud
• trio
• kafka
• shapely

Install & build

To build wheel package & docker image :

$ cd docker && make build

To install:

$ make install

How to use

API REST

CADoR exposes the same REST API as the GeoProcessing SDK :

• http://server:port/describe : GET request, returning the service information in JSON format,
• http://server:port/jobs : POST request, run the processing and deal with the results locally on the server
• http://server:port/health : GET request to test if service is alive
• http://server:port/swagger : GET request giving access to swagger API description

It deals with these requests and manage the docker algorithm.

KAFKA

Image can be read in input queue if it has been configured. In this case, results are push to a queue or an object store given the configuration.

Accepted payload

CADoR behaviour can be tunned by adding a 'technicalMetadata' field to the algorithm payload :

• outputStorage : configure S3, GCS ou POSIX storage to store output images
• referential : if present, convert all output geojson coordinates
• postProcessing.tags : automatic transformation of geojson output into tags

 {
     ... // algorithm payload
     
     "technicalMetadata" : {
         "outputStorage" : {
             ... // S3, GCS ou POSIX configuration (cf. Storage configurations)
         },
         "postProcessing": {
            "tags": true
         },         
         "referential": {
            "upperLeft": {
              "lat": 41.2,
              "lon": 2.07
            },
            "latStep": 1.07E-5,
            "lonStep": 1.072E-5
         }         
     }
 }

S3 storage :

  "outputStorage": {
    "S3": {
      "url": "http://object-storage:9000/api/v1/s3",
      "containerPath": "folder0/folder1/folder2/test",
      
      "ssl": false,
      "key": "xxxxx",
      "secret": "xxxxx"
    }
  },

GCS storage :

  "outputStorage": {
    "GCS": {
      "containerPath": "folder0/folder1/folder2/test",
      
      "key": "xxxxx",
      "secret": "xxxxx"
    }
  },

POSIX storage :

  "outputStorage": {
    "POSIX": {
      "folder": "/path/to/Storage"
    }
  },

Configuration

You can configure the server with the following environement variables :

• CADOR_PORT : port where the server run
• CADOR_USE_SSL : whether to use SSL or not (call the service with https scheme if activated)
• PROCESSING_SERVER : address of the algorithm docker to bin with
• KAFKA_BROKERS_REQUEST : queue_in to read request orders (only kafka is available for now). Not mandatory, if not provided only REST API is available
• EOPAAS_JOB_REQUEST_TOPIC : queue_in topic, mandatory if queue_in specified
• KAFKA_CONSUMER_GROUP : group used by all Kafka consumers, all instances of cador for one algorithm needs to have the same
• KAFKA_BROKERS_OUTPUT : queue_out to push results
• EOPAAS_JOB_OUTPUT_TOPIC : topic for queue out
• KAFKA_BROKERS_STATUS : queue to push status, not mandatory
• EOPAAS_STATUS_UPDATE_TOPIC : topic to push status
• KAFKA_BROKERS_TAGS : queue to push tags, not mandatory
• EOPAAS_JOB_TAGS_TOPIC : topic to push tags