Skip to content

Solver REST API App

Jump to bottom
Jussi Saarivirta edited this page Feb 5, 2022 · 9 revisions

TOC

Overview

In addition to the console Solver App, the REST API app is provided as an alternative means to provide the solver as a service. The API is a .NET 6 Web API project, published as a single self-contained executable. It's as easy to run as the console solver. Once you run it, it starts to serve the REST API, accepts requests and queues and executes solves.

The API was created for two main purposes:

  1. To enable using the solver by a group/community, with the solver running on one machine and many users running solves on it. There's at least some demand for such a thing, which made it worth spending the effort to provide it.
  2. To provide a compatibility layer so that the solver can be immediately adopted without having to make changes to existing software. The Astrometry.net online solver (nova.astrometry.net) is already supported by most astronomy software that provide solver functionality. This makes it a de-facto standard and by supporting the same interface it's easy to test and adopt Watney as well.

The API is "automatically documented" by generating an OpenAPI definition from the sources. Also SwaggerUI is available to help with testing. Examples will be provided here to help with the usage and configuration.

The Interfaces

The API contains two interfaces (or controllers in ASP.NET terms): The Watney "canonical" API (/api/watney/* routes) and the Astrometry.net compatibility API (/api/* routes). Both can be used at the same time. Enabling/disabling the compatibility API can be done via configuration.

The "canonical" Watney API was designed to be the primary API, meaning that its parameters and endpoints were designed with the solver implementation in mind. By design it is also versioned: this is to ensure that backwards compatibility remains. If later on there are changes to the API (parameters changed, fields removed or renamed, etc) a new version will be published. So the original API remains as such under /api/watney/v1 while the new version will be made available under /api/watney/v2 and so forth. This should hopefully bring some stability to the interfaces and give time for any developers to switch to use the new API version.

Watney API interface (/api/watney)

The Watney interface is a pretty minimal set of endpoints to serve the need of simply solving an image and retrieving the results. The v1 contains the following endpoints:

Route Method Description
/api/watney/v1/jobs POST Posts a new image for the solver to solve. Queues it if the solver is currently solving the max number of concurrent solves.
/api/watney/v1/jobs/{id} GET Gets the full currently available job data (input parameters, status, results).
/api/watney/v1/jobs/{id}/status GET Gets the status of the job (Queued, Solving, Success, Failure, Error, Timeout, Canceled)
/api/watney/v1/jobs/{id}/wcs GET Gets the WCS FITS file from a successful solve
/api/watney/v1/jobs/{id} DELETE Cancels the job (removes it from queue if it's waiting, or cancels the running job)

Posting a new job

Endpoint: POST /api/watney/v1/jobs

This is a multipart/form-data type endpoint, i.e. it takes the image as one parameter, and the rest of the parameters by its side as form fields.

The image is simply an file (jpg, png, fits) that is provided as the image parameter.

Then in addition to the image, we have a set of general parameters that are there regardless of whether we're performing a blind or a nearby solve. These are the args.* fields.

The rest of the parameters depend on whether you want to perform a 'blind' or a 'nearby' solve. For 'blind' solve, the fields args.blind.* are to be filled in. For a 'nearby' solve the fields args.nearby.* are to be filled in.

Example of blind solve parameters:

image = m31.jpg
args.maxStars = 300
args.mode = blind
args.sampling = 6
args.blind.minRadius = 0.5
args.blind.maxRadius = 8

And as a CURL command:

curl -X 'POST' \
  'https://localhost:5001/api/watney/v1/jobs' \
  -H 'accept: application/json' \
  -H 'apikey: demoapikey' \
  -H 'Content-Type: multipart/form-data' \
  -F 'args.blind.minRadius=0.5' \
  -F 'args.blind.maxRadius=8' \
  -F 'args.maxStars=300' \
  -F 'args.mode=blind' \
  -F 'args.sampling=6' \
  -F 'image=@m31.jpg;type=image/jpeg'

If the parameters are valid, the successful response will contain the accepted job information:

{
  "id": "tm5lx0174EuriHZUVMZC6A",
  "parameters": {
    "maxStars": 300,
    "sampling": 6,
    "lowerDensityOffset": 1,
    "higherDensityOffset": 1,
    "mode": "blind",
    "nearbyParameters": {
      "ra": null,
      "dec": null,
      "maxFieldRadius": null,
      "minFieldRadius": null,
      "intermediateFieldRadiusSteps": "0",
      "useFitsHeaders": null,
      "searchRadius": 20
    },
    "blindParameters": {
      "minRadius": 0.5,
      "maxRadius": 8,
      "raSearchOrder": null,
      "decSearchOrder": null
    }
  },
  "status": "Solving",
  "imageWidth": 4479,
  "imageHeight": 3375,
  "solution": null,
  "updated": "2022-01-28T20:15:39.0574863+00:00",
  "solveStarted": "2022-01-28T20:15:39.0572873+00:00",
  "originalFilename": "m31.jpg",
  "starsDetected": 7482,
  "starsUsed": null
}

Getting job status

Endpoint: GET /api/watney/v1/jobs/{id}/status

This is a simple GET, with the job ID returned by the new job endpoint.

CURL example:

curl -X 'GET' \
  'https://localhost:5001/api/watney/v1/jobs/tm5lx0174EuriHZUVMZC6A/status' \
  -H 'accept: application/json' \
  -H 'apikey: demoapikey'

It returns the current status in the response (Queued, Solving, Success, Failure, Error, Timeout, Canceled):

{
  "status": "Success"
}

Getting job result and information

Endpoint: GET /api/watney/v1/jobs/{id}

This is a simple GET, with the job ID returned by the new job endpoint.

CURL example:

curl -X 'GET' \
  'https://localhost:5001/api/watney/v1/jobs/tm5lx0174EuriHZUVMZC6A' \
  -H 'accept: application/json' \
  -H 'apikey: demoapikey'

It returns the full information of the job, including the input parameters and the solve result if it's available.

{
  "id": "tm5lx0174EuriHZUVMZC6A",
  "parameters": {
    "maxStars": 300,
    "sampling": 6,
    "lowerDensityOffset": 1,
    "higherDensityOffset": 1,
    "mode": "blind",
    "nearbyParameters": {
      "ra": null,
      "dec": null,
      "maxFieldRadius": null,
      "minFieldRadius": null,
      "intermediateFieldRadiusSteps": "0",
      "useFitsHeaders": null,
      "searchRadius": 20
    },
    "blindParameters": {
      "minRadius": 0.5,
      "maxRadius": 8,
      "raSearchOrder": null,
      "decSearchOrder": null
    }
  },
  "status": "Success",
  "imageWidth": 4479,
  "imageHeight": 3375,
  "solution": {
    "ra": 10.723063958252373,
    "dec": 41.25031509932796,
    "fieldRadius": 1.8667623443425037,
    "orientation": -61.80872186430443,
    "pixScale": 2.3966076382916826,
    "parity": "flipped",
    "timeSpent": 0.5666624,
    "searchIterations": 1024,
    "quadMatches": 38,
    "fitsWcs": {
      "cd1_1": 0.00031430850287252554,
      "cd1_2": -0.0005869870395382856,
      "cd2_1": 0.0005869455550211455,
      "cd2_2": 0.0003146246544076803,
      "cdelt1": 0.0006658039647952308,
      "cdelt2": 0.0006659898330658458,
      "crota1": -61.80872186430443,
      "crota2": -61.83101443764876,
      "crpix1": 2239.394729904014,
      "crpix2": 1687.1009478319327,
      "crval1": 10.723150162308315,
      "crval2": 41.25057854549978
    }
  },
  "updated": "2022-01-28T19:16:24.8004444+00:00",
  "solveStarted": "2022-01-28T19:16:24.1107664+00:00",
  "originalFilename": "m31.jpg",
  "starsDetected": 7482,
  "starsUsed": 300
}

Getting job result as WCS

Endpoint: GET /api/watney/v1/jobs/{id}/wcs

This is a simple GET, with the job ID returned by the new job endpoint. This endpoint provides the solution as a WCS FITS file, i.e. the coordinates, CRPIX & CRVAL and the CD matrix are written to FITS headers of an otherwise empty FITS file.

It returns a file (content-disposition: attachment; filename=wcs.fits; filename*=UTF-8''wcs.fits )

CURL example:

curl -X 'GET' \
  'https://localhost:5001/api/watney/v1/jobs/tm5lx0174EuriHZUVMZC6A/wcs' \
  -H 'accept: text/plain' \
  -H 'apikey: demoapikey'

Canceling a job

Endpoint: DELETE /api/watney/v1/jobs/{id}

This method/endpoint provides a way to cancel a queued or in-progress job. If the job is queued, it will be removed from the queue. If the job is already solving, the solver will be signaled to stop it.

CURL example:

curl -X 'DELETE' \
  'https://localhost:5001/api/watney/v1/jobs/abc123' \
  -H 'accept: application/json' \
  -H 'apikey: demoapikey'

The response is a simple message, regardless of what was performed in the background:

{
  "message": "Job abc123 was signaled cancellation if it was queued or running"
}

The Astrometry.net compatibility interface

This interface implementation follows the documentation laid out in Astrometry.net API documentation. The main goal is to be compatible with the online nova.astrometry.net solver but currently all extra features outside the scope of simple solving are scoped out. This means that both file uploads and URL uploads are supported and all the standard API endpoints are covered. Only the endpoints mentioned at the end of the documentation except for the WCS file endpoint are not implemented. The assumption is that for most solving tasks those endpoint are generally not required or used - if this assumption proves to be wrong, obviously these endpoints will be implemented one way or another.

The full list of endpoints provided by the compatibility API:

Route Method Notes
/api/login POST Login using API key
/api/upload POST Upload an image
/api/url_upload POST Upload an image via URL
/api/submissions/{id} GET, POST
/api/jobs/{id} GET, POST
/api/jobs/{id}/calibration GET, POST
/api/jobs/{id}/info GET, POST
/wcs_file/{id} GET, POST
/api/jobs/{id}/objects_in_field GET, POST Always returns an empty array
/api/jobs/{id}/annotations GET, POST Always returns an empty array
/api/jobs/{id}/tags GET, POST Always returns an empty array
/joblog/{id} GET, POST Always returns "no logs"

Note: POST method is accepted for all of the endpoints because after testing it seems that Nova accepts it.

For what these endpoint do, refer to the Astrometry.net API documentation.

Compatibility implications

Since the parameters for the Watney solver are not the same as for the astrometry.net solver, there are some parameters that cannot be conveyed to the solver. Fortunately, most of the basic parameters like initial coordinates, scope field radius, etc. match and for the rest if we just use defaults we should be ok. So in the end the compatibility API should work quite well. The compatibility API has been tested in live conditions using KStars/Ekos with happy results. The only real con is the speed, meaning that for example with KStars/Ekos the code which utilizes the online solver does not assume fast solves and does not poll the results so very often - so while the solver completes quickly, some seconds are wasted in the status polling process.

Configuration

The API can be configured with a simple configuration file, config.yml. It contains all configurable values in regards to the solver.

Authentication using API keys is supported, and a separate file apikeys.yml exists where you may define one or more API keys. It's also possible to run the API without authentication.

The solver can run in two "persistency modes": either keeping the data fully in-memory (queues, results and so on) or persisting them on disk. In both modes, images are never persisted - the first step is to extract stars and dimensions from them, and after that the files are discarded. Only the relevant information is kept in memory or disk, so using the in-memory mode should not be too memory intensive. In addition, cleanup is performed periodically to remove old jobs from memory/disk in order to not waste memory or space.

The configuration file itself has the configurable options documented quite well.

For a reference, here's a copy of the default configuration options at the time of writing:

# This is the configuration file for Watney solver API.
# The format of this file is YAML.

# ================================================================================
# The path to the quad database. If using relative directory, start with './'
# Examples: 
#   quadDbPath: ./db
#   quadDbPath: /opt/watney-astrometry/db
#   quadDbPath: 'C:\Program Files\watney-astrometry\db'
# ================================================================================ #
quadDbPath: './db'


# ================================================================================
# Does the API use persistency or not.
# Persistency means saving the queued and solved jobs to disk.
# If false, the jobs and the queue will be held in memory only and
# will be cleared when the process restarts.
# ================================================================================
usePersistency: false


# ================================================================================
# The directory for persistency, i.e. where jobs will be stored.
# If using relative directory, start with './'
# This is literally a work directory, and it can be emptied to free space if 
# needed.
# ================================================================================
workDirectory: './work'


# ================================================================================
# Maximum image upload size in bytes.
# ================================================================================
maxImageSizeBytes: 50000000


# ================================================================================
# The lifetime of a job, once it has been queued.
# The jobo will remain this long in memory after it has been queued
# or solved.
#
# Example 1: if the lifetime of a job is 00:10:00 (10 minutes) but the queue
# is so long that it would take 11 minutes for the job to be started, at 
# 10 minutes the job will be abandoned.
#
# Example 2: after the job has been solved, it will remain in
# memory (you can request its status and results) for this long. Once the job's 
# age passes this value, the job will automatically be removed.
#
# The format is hh:mm:ss
# ================================================================================
jobLifetime: '00:10:00'


# ================================================================================
# The timeout for a single solve task.
# If the field cannot be solved in this time, the solve task will be automatically
# aborted and the job will be marked as canceled.
# ================================================================================
solverTimeoutValue: '00:00:30'


# ================================================================================
# How many concurrent solves the API will handle.
# The more solves are running concurrently, the longer a single solve will take.
# Executing too many blind solves concurrently is generally not recommended.
# Note that load balancing between jobs is not guaranteed.
# ================================================================================
allowedConcurrentSolves: 5


# ================================================================================
# Enables Swagger and Swagger UI (OpenAPI 3 specification) if enabled.
# Mostly useful for debugging.
# ================================================================================
enableSwagger: false


# ================================================================================
# Authentication method.
# Available authentication methods are:
# - apikey
#
# If left empty, no authentication is required.
# Examples:
#   authentication: 'apikey'
#   authentication: ''
#   authentication:
# ================================================================================
authentication: 'apikey'


# ================================================================================
# API keys to use, if using API key authentication.
# This is a path to a file where the keys are defined.
# When authentication mode is set to 'apikey', this file will be read and the 
# user:apikey pairs will be used for authentication.
# When using API key authentication, an API key is required to be present in 
# all requests. It can be sent in either headers or as a query parameter.
# See the README.md for more information.
# ================================================================================
apikeys: './apikeys.yml'


# ================================================================================
# If true, enables the Astrometry.net compatibility API.
# With this turned on, you may use this API like Astrometry.net web API for
# a limited degree. Note that since this is a compatiblity API, there will be some 
# limitations in usage.
# ================================================================================
enableAstrometryNetCompatibilityApi: true

API startup arguments

In addition to the configuration file, the ASP.NET API which runs on Kestrel allows a few arguments to be given upon startup. Notably the --urls:

./watney-api --urls "http://localhost:6000;https://localhost:6001"

Or alternatively you could use the ASPNETCORE_URLS environment variable:

ASPNETCORE_URLS="http://localhost:6000;https://localhost:6001" ./watney-api

This allows you to change the port where the API runs. By default without arguments it runs on ports 5000 (http) and 5001 (https).

Table of contents

Clone this wiki locally