Skip to content

REST API Refresh

Jump to bottom
mattkrusz edited this page Mar 27, 2017 · 61 revisions

REST API

The REST API for GeoServer is popular, but not well maintained, collecting a large number of outstanding bugs. Many of these complain about functionality and lack of documentation. The large number of bugs (API called correctly but produced an error) may be producing more requests for documentation (developer assumed they called it incorrectly, and asks for improved documentation with an example that works).

Internally the REST API is written using an early java library called "restlet" (http://restlet.com/). There is a desire to migrate to spring rest api which is annotation driven and better supported. The risk in performing a migration to Spring MVC is introducing more bugs than are fixed. This is somewhat offset by having a larger pool of developers familiar with the codebase and the technologies used.

This activity will require quite a bit of people (all hands on deck).

Reference:

Rough plan:

Create gs-rest-ng

Migrate existing rest functionality

See the full list of endpoints here.

  1. Migrate core functionality

  2. GWC compatibility (this is likely to need a dedicated resource with commit access)

    • gwc-sqlite (currently uses mvc, multiple mvc contexts are problematic)

  3. Migrate extensions

    • importer - may need its own team, initial evaluation is "not too bad"

  4. Migrate community modules (the ones that compile, priority to communityRelease modules)

    • authkey
    • script
    • rest-ext
    • jms-geoserver
    • rest-upload
    • geofence - has an existing spring mvc to look out for
    • sldservice
    • geogig - this has its own rest api which we depend on, this will require a team with geogig commit
    • backuprestore-rest
    • notification

  5. Test cases should remain unchanged to verify no regressions

    • Initial review shows test-coverage for XML is pretty good, json and html are poor
    • IMPORTANT: For each endpoint - ensure that one deserialize/serialize test case is in place prior to migrating

Documentation

  1. Set up documentation team operating concurrently, capturing each resource as it is completed

    • Add any working cURL examples to google doc for documentation team to use as inspiration

  2. For REST API examples - go for a resource description, JSON and XML example (potentially HTML)

Audit functionality against GUI

As time permits make a note of any functionality missing from the REST API:

  • Shortlist missing functionality for proposal and implementation (examples recalculate feature type columns, rest layer bounds from SRS bounds, ...)
  • If there is already a JIRA issue, add the link to spreadsheet above

Set up Tests prior to Migration

We are looking for two tests:

  1. Regression Tests with xml files committed to the codebase (for use as inline documentation example)

    • Reference XML Request for CREATE
    • Reference XML Response for GET
    • Reference JSON Request for CREATE
    • Refernece JSON Response for GET

  2. Serialization / Deserialization test

    • This captures 90% of any XStream issues

Migration Approach and Notes

These are loose notes on converting the GeoServer REST API to Spring MVC. They are derived from the experience of converting the existing Styles end point. This document is meant to be a companion to the actual code.

Approach

The general approach that I found most useful was:

  • Copy over the Unit Tests for the end point you're working on.
  • Update the URLs
  • Run, watch all the failures.
  • Create your new end point and fill it in, running the test cases as needed.
  • In general, start with the POST/PUT end points. Those tend to be the trickiest. GET and DELETE are simpler.

Sample Branch

The sample Spring MVC conversion branch is here:

https://github.com/geoserver/geoserver/tree/rest-api-refresh

Spring MVC Documentation

The Spring MVC reference docs are decent and worth at least a skimming.

https://docs.spring.io/spring/docs/current/spring-framework-reference/html/mvc.html

Architecture

The basic parts of the architecture are as follows. More information can be found in the JavaDocs for each object.

Controllers

These are the main REST request handlers. They map roughly to the existing *Resource classes in the existing rest module. For example

@GetMapping(
    path = "/styles/{styleName}",
    produces = {
        MediaType.APPLICATION_JSON_VALUE, 
        MediaType.APPLICATION_XML_VALUE, 
        MediaType.TEXT_HTML_VALUE})
protected RestWrapper<StyleInfo> getStyle(
        @PathVariable String styleName) {
    return wrapObject(getStyleInternal(styleName, null), StyleInfo.class);
}

HttpMessageConverters

These are responsible for serialization and deserialization of response objects. These correlate to the *Format objects in the existing REST API. In most cases these just need to tie into our existing serialization (XStreamPersister).

/**
 * Message converter implementation for JSON serialization via XStream
 */
public class JSONMessageConverter extends BaseMessageConverter {

    public JSONMessageConverter(ApplicationContext applicationContext) {
        super(applicationContext);
    }
    @Override
    public boolean canRead(Class clazz, MediaType mediaType) {
        return !XStreamListWrapper.class.isAssignableFrom(clazz) &&
            MediaType.APPLICATION_JSON.equals(mediaType);
    }
    @Override
    public boolean canWrite(Class clazz, MediaType mediaType) {
        return !XStreamListWrapper.class.isAssignableFrom(clazz) &&
            MediaType.APPLICATION_JSON.equals(mediaType);
    }
    @Override
    public List<MediaType> getSupportedMediaTypes() {
        return Arrays.asList(MediaType.APPLICATION_JSON);
    }
    @Override
    public Object read(Class clazz, HttpInputMessage inputMessage)
        throws IOException, HttpMessageNotReadableException
    {
        XStreamPersister p = xpf.createJSONPersister();
        p.setCatalog(catalog);
        return p.load(inputMessage.getBody(), clazz);
    }
    @Override
    public void write(Object o, MediaType contentType, HttpOutputMessage outputMessage)
        throws IOException, HttpMessageNotWritableException {
        XStreamPersister xmlPersister = xpf.createJSONPersister();
        xmlPersister.setCatalog(catalog);
        xmlPersister.setReferenceByName(true);
        xmlPersister.setExcludeIds();
        xmlPersister.save(o, outputMessage.getBody());
    }

}

RestWrapper

RestWrappers are used to provide additional configuration used by the converters alongside the objects returned by the controllers. The base controller class RestController provides utility methods wrapObject and wrapList for constructing these wrappers. All objects that get serialized by the XStreamXmlConverter, XStreamJSONConverter, or FreemarkerHtmlConverter should be wrapped (this generally only applies to GETs).

MVC configuration

MVCConfiguration is the class responsible for doing Spring MVC configuration. In particular adding converters, configuring content type negotiation, and adding intercepters. See the documentation here:

http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/servlet/config/annotation/WebMvcConfigurationSupport.html

Controller Advice

RestControllerAdvice.java is primarily used to configure error message handling, but it can also be used for a number of things. See here for more controller advice functionality.

@ExceptionHandler(RestException.class)
public void handleRestException(RestException e, HttpServletResponse response, WebRequest request, OutputStream os)
    throws IOException {
    response.setStatus(e.getStatus().value());
    StreamUtils.copy(e.getMessage(), Charset.forName("UTF-8"), os);
}

HTML Output

The REST HTML format works slightly differently from the XML and JSON formats.

First of all, the HTML format only supports GET requests.

HTML output is generated using a Freemarker template (*.ftl) file. These files will generally exist alongside the controller code. Because of this, HTML gets require a bit of additional context. This is achieved by wrapping the response object in a FreemarkerConfigurationWrapper when returning from the controller. The RestController base class provides some utility methods for constructing this wrapper. This does mean that a separate get method is required for all controller endpoints that support HTML output.

For example:

@RequestMapping(
    path = "/styles/{styleName}", 
    method = RequestMethod.GET, 
    produces = {MediaType.TEXT_HTML_VALUE})
protected FreemarkerConfigurationWrapper getStyleFreemarker(
    @PathVariable String styleName) 
{
    return toFreemarkerMap(getStyleInternal(styleName, null)); //return FreemarkerConfigurationWrapper containing style object
}

Debugging Tips

Controller not hit/Response Code 415

The most common issue I've run into during the conversion was the handler method not being hit at all. This usually results in a response code 415 from Spring (media type not accepted). Debugging this ranges from simple to aggravating. Here are a few tips, from most obvious to least:

  • Is the request path correct?
  • Does your request Content-Type match the "consumes" parameter of the handler
  • Are all your path elements matched correctly?
  • Is the HttpMessageConverter you expect to be hit -- based on the requested content type -- actually being invoked? Be sure to check the canWrite/canRead method to see that it's returning true as expected.
  • Are you requesting something via extension (say .zip) that can't actually be produced (ie. POSTING to .zip when the controller only produces XML)
  • org.springframework.web.servlet.handler.AbstractHandlerMethodMapping#addMatchingMappings: This method goes through all the handlers to find the one that matches. Useful for debugging why a controller isn't being hit (415 response code). Digging around here is your last resort to find out WHY a specific handler is being rejected.

REST Documentation

And there is an outstanding bug to "improve" the documentation GEOS-7931.

Sphinx Documentation

The existing documentation is written in RST, reviewing the 70+ REST API tickets documentation fixes may close a large portion of them (as many of are due to documentation confusion).

The MapBox REST API provides a good template with info / curl example / request / response. Compare:

Approach:

  1. Review the REST API tickets and prioritize what to focus on for rest api docs.

  2. Decide if we should document the protocol in RST by hand, or if we can use a documentation generator.

    • RST: Refactor current REST API content into a more approachable presentation for each REST Endpoint (syntax, resource structure (xml,json), supported request types, response types/codes)
    • Documentation Generation: link from user manual

    Decision: Hand update the RST, but using inline examples from test cases in the code.

  3. In both cases we will need to maintain specific REST API examples in the user manual, goal is to have cut and paste examples.

    • We will have to decide what to do with the more complex examples that use several different endpoints. They can probably remain in a dedicated examples section, or broken out into a tutorial.

REST API Examples

The user manual includes some examples directly:

And library examples are available online:

Automated Documentation Generation

There are a number of tools available for Spring MVC:

Swagger

A swagger "document" describes a REST interface:

  • Can be used to generate Spring MVC (if you are making a new API from scratch)
  • Can be used to generate human readable documentation
  • There are editors http://editor.swagger.io/

The "document" can be play tested, there are open source tools you can use to try out sample requests prior to code generation/documentation.

For our purposes a swagger "document" can be generated from annotations (the spring-fox extension is widely used can read spring mvc annotations, and has extra annotations for descriptions):

Choose Spring-fox Annotations?

Decision: We are not using Spring-fox, because it relies on static content, but Xstream is doing its serialization programmatically.

Pros: - we have been able to work with annotations for process definition

Cons: - is spring-fox stable enough for geoserver (will it be around in five years)

Swagger

Decision: We are going with Swagger.

Pros:

  • easy to edit and update the docs
  • Uses Maven to generate output, so integrates well with workflow

Cons:

  • Is swagger stable enough for GeoServer (will it be around in five years)

Examples of Swagger

  • See the /styles endpoint, written in Swagger, committed to the rest-api-refresh branch: styles.yaml

  • The following example can be cut and pasted into the http://editor.swagger.io/ allowing you to play test the api and the doc generation.

swagger: '2.0'
info:
  version: 1.0.0
  title: GeoServer REST API
  description: |
    **This example is for the workspace rest api endpoint**

    You can try all HTTP operation described in this Swagger spec.
    
    Find source code of this API [here](https://github.com/mohsen1/petstore-api)
host: petstore-api.herokuapp.com
basePath: /
schemes:
  - http
  - https
consumes:
  - application/json
  - text/xml
produces:
  - application/json
  - text/xml
  - text/html
paths:
  /geoserver/rest/workspaces:
    get:
      parameters:
        - name: limit
          in: query
          description: number of workspaces to list
          type: integer
          default: 10000
          minimum: 1
          maximum: 10000
      responses:
        200:
          description:  List all workspaces
          schema:
            title: Workspaces
            type: array
            items:
              $ref: '#/definitions/WorkspaceList'
    post:
      parameters:
        - name: workspace
          in: body
          description: The workspace XML or JSON you want to post
          schema:
            $ref: '#/definitions/Workspace'
          required: true
      responses:
        200:
          description: Make a new pet
    put:
      parameters:
        - name: workspace
          in: body
          description: The workspace XML or JSON you want to post
          schema:
            $ref: '#/definitions/Workspace'
          required: true
      responses:
        200:
          description: Updates the workspace
  /geoserver/rest/workspaces/{workspaceId}:
    get:
      parameters:
        - name: workspaceId
          in: path
          type: string
          description: ID of the workspace
          required: true
      responses:
        200:
          description: Sends the workspace with workspace Id

definitions:
  WorkspaceList:
    type: array
    items: 
      $ref:  '#/definitions/WorkspaceListItem'
  WorkspaceListItem:
    type: object
    required: [name,href]
    properties:
      name:
        type: string
      href:
        type: string
      default:
        type: boolean
  Workspace:
    type: object
    required: [name,dataStores,coverageStores,wmsStores]
    properties:
      name:
        type: string
        description: name of workspace
      default:
        type: boolean
      dataStores:
        type: string
      coverageStores:
        type: string
      wmsStores:
        type: string

Here is what that looks like:

swagger editor example for workspace

Swagger Implementation

Location: Swagger API documentation lives in /doc/en/src/main/resources/api.

Build: Use the new pom.xml in /doc/en to execute mvn clean process-resources, which will build only the swagger docs without also building the sphinx docs.

This goal runs swagger-codegen-maven-plugin to generate an html output using a default template. See swagger-codegen and swagger-codegen-maven-plugin. The default templates can be customized or replaced completely, and custom swagger-codegen generators can be written to change how we render the parsed swagger document.

Output: The generated API docs are built to /doc/en/target/api

How to document an endpoint:

The parent swagger file is /doc/en/target/api/api.yaml, and this file can reference components and definitions provided in other files.

For example, to add a new endpoint for /workspaces, you could add:

# api.yaml
# ...
paths:
  # ...
  /workspaces:
    $ref: "./workspaces/workspaces.yaml#/paths/Workspaces"
  /workspaces/{workspaceName}:
    $ref: "./workspaces/workspaces.yaml#/paths/WorkspacesByName"

In this ref, the part before the # refers to a relative file path, and the part after the # refers to a definition inside the file. The above would expect a file at /doc/en/target/api/workspaces/workspaces.yaml that contains at least the following:

paths:
  Workspaces:
    # ... 
    # e.g., get: ...  
  WorkspacesByName
    # ...
    # e.g., get: ...

Note that the way you choose to structure the referenced yaml file is up to you. (The 'paths' lookup could be something else, or it could be left out, in which case the $ref would be "./workspaces/workspaces.yaml#Workspaces.)

These yaml files can also reference further yaml files if it's necessary to break things up further. In general, you can break out definitions for anything in the spec that allows a $ref value.

See the existing /styles endpoints for reference.

Reference

Swagger Specification

LOOK OUT When defining a parameter, the "in" property MUST BE LOWER CASE due to a bug in swagger-codegen:

        - name: styleName
          in: path # DEFINITELY NOT: Path
          required: true
          description: The name of the style to retrieve.
          type: string

©2020 Open Source Geospatial Foundation

Home
Proposals
Release Schedule

Clone this wiki locally