Skip to content

Creating Custom Plugins

Jump to bottom
Tim Gion edited this page Nov 1, 2018 · 5 revisions

Overview

The Ingest Client uses "plugins" to support the many different ways users can store and organize their data. The ingest service works by computing "upload tasks" based on the ingest job configuration file, where an upload task specifies a single image tile to upload to the Boss. Upload tasks contain tile indices in x, y, z, and t that specify the target tile to upload. Plugins are responsible for interpreting the tile indices and providing to the ingest client a file handle for the correct bit of data.

To learn more about plugins that are provided by default with the ingest client, view the Plugins wiki page.

Ingest Types

There are currently two types of ingests: tile based and volumetric.

Tile based ingests use these processors:

  • Path
  • Tile

Volumetric ingests use these processors:

  • Path
  • Chunk

Path Processor

The Path Processor is responsible for processing tile indices and returning an absolute path to the file that contains the data to be uploaded for a given upload task. This path is then provided to the Tile Processor. If your data is organized in a way that does not directly need to point to a file, you can simply return an empty string.

The abstract parent class PathProcessor contains two methods that must be implemented in any path processor plugin. The PathProcessor.setup() method provides an opportunity to perform any one time configuration or setup of the plugin. It is passed any custom parameters specified in the ingest job configuration file and is called once on the loading of the plugin, before any uploading tasks are processed. The PathProcessor.process() method is where tile indices are converted to a file path.

Tile Processor

The Tile Processor is responsible for processing tile indices and the file path generated by the Path Processor and returning a file handle to the data to be uploaded for a given upload task. This handle is then used by the ingest client to complete the upload task.

Currently, the file handle must be to a single PNG or TIFF encoded image file that is of the tile size specified in the ingest configuration file.

The abstract parent class TileProcessor contains two methods that must be implemented in any tile processor plugin. The TileProcessor.setup() method provides an opportunity to perform any one time configuration or setup of the plugin. It is passed any custom parameters specified in the ingest job configuration file and is called once on the loading of the plugin, before any uploading tasks are processed.

The TileProcessor.process() method is where the file path and tile indices are converted to a file handle. This is where customization in file format and storage can be handled, with some examples being:

  • Open a JPEG2000 file, re-encode as a PNG file in memory, and return handle to in-memory PNG file
  • Open an HDF5 file containing an image volume, re-encode tile as a PNG file in memory, and return handle to in-memory PNG file
  • Query on-prem database for data and return handle to in-memory PNG file

Chunk Processor

The Chunk Processor supports volumetric ingests. It accepts a file path and chunk indices and returns a Numpy array with the 3D chunk of the data specified by the chunk indices.

The abstract parent class ChunkProcessor contains two methods that must be implemented by any chunk processor plugin. The ChunkProcessor.setup() method provides an opportunity to perform any one time configuration or setup of the plugin. It is passed any custom parameters specified in the ingest job configuration file and is called once on the loading of the plugin, before any uploading tasks are processed.

The ChunkProcessor.process() method is where the file path and chunk indices are converted to a Numpy array. This is where customization in file format and/or storage can be handled.

Creating a Custom Plugin

Developing a custom plugin is simple:

  • Write your own PathProcessor, TileProcessor and ChunkProcessor classes to handle your unique data organization and file format requirements

    • Be sure to inherit from the abstract PathProcessor, TileProcessor, and ChunkProcessor classes as appropriate for your ingest type
    • You must implement both the setup() and process() methods in each, even if in your implementation they are not all needed

  • Add your plugins to your ingest job configuration file

    • Set client.path_processor.class to the absolute import statement for your custom PathProcessor class
    • Set client.tile_processor.class to the absolute import statement for your custom TileProcessor class for a tile based ingest
    • Set client.chunk_processor.class to the absolute import statement for your custom ChunkProcessor class for a volumetric ingest

  • Make sure your plugin classes are on your Python path

    • The ingest client will automatically import and load your custom classes, but to do this they must be on the Python path

    • On Linux or OSX systems:

      export PYTHONPATH=$PYTHONPATH:/<path_to_classes>

  • Run the ingest client

Contributing a Custom Plugin

While we are still waiting for development of the ingest client and service to stabilize, in the future you will be able to contribute plugins through pull requests. More details on this process will be provided once pull requests will be accepted.

Utilities

  • Coming soon
Clone this wiki locally