FAIR Digital Object Lab

The FAIR DO Lab Software Stack is an extendable and adjustable architecture for generic FAIR Digital Object (FAIR DO) tasks. It consists of a set of interacting components with services and tools for creation, validation, discovery, curation, and more. It is the successor of the FAIR DO Ecosystem Testbed, but has the main goal to get to a production-ready state soon. Additionally, instead of being a set of background services, we aim to integrate generic tools directly to ease the interaction with FAIR DOs.

Current high-level functionality roadmap:

• Create, update, retrieve and validate FAIR DOs (status: done)
• Search within PID records (status: done)
• Notification of PID activities for other services (status: done)
• Includes an instance of our collection registry to create and manage collections (status: done)
• browse FAIR DOs (status: done)
• curate FAIR DOs (status: planning phase)

Related Publications:

• Poster from RDA VP17 Poster Session (about its precedessor, the testbed)

Goals

• be a production-ready, customizable FAIR DO Lab
• offer generic tools for the direct interaction with FAIR DOs
• offer important components for integration with existing tools (APIs)

Components / Status

component	status
FAIRDOscope (a FAIR DO Browser)	production-ready
Typed PID Maker	production-ready
Messaging of FAIR DO activities	production-ready
Collection Registry	production-ready
Elasticsearch (search index)	production-ready, external development
Kibana (Elasticsearch management)	production-ready, external development
RabbitMQ (Message broker)	production-ready, external development

"external development": These services are reused open-source components, not developed in our department.

Getting Started

You will need:

• Git to clone this repo. Alternatively, download it as a zip file to skip this requirement.

• Docker Desktop (Installation on Windows, Linux, macOS, macOS on ARM).

  Note

  Before you start, make sure your docker environment has at least 4 GB of memory (RAM) available. This is especially required if it runs in a virtual machine (which is the case e.g., if you use Apples macOS). You can do this the easiest using Docker Desktop in the preference menu under "Resources". The FAIR DO Lab consists of several services, and in sum the default of 2 GB RAM will not be enough.

  Note

  If the docker compose command (preferred) is not available, you can use docker-compose (note the "-") as a fallback. This may be the case on older systems, for example.

All other requirements will be available through the docker setup.

Using the command line, the Lab can be started like this:

# Skip this command if you download the zip file instead
git clone https://github.com/kit-data-manager/FAIR-DO-Lab.git
cd fair-do-lab
# start all services defined in docker-compose.yml
docker compose up
# use `docker-compose up` (note the dash), if "docker compose" (preferred) is not available.

Wait until the containers started. After that, you may explore the possibilities of the FAIR DO Lab using the following URLs:

• FAIRDOscope
  • To browse real FAIR DOs in a graphical and human readable way.
  • The FAIRDOscope does currently not support sandboxed PIDs.
• PIT-Service API Documentation: http://localhost:8090/swagger-ui.html
  • Use this API to manage PID records. You can find details about the PIT-Service here.
• Message Broker Management UI
  • The message broker will receive messages from the PIT service when records are created or updated. The broker will persist those messages in the queues of interested (registered) clients.
• Search Index Management UI: http://localhost:5601/app/kibana
  • Use this interface to create a view on the record index and create queries. There are also APIs available directly to the elasticsearch index, which enables developers to create different kinds of search experiences.
• Collection Service API Documentation: http://localhost:8091/swagger-ui.html
  • Use this API to manage collections. You can find details about the collection service here.
• Collection Service Content Map: http://localhost:8091/static/overview.html
  • If you registered Collections using the API, you can get an overview of the registered collections using this interface.

Customization

Detailed usage documentation beyond the short "getting started" introduction.

The FAIR DO Lab consists of several services, managed with docker-compose. Some of them do not yet have an image on docker hub and need to be built on the machine. This will happen automatically when starting them.

Use docker compose up to start all services or docker compose up fairris pit-service to only start a selection, analogous to the build steps.

Update services

git pull
docker compose down  # shut down all services and delete its containers
docker compose pull  # pull new image versions
docker compose up    # starts all services

Repository structure

• The documentation folder contains advanced documentation about the FAIR DO Lab and some workflows within it for some better understanding of the architecture.
• Service-specific files are in folders named after the service. The configurations inside the folders are used either at build time or container creation time. So if you want to change the configuration, you need to make sure to execute the corresponding action afterwards, in order to use the new configurations.
• docker-compose.yml contains all services and how they are wired together.
  • It is recommended to use override files to make custom modifications. It will make pulling new versions of the repository easier for you.
  • Use the compose file as a documentation of exposed ports etc.
• java-release-run.sh will be copied into the images of java-based services to run them properly (helper script).