GA4GH Service Registry API implementation for the ELIXIR Cloud. The registry is currently deployed at https://cloud-registry.rahtiapp.fi, but will likely move soon.
Service entries comply with the ExternalService schema
defined in the GA4GH Service Registry API.
For developers, an API documentation (of the code, not the exposed API) is available here.
Once deployed, the API will be served here:
http://localhost:8080/ga4gh/registry/v1/
You can explore the service via the Swagger UI.
The app listens on the port set in the config (
server.port);docker-compose.yamlpublishes it on the host. If you change either, update the URLs above accordingly.
Endpoints can be probed with curl, for example:
curl -X GET "http://localhost:8080/ga4gh/registry/v1/services" -H "accept: application/json"The app is configured by cloud_registry/config.yaml. To override
some of its values, write them to a config overlay file and list it in
environment variable FOCA_CONFIG_OVERLAY (several files separated by :);
FOCA merges overlays into the config on startup. The
Helm chart uses the same mechanism.
Each deployment must set two values that depend on it, otherwise the app refuses to start:
security.auth.audience: the client IDs of the apps whose LS Login tokens are accepted (e.g., the registry's front end);security.access_control.group_mapping: the group(s) whose members are admins (see Access control), orcustom.access_control.require_admin: falseto run without admins.
Docker Compose mounts config.dev.yaml, an overlay for local
development that accepts tokens for any audience and requires no admins. To
override more values locally, put them in config.local.yaml and mount it
in a docker-compose.override.yaml (both are ignored by Git):
services:
cloud-registry:
environment:
FOCA_CONFIG_OVERLAY: /etc/cloud-registry/config.dev.yaml:/etc/cloud-registry/config.local.yaml
volumes:
- ./config.local.yaml:/etc/cloud-registry/config.local.yaml:roGET /service-info serves the service info from the config
(custom.endpoints.service_info), without the database. On startup, the app
checks it against the GA4GH Service schema and refuses to start if it is
invalid, naming the invalid field. version defaults to the version of the
installed app; createdAt and updatedAt are served only if set, and must be
RFC 3339 date-times (e.g., 2020-11-08T00:00:00Z). Fields and list items set
to null (e.g., in a config overlay) are not served. To change the service
info, change the config and restart the app (the Helm chart restarts it when
the config value changes).
Previously, service info could be set via POST /service-info and was stored
in the database. That endpoint is gone, and stored service info is no longer
served (the service_info collection can be dropped): put it in the config
when upgrading.
Public URLs, e.g., in Location headers, are built from
custom.endpoints.service in config.yaml. Set api_path to the path prefix
in front of the API base path when the app is served behind a reverse proxy
under a subpath (e.g., registry), or leave it empty. Previously, api_path
held the API base path itself (e.g., ga4gh/registry/v1); remove that value
when upgrading.
Services and service info can be read by anyone. Writing requires an access token issued by LS Login:
-
Any authenticated user can register a service (
POST /services) and becomes its owner. -
Owners can update (
PUT) and delete (DELETE) their services, and grant other users the rolesownerormaintaineron them, by their LS Login user identifier (subclaim):curl -X PUT "http://localhost:8080/ga4gh/registry/v1/services/<id>/permissions/<user-id>" \ -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \ -d '{"role": "maintainer"}'
GET /services/<id>/permissionslists the roles granted on a service (with the users' names, once they have used the registry since being granted the role; names are deleted with the role), andDELETE /services/<id>/permissions/<user-id>revokes them. The last owner of a service can't be removed or downgraded (409 Conflict); delete the service instead. -
Maintainers can update, but not delete a service.
-
Members of the groups that the deployment maps to the role
admin(security.access_control.group_mapping) are admins: they can do everything, including registering services with chosen identifiers (PUTof a new service). For ELIXIR, that is the LS Login groupECP_CLN:ADMIN:security: access_control: group_mapping: "urn:geant:lifescience-ri.eu:group:elixir:ECP_CLN:ADMIN#aai.lifescience-ri.eu": admin
On startup, the app logs the groups mapped to
admin.
GET /me returns the current user's identifier (to give to owners), whether
they are an admin, and their roles on services; it also stores their name for
these services. Owners are not shown in the
public service responses.
Access control is implemented with FOCA; see the
security section of the configuration.
A service is identified by its base URL (url) and its API type (type.group
and type.artifact, case-insensitive, but not type.version, so that
upgrading a service to a new API version updates its entry). URLs are compared
after normalizing common spellings: lowercase schemes and hosts, encoded
international host names (IDNA 2008), canonical IPv6 addresses, percent-encoded
non-ASCII characters in paths, and without trailing dots of hosts, default
ports, dot segments (/./, /../), repeated and trailing slashes, needless
percent-encoding in paths, queries and fragments. URLs with other schemes
(e.g., http and https) are different services. Invalid URLs, URLs without
scheme or host, and URLs with user info (user:password@) are rejected (400 Bad Request, saying why).
Registering a service again (POST /services), or changing a service to the
URL and API type of another one (PUT), returns 409 Conflict, with the
identifier of the registered service in serviceId, so that users can
contact its owner or an admin.
As anyone logged in can register a URL first, including one of a deployment
they don't operate, a squatter could block the real operator of a service.
Admins resolve such cases: they can delete the entry, or grant the real
operator the role owner on it and revoke the squatter's role. Consumers that
need trustworthy entries should rely on entries checked by admins (a
"verified" flag is planned, see #64). Proving control of a
URL (e.g., by fetching {url}/service-info on registration) was rejected:
the identifier in service info is self-declared, and fetching user-supplied
URLs would allow server-side request forgery.
Services stored before this check get their key on startup. If some of them
have invalid URLs, or the same URL and API type as another one, the app
refuses to start and names all of them. Fix or delete them in the database,
e.g., with mongosh (see the deployment instructions
for access to the database of a Helm deployment):
db.getSiblingDB("serviceStore").services.deleteOne({id: "<id>"})
db.getSiblingDB("serviceStore").services.updateOne({id: "<id>"}, {$set: {url: "<URL>"}})Then start the app again; it adds the missing keys.
To quickly install the service for development/testing purposes, we recommend
deployment via Docker Compose, as described below. For
more durable deployments on cloud native infrastructure, we also provide a
Helm chart in the deployment/ directory, together with
basic deployment instructions (details may need to
be adapted for your specific infrastructure).
The following software needs to be available on your system:
git 2.17.1docker, with the Compose plugin (version 2)
Where versions are indicated, they were used to test deployment. Other versions with the same major version numbers will likely work as well, especially newer ones.
First, clone the repository and traverse into the root directory with:
git clone git@github.com:elixir-cloud-aai/cloud-registry.git
cd cloud-registryThen simply start up the service with:
docker compose up --build -dMongoDB is only reachable from the app container; its data is kept in the
Compose project's mongodb-data volume (e.g., cloud-registry_mongodb-data,
see docker volume ls).
Earlier versions kept the data in
../data/cloud-registry/db. To keep using it, bind-mount it in adocker-compose.override.yaml:services: mongodb: volumes: - ../data/cloud-registry/db:/data/db
That't it!
You should now be able to use/explore the API as described in the usage section.
To shut down the service, run:
docker compose downAdd --volumes to also delete the database.
If you need to inspect the logs, call:
docker compose logsThis project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please refer to our organization's contribution guidelines if you are interested to contribute. Please mind the code of conduct for all interactions with the community.
To set up a development environment and run the checks that CI runs:
pip install -r requirements.txt -r requirements_dev.txt
pre-commit install # run the checks on every commit
pre-commit run --all-files # ruff, mypy, semgrep, ast-grep
pre-commit run --all-files --hook-stage manual gitleaks-all # secrets
pytestRun pre-commit in the environment with the dev requirements: the mypy and
ast-grep hooks use the installed packages. The first run sets up the other
hooks, which takes a few minutes.
The checks are configured in .pre-commit-config.yaml and pyproject.toml.
Project-specific rules are in .semgrep/ (security) and .ast-grep/
(anti-patterns such as HTTP status codes as strings). To suppress a false
positive, add a comment with the reason to the line: # noqa: <code> (ruff),
# nosemgrep: <rule-id>, # ast-grep-ignore: <rule-id> or
# gitleaks:allow.
The project adopts the semantic versioning scheme for versioning. Currently the service is in beta stage, so the API may change without further notice.
This project is covered by the Apache License 2.0 also shipped with this repository.
The project is a collaborative effort under the umbrella of ELIXIR Cloud & AAI. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.