Setup for database 'meta-db'

Introduction

This Docker image manages the database migration scripts for the 'meta-db' database used by MIP.

This database contains the metadata used for reference, including:

• the list of variables and groups for the Common Data Elements (CDE) defined by MIP.
• the list of variables and groups for other datasets available in MIP.

It uses Flyway to perform the versioned database migrations.

Usage

Run:

$ docker run -i -t --rm -e FLYWAY_HOST=`hostname` hbpmip/meta-db-setup:2.5.0 migrate

where the environment variables are:

• FLYWAY_HOST: database host, default to 'db'.
• FLYWAY_PORT: database port, default to 5432.
• FLYWAY_DATABASE_NAME: name of the database or schema, default to 'meta'
• FLYWAY_URL: JDBC url to the database, constructed by default from FLYWAY_DBMS, FLYWAY_HOST, FLYWAY_PORT and FLYWAY_DATABASE_NAME
• FLYWAY_DRIVER: Fully qualified classname of the jdbc driver (autodetected by default based on flyway.url)
• FLYWAY_USER: database user, default to 'meta'.
• FLYWAY_PASSWORD: database password, default to 'meta'.
• FLYWAY_SCHEMAS: Optional, comma-separated list of schemas managed by Flyway
• FLYWAY_TABLE: Optional, name of Flyway's metadata table (default: schema_version)

Child images

Child images should follow the following procedure to be able to load their metadata:

List of Data Elements (aka hierarchy of variables)

Define the environment variable TAXONOMIES defining a space-separated list of data elements definitions, where each element definition is of the form [data elements name]|[target table]|[list of groupings for histograms view].

• data element name is the name of this organisation of variables. It should point to file /src/variables/[data elements name].json located inside the Docker image.
• target table should be the name of the table or view to use to retrieve features for algorithms and data exploration.
• list of groupings for histograms view should be a comma separated list of columns in the target table that defines a default breakdown of a dataset into several histograms with groupings, where each column defined here will create an histogram grouping values on the group by column.

For example,

  ENV TAXONOMIES=test-set|main_features_table|dataset,gender,agegroup,alzheimerbroadcategory

Then for each data element definition, place a file named [data elements name].json into folder /src/variables/ describing the hierarchy of variables and following schema variables_schema.json

List of Json patches to apply to existing Data Elements definitions

Define the environment variable TAXONOMY_PATCHES defining a space-separated list of Json patches to apply to existing data elements, where each patch description has the form [data elements name to patch]|[new data elements name]|[target table]|[list of groupings for histograms view].

• data elements name to patch is the name of the original organisation of variables that we want to update.
• new data elements name is the name of the new organisation of variables. It should point to file /src/patches/[data elements name to patch].patch.json located inside the Docker image and containing the JSON patch definition to apply to existing list of variables for [data elements name to patch].
• target table should be the name of the new table or view to use to retrieve features for algorithms and data exploration.
• list of groupings for histograms view should be a comma separated list of columns in the target table that defines a default breakdown of a dataset into several histograms with groupings, where each column defined here will create an histogram grouping values on the group by column.

For example,

  ENV TAXONOMY_PATCHES=test-set|test-set-with-custom-vars|main_features_table|dataset,gender,custom_category

Each new data elements definition, target table and hierarchy json will be inserted into the meta table. For each new data elements, we generate the json describing the hierarchy of variables by taking the hierarchy json from the original data elements definition and applying to it a Json patch loaded from the container folder /src/patches/ and named [new data elements name].json

Build

Run: ./build.sh

Publish on Docker Hub

Run: ./publish.sh

License

Meta-db-setup

(this project)

Copyright (C) 2017 LREN CHUV

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Apache-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Flyway

Copyright (C) 2016-2017 Boxfuse GmbH

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Trademark

Flyway is a registered trademark of Boxfuse GmbH.

Contribution policy

Contributions via GitHub pull requests are gladly accepted from their original author. Along with any pull requests, please state that the contribution is your original work and that you license the work to the project under the project's open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project's open source license and warrant that you have the legal authority to do so.

Acknowledgements

This work has been funded by the European Union Seventh Framework Program (FP7/2007­2013) under grant agreement no. 604102 (HBP)

This work is part of SP8 of the Human Brain Project (SGA1).