Compass is a search and discovery engine built for querying application deployments, datasets and meta resources. It can also optionally track data flow relationships between these resources and allow the user to view a representation of the data flow graph.
Discover why users choose Compass as their main data discovery and lineage service
- Full text search Faster and better search results powered by ElasticSearch full text search capability.
- Search Tuning Narrow down your search results by adding filters, getting your crisp results.
- Data Lineage Understand the relationship between metadata with data lineage interface.
- Scale: Compass scales in an instant, both vertically and horizontally for high performance.
- Extensibility: Add your own metadata types and resources to support wide variety of metadata.
- Runtime: Compass can run inside VMs or containers in a fully managed runtime environment like kubernetes.
Explore the following resources to get started with Compass:
- Guides provides guidance on ingesting and querying metadata from Compass.
- Concepts describes all important Compass concepts.
- Reference contains details about configurations, metrics and other aspects of Compass.
- Contribute contains resources for anyone who wants to contribute to Compass.
Compass is written in Golang, and requires go version >= 1.16. Please make sure that the go toolchain is available on your machine. See Golang’s documentation for installation instructions.
Alternatively, you can use docker to build Compass as a docker image. More on this in the next section.
Compass uses PostgreSQL 13 as its main storage and Elasticsearch v7 as the secondary storage to power the search. In order to run compass locally, you’ll need to have an instance of postgres and elasticsearch running. You can either download them and run it manually, or you can run them inside docker by using docker-compose with docker-compose.yaml provided in the root of this project.
PostgreSQL details and Elasticsearch brokers can alternatively be specified via the environment variable, ELASTICSEARCH_BROKERS for elasticsearch and DB_HOST, DB_NAME, DB_USER, DB_PASSWORD for postgres.
If you use Docker to build compass, then configuring networking requires extra steps. Following is one of doing it by running postgres and elasticsearch inside with docker-compose first.
Go to the root of this project and run docker-compose.
$ docker-compose up
Once postgres and elasticsearch has been ready, we can run Compass by passing in the config of postgres and elasticsearch defined in docker-compose.yaml file.
Begin by cloning this repository then you have two ways in which you can build compass
- As a native executable
- As a docker image
To build compass as a native executable, run make inside the cloned repository.
$ make
This will create the compass binary in the root directory
Building compass' Docker image is just a simple, just run docker build command and optionally name the image
$ docker build . -t compass
Before serving Compass app, we need to run the migration first. Run this docker command to migrate Compass.
$ docker run --rm --net compass_storage -p 8080:8080 -e ELASTICSEARCH_BROKERS=http://es:9200 -e DB_HOST=postgres -e DB_PORT=5432 -e DB_NAME=compass -e DB_USER=compass -e DB_PASSWORD=compass_password odpf/compass compass migrate
If you are using Compass binary, you can run this command.
./compass -elasticsearch-brokers "http://<broker-host-name>" -db-host "<postgres-host-name>" -db-port 5432 -db-name "<postgres-db-name>" -db-user "<postgres-db-user>" -db-password "<postgres-db-password> migrate"
Once the migration has been done, Compass server can be started with this command.
$ docker run --net compass_storage -p 8080:8080 -e ELASTICSEARCH_BROKERS=http://es:9200 -e DB_HOST=postgres -e DB_PORT=5432 -e DB_NAME=compass -e DB_USER=compass -e DB_PASSWORD=compass_password odpf/compass compass serve
If you are using Compass binary, you can run this command.
./compass -elasticsearch-brokers "http://<broker-host-name>" -db-host "<postgres-host-name>" -db-port 5432 -db-name "<postgres-db-name>" -db-user "<postgres-db-user>" -db-password "<postgres-db-password> serve"
If everything goes ok, you should see something like this:
time="2022-04-27T09:18:08Z" level=info msg="compass starting" version=v0.2.0
time="2022-04-27T09:18:08Z" level=info msg="connected to elasticsearch cluster" config="\"docker-cluster\" (server version 7.6.1)"
time="2022-04-27T09:18:08Z" level=info msg="New Relic monitoring is disabled."
time="2022-04-27T09:18:08Z" level=info msg="statsd metrics monitoring is disabled."
time="2022-04-27T09:18:08Z" level=info msg="connected to postgres server" host=postgres port=5432
time="2022-04-27T09:18:08Z" level=info msg="server started"
# Run tests
$ make test
The tests combine both unit and integration tests, the test suite requires docker to run elasticsearch. In case you wish to test against an existing
elasticsearch cluster, set the value of ES_TEST_SERVER_URL to the URL of the elasticsearch server.
Development of Compass happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Compass.
Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Compass.
To help you get your feet wet and get you familiar with our contribution process, we have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started.
This project exists thanks to all the contributors.
Compass is Apache 2.0 licensed.