gitlab-ci-pipelines-exporter allows you to monitor your GitLab CI pipelines with Prometheus or any monitoring solution supporting the OpenMetrics format.
You can find more information on GitLab docs about how it takes part improving your pipeline efficiency.
Here is a Grafana dashboard I have been able to craft, using those metrics:
If you are interested into trying it out, have a look into the examples/docker-compose folder which contains documentation to provision test version of the exporter, prometheus and also grafana in ~5min using docker-compose
~$ go get -u github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd/gitlab-ci-pipelines-exporter~$ brew install mvisonneau/tap/gitlab-ci-pipelines-exporter~$ docker run -it --rm mvisonneau/gitlab-ci-pipelines-exporter~$ scoop bucket add https://github.com/mvisonneau/scoops
~$ scoop install gitlab-ci-pipelines-exporterHave a look onto the latest release page to pick your flavor and version. Here is an helper to fetch the most recent one:
~$ export GCPE_VERSION=$(curl -s "https://api.github.com/repos/mvisonneau/gitlab-ci-pipelines-exporter/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')# Binary (eg: linux/amd64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz
~$ tar zxvf gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz -C /usr/local/bin
# DEB package (eg: linux/386)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb
~$ dpkg -i gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb
# RPM package (eg: linux/arm64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpm
~$ rpm -ivh gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpmIf you want to make it run on kubernetes, there is a helm chart available for this purpose.
You can check deployments/helm/values.yml for configuration options.
# Clone the repository locally
~$ git clone git@github.com:mvisonneau/gitlab-ci-pipelines-exporter.git
# Configure a minimal configuration for the exporter
~$ cat <<EOF > values.yml
config:
gitlab:
url: https://gitlab.example.com
# You can also configure the token using --gitlab-token
# or the $GCPE_GITLAB_TOKEN environment variable
token: xrN14n9-ywvAFxxxxxx
projects:
- name: foo/project
EOF
# Release the chart on your Kubernetes cluster
~$ helm upgrade -i gitlab-ci-pipelines-exporter ./deployments/helm -f values.ymlThe complete configuration syntax is maintained here.
# Write a minimal config file somewhere on disk
~$ cat <<EOF > $(pwd)/config.yml
gitlab:
url: https://gitlab.example.com
# You can also configure the token using --gitlab-token
# or the $GCPE_GITLAB_TOKEN environment variable
token: <your_token>
projects:
- name: foo/project
- name: bar/project
wildcards:
- owner:
name: foo
kind: group
EOF
# If you have installed the binary
~$ gitlab-ci-pipelines-exporter --config /etc/config.yml
# Otherwise if you have docker available, it is as easy as :
~$ docker run -it --rm \
--name gitlab-ci-pipelines-exporter \
-v $(pwd)/config.yml:/etc/config.yml \
-p 8080:8080 \
mvisonneau/gitlab-ci-pipelines-exporter:latest \
--config /etc/config.ymlYou should then be able to see the following logs
INFO[0000] starting exporter gitlab-endpoint="https://gitlab.com" on-init-fetch-refs-from-pipelines=true polling-pipelines-every=60s polling-projects-every=15s polling-refs-every=10s polling-workers=2 rate-limit=10rps
INFO[0000] configured wildcards count=1
INFO[0000] found new project project-name=foo/project wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] found new project project-name=foo/bar wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] configured projects count=3
INFO[0000] started, now serving requests listen-address=":8080"
INFO[0000] found project refs project-path-with-namespace=foo/project project-ref=main
INFO[0000] found project refs project-path-with-namespace=bar/project project-ref=main
INFO[0000] found project refs project-path-with-namespace=foo/bar project-ref=mainAnd this is an example of the metrics you should expect to retrieve
~$ curl -s localhost:8080/metrics | grep gitlab_ci
# HELP gitlab_ci_pipeline_last_run_duration_seconds Duration of last pipeline run
# TYPE gitlab_ci_pipeline_last_run_duration_seconds gauge
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="dev",topics="",variables=""} 81
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="main",topics="",variables=""} 420
gitlab_ci_pipeline_last_run_duration_seconds{project="bar/project",ref="main",topics="",variables=""} 334
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 55
# HELP gitlab_ci_pipeline_last_run_id ID of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_id gauge
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="dev",topics="",variables=""} 4.0059611e+07
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="main",topics="",variables=""} 1.25351545e+08
gitlab_ci_pipeline_last_run_id{project="bar/project",ref="main",topics="",variables=""} 1.33308085e+08
gitlab_ci_pipeline_last_run_id{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 1.40420947e+08
# HELP gitlab_ci_pipeline_last_run_status Status of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_status gauge
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="failed",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="success",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="canceled",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="failed",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="manual",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="pending",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="running",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="skipped",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="success",topics="",variables="FOO:BAR"} 1
# HELP gitlab_ci_pipeline_run_count GitLab CI pipeline run count
# TYPE gitlab_ci_pipeline_run_count counter
gitlab_ci_pipeline_run_count{project="foo/project",ref="dev",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/project",ref="main",topics="",variables=""} 2
gitlab_ci_pipeline_run_count{project="bar/project",ref="main",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 2
# HELP gitlab_ci_pipeline_time_since_last_run_seconds Elapsed time since most recent GitLab CI pipeline run.
# TYPE gitlab_ci_pipeline_time_since_last_run_seconds gauge
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="dev",topics="",variables=""} 4.3368877e+07
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="main",topics="",variables=""} 4.151883e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="bar/project",ref="main",topics="",variables=""} 1.907042e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 65456If fetch_pipeline_job_metrics is enabled, expect additional metrics:
~$ curl -s http://localhost:8080/metrics | grep job
# HELP gitlab_ci_pipeline_job_run_count GitLab CI pipeline job run count
# TYPE gitlab_ci_pipeline_job_run_count counter
gitlab_ci_pipeline_job_run_count{job="build",project="bar/project",ref="main",stage="build",topics=""} 1
gitlab_ci_pipeline_job_run_count{job="test",project="bar/project",ref="main",stage="build",topics=""} 1
# HELP gitlab_ci_pipeline_last_job_run_artifact_size Filesize of the most recent job artifacts
# TYPE gitlab_ci_pipeline_last_job_run_artifact_size gauge
gitlab_ci_pipeline_last_job_run_artifact_size{job="build",project="bar/project",ref="main",stage="build",topics=""} 1.3793677e+07
gitlab_ci_pipeline_last_job_run_artifact_size{job="test",project="bar/project",ref="main",stage="build",topics=""} 257737
# HELP gitlab_ci_pipeline_last_job_run_duration_seconds Duration of last job run
# TYPE gitlab_ci_pipeline_last_job_run_duration_seconds gauge
gitlab_ci_pipeline_last_job_run_duration_seconds{job="build",project="bar/project",ref="main",stage="build",topics=""} 826.064469
gitlab_ci_pipeline_last_job_run_duration_seconds{job="test",project="bar/project",ref="main",stage="test",topics=""} 519.873374
# HELP gitlab_ci_pipeline_last_job_run_status Status of the most recent job
# TYPE gitlab_ci_pipeline_last_job_run_status gauge
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="main",stage="build",status="failed",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="main",stage="build",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="main",stage="build",status="success",topics=""} 1
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="failed",topics=""} 1
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="success",topics=""} 0
# HELP gitlab_ci_pipeline_time_since_last_job_run_seconds Elapsed time since most recent GitLab CI job run.
# TYPE gitlab_ci_pipeline_time_since_last_job_run_seconds gauge
gitlab_ci_pipeline_time_since_last_job_run_seconds{job="build",project="bar/project",ref="main",stage="build",topics=""} 52422
gitlab_ci_pipeline_time_since_last_job_run_seconds{job="test",project="bar/project",ref="main",stage="test",topics=""} 1.0260727e+07If the number of metrics generated by fetching jobs becomes a problem, you can enable output_sparse_status_metrics on a global, per-project or per-wildcard basis. When enabled, only labels matching the previous pipeline or job status will be submitted (with value 1) rather than all label combinations submitted but with 0 value where the status does not match the previous run, for example:
# output_sparse_status_metrics: false
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="canceled",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="failed",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="manual",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="pending",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="skipped",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="success",topics=""} 1
# output_sparse_status_metrics: true
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="main",stage="test",status="success",topics=""} 1~$ gitlab-ci-pipelines-exporter --help
NAME:
gitlab-ci-pipelines-exporter - Export metrics about GitLab CI pipelines statuses
USAGE:
gitlab-ci-pipelines-exporter [global options] command [command options] [arguments...]
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--config file, -c file config file (default: "~/.gitlab-ci-pipelines-exporter.yml") [$GCPE_CONFIG]
--redis-url url redis url for an HA setup (format: redis[s]://[:password@]host[:port][/db-number][?option=value]) [$GCPE_REDIS_URL]
--enable-pprof Enable profiling endpoints at /debug/pprof (default: false) [$GCPE_ENABLE_PPROF]
--gitlab-token token GitLab access token. Can be use to override the gitlab token in config file [$GCPE_GITLAB_TOKEN]
--listen-address address:port, -l address:port listen-address address:port (default: ":8080") [$GCPE_LISTEN_ADDRESS]
--log-level level log level (debug,info,warn,fatal,panic) (default: "info") [$GCPE_LOG_LEVEL]
--log-format format log format (json,text) (default: "text") [$GCPE_LOG_FORMAT]
--help, -h show help (default: false)If you use docker, you can easily get started using :
~$ make dev-env
# You should then be able to use go commands to work onto the project, eg:
~docker$ make fmt
~docker$ gitlab-ci-pipelines-exporterContributions are more than welcome! Feel free to submit a PR.