Add Uptime Monitoring

[aldy505]

Problem Statement

Uptime Monitoring feature is out and it's on Early Release. We should bring that here if it hits General Availability.

Product discussion: getsentry/sentry#51110
Announcement: https://sentry.io/changelog/uptime-monitoring-early-adopters-release/
Documentation: https://docs.sentry.io/product/alerts/uptime-monitoring/

Solution Brainstorm

The way we'll tell you how to configure this should be versioned (learned this the hard way from User Feedback feature). Grab the version you're currently on (or if yours is higher than any of these, choose the highest one).

• Setup guide for 24.8.0
• Setup guide for 25.1.0 -- plus custom CA certificates + allow internal IPs
• Setup guide for 25.2.0 -- allow internal IPs is now bundled inside the upstream repository & there is a web UI to see the uptime history
• Setup guide for 25.3.0 -- some new configs, see here https://github.com/getsentry/uptime-checker/blob/b238224f70e23267febdd4100f02068cb2ba18bc/src/app/config.rs#L122-L144
• Setup guide for 25.4.0
• Setup guide for 25.6.0 or 25.6.1

[aldy505]

Setup guide for 24.8.0

Date of writing: 2024-08-31

Does this works for versions below 24.8.0? I have no idea.

On your docker-compose.yml, add a few new containers:

  uptime-results:
    <<: *sentry_defaults
    command: run consumer uptime-results --consumer-group uptime-results
    profiles:
      - feature-complete
  uptime-checker:
    <<: *restart_policy
    # IMPORTANT, READ THIS!
    # If you trust me enough (although we probably never met), you can use the Docker image I built to save some time, so you can use this:
    # image: "aldy505/sentry-uptime-checker:8ffa29b89b319e4b8e51154e82569796ea74c3af"
    #
    # But if you don't trust me enough and prefer to build from source yourself, use this instead (I'm using this on my instance).
    # If you're using the `image` section above, obviously you'll need to remove this `build` section (or just comment it out).
    build: ./uptime-checker-git
    command: run --config /config.yml
    volumes:
      - type: bind
        read_only: true
        source: ./uptime-checker/config.yml
        target: /config.yml
    # If you don't fancy a config file and prefer environment variables, use this instead (and remove the `volume` block above):
    # environment:
    #  UPTIME_CHECKER_REDIS_HOST: "redis://redis:6379"
    #  UPTIME_CHECKER_RESULTS_KAFKA_CLUSTER: "kafka:9092"
    #  UPTIME_CHECKER_RESULTS_KAFKA_TOPIC: "uptime-results"
    #  UPTIME_CHECKER_CONFIGS_KAFKA_CLUSTER: "kafka:9092"
    #  UPTIME_CHECKER_CONFIGS_KAFKA_TOPIC: "uptime-configs"
    #  UPTIME_CHECKER_CHECKER_CONCURRENCY: "200"
    #  UPTIME_CHECKER_LOG_LEVEL: "error"
    depends_on:
      kafka:
        <<: *depends_on-healthy
      redis:
        <<: *depends_on-healthy
    profiles:
      - feature-complete

On your sentry/sentry.conf.py, add some new feature flags:

SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
          # The rest of your features
        )
        # Uptime Monitoring related flags
        + (
            "organizations:uptime-api-create-update",
            "organizations:uptime-automatic-hostname-detection",
            "organizations:uptime-automatic-subscription-creation",
            "organizations:uptime-rule-api",
            "organizations:uptime-create-issues",
            "organizations:uptime-settings",
            "organizations:uptime-visible", # This is for issues platform
            "organizations:uptime-ingest",
            "organizations:uptime-post-process-group",
        )
    }
)

Create a new directory called uptime-checker, create a config.yml file. The resulting path is uptime-checker/config.yml. You don't need to do this if you're using environment variable as config on the docker-compose.yml:

# See https://github.com/getsentry/uptime-checker/blob/8ffa29b89b319e4b8e51154e82569796ea74c3af/src/app/config.rs
# You could technically use environment variable instead of this file, if you prefer, but I think it's better to use this instead.
# If you really want to use environment variables, do something like: UPTIME_CHECKER_SENTRY_DSN UPTIME_CHECKER_REDIS_HOST, you know the rest.

# sentry_dsn: <string, optional>
# sentry_env: <string, optional>
log_level: error # valid values: off, trace, debug, info, warn, error
log_format: auto # valid values: auto, pretty, simplified, json

checker_concurrency: 200

results_kafka_cluster: "kafka:9092"
results_kafka_topic: "uptime-results"
configs_kafka_cluster: "kafka:9092"
configs_kafka_topic: "uptime-configs"

redis_host: "redis://redis:6379"

If on the docker-compose.yml file you're not using the Docker image that I built, add a line on install.sh:

# ...
source install/update-docker-images.sh
source install/build-uptime-checker.sh # Add this (also remove this comment)
source install/build-docker-images.sh
# ...

And create the file install/build-uptime-checker.sh (still if you're not using the Docker image that I built):

echo "${_group}Building uptime-checker ... "

rm -rf uptime-checker-git

git clone --depth 1 https://github.com/getsentry/uptime-checker.git uptime-checker-git

echo "${_endgroup}"

After all that, run the ./install.sh script. Good luck.

[aldy505]

@evanpurkhiser @wedamija Hey, someone on Discord (link: https://discord.com/channels/621778831602221064/1320980341061849109) reported that no issues are registered when the server is down, they have set up the alerts for uptime checker. They're building the Docker image from source. Is there anything that's missing from my setup/config above?

[aldy505]

Setup Guide for 25.1.0

Date of writing: 2025-01-16

Add some new containers on your docker-compose.yml:

  uptime-checker:
    <<: *restart_policy
    image: "$UPTIME_IMAGE"
    command: run
    environment:
      UPTIME_CHECKER_SENTRY_DSN: "$SENTRY_DSN"
      UPTIME_CHECKER_RESULTS_KAFKA_CLUSTER: kafka:9092
      UPTIME_CHECKER_RESULTS_KAFKA_TOPIC: uptime-results
      UPTIME_CHECKER_CONFIGS_KAFKA_CLUSTER: kafka:9092
      UPTIME_CHECKER_CONFIGS_KAFKA_TOPIC: uptime-configs
      UPTIME_CHECKER_REDIS_HOST: redis://redis:6379
      UPTIME_CHECKER_LOG_LEVEL: debug # Valid options: info, warn, error, debug
    depends_on:
      kafka:
        <<: *depends_on-healthy
      redis:
        <<: *depends_on-healthy
    profiles:
      - feature-complete
  uptime-results:
    <<: *sentry_defaults
    command: run consumer uptime-results --consumer-group uptime-results
    profiles:
      - feature-complete
  snuba-uptime-results-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage uptime_monitor_checks --consumer-group snuba-uptime-results --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
    profiles:
      - feature-complete

For snuba-uptime-results-consumer if you have problem with rust-consumer just use regular Python consumer.

On your .env or .env.custom file, just add this:

UPTIME_IMAGE=aldy505/sentry-uptime-checker:latest

To be honest, I'm tired of having the install script that builds the Docker image locally, so I'll just build this regularly every release (or every time I do upgrades).

Then on your sentry/sentry.conf.py add these feature flags:

SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            # ... the original feature flags
        )
        # Uptime Monitoring related flags
        + (
            "organizations:uptime-automatic-hostname-detection",
            "organizations:uptime-automatic-subscription-creation",
            "organizations:uptime-create-issues",
            "organizations:uptime-settings",
            "organizations:uptime-visible", # For issues platform, I don't know if this is still required
            "organizations:uptime-ingest",
            "organizations:uptime-post-process-group",
            "organizations:insights-uptime", # Enables `Uptime` section on the web UI, but this don't really works, you can't access it.
        )
    }
)

After all that, re run ./install.sh script.

---

Allow internal IPs

By default, uptime-checker container deny request to internal IPs. Since we're on self-hosted, we can bypass that. But that requires something to be changed. Simply set your Docker image on .env file to:

UPTIME_IMAGE=aldy505/sentry-uptime-checker:allow-internal-ips

Custom CA Certificates

Again, on self-hosted, we're bound to weird setups. Store your CA certificates on the certificates directory (the directory already exists, with gitignore in it). Modify your docker-compose.yml (notice on image, build and volumes section are different than the usual).

  uptime-checker:
    <<: *restart_policy
    image: uptime-checker-local
    build:
      context: .
      dockerfile: ./uptime-checker/Dockerfile
      args:
        - UPTIME_IMAGE
    command: run
    environment:
      UPTIME_CHECKER_SENTRY_DSN: "$SENTRY_DSN"
      UPTIME_CHECKER_RESULTS_KAFKA_CLUSTER: kafka:9092
      UPTIME_CHECKER_RESULTS_KAFKA_TOPIC: uptime-results
      UPTIME_CHECKER_CONFIGS_KAFKA_CLUSTER: kafka:9092
      UPTIME_CHECKER_CONFIGS_KAFKA_TOPIC: uptime-configs
      UPTIME_CHECKER_REDIS_HOST: redis://redis:6379
      UPTIME_CHECKER_LOG_LEVEL: debug
    volumes:
      - ./certificates:/ca-certificates:ro # This!
    depends_on:
      kafka:
        <<: *depends_on-healthy
      redis:
        <<: *depends_on-healthy
    profiles:
      - feature-complete

Create uptime-checker/Dockerfile file (a Dockerfile file within uptime-checker directory):

ARG UPTIME_IMAGE
FROM ${UPTIME_IMAGE}

USER root
COPY ./certificates /ca-certificates
RUN cat /ca-certificates/* >> /etc/ssl/certs/ca-certificates.crt
USER app

No need to run install.sh, just do: sudo docker compose --env-file .env --env-file .env.custom up -d --build --wait uptime-checker

[aldy505]

Setup Guide for 25.2.0

Date of writing: 2025-02-27

If you're using custom CA certificate or self-signed TLS certificate, read until the end, don't waste your time by executing install.sh multiple times.

Modify your Docker Compose file:

  uptime-checker:
    <<: *restart_policy
    image: "$UPTIME_IMAGE"
    command: run
    environment:
      UPTIME_CHECKER_SENTRY_DSN: "$SENTRY_DSN" # Set this to your internal SENTRY_DSN to monitor this container
      UPTIME_CHECKER_RESULTS_KAFKA_CLUSTER: "kafka:9092"
      UPTIME_CHECKER_CONFIGS_KAFKA_CLUSTER: "kafka:9092"
      UPTIME_CHECKER_REDIS_HOST: "redis://redis:6379"
      UPTIME_CHECKER_LOG_LEVEL: "debug" # Valid options: info, warn, error, debug
      # If you are using statsd to monitor metrics and stuff, you can uncomment this:
      # UPTIME_CHECKER_STATSD_ADDR: "127.0.0.1:8125"
      # If you want to enable HTTP connection reuse to save up connection time, you can set this to true, otherwise, just comment it out:
      # UPTIME_CHECKER_DISABLE_CONNECTION_REUSE: true
    depends_on:
      kafka:
        <<: *depends_on-healthy
      redis:
        <<: *depends_on-healthy
    profiles:
      - feature-complete
  uptime-results:
    <<: *sentry_defaults
    command: run consumer uptime-results --consumer-group uptime-results
    profiles:
      - feature-complete
  snuba-uptime-results-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage uptime_monitor_checks --consumer-group snuba-uptime-results --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
    profiles:
      - feature-complete

On your .env file (or .env.custom), add this:

UPTIME_IMAGE=aldy505/sentry-uptime-checker:9574780ba9ca367e87e656ab6468e4923020bae9

On your sentry/sentry.conf.py, update your feature flags:

SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            # ... the original feature flags
        )
        # Uptime Monitoring related flags
        + (
            "organizations:uptime",
            "organizations:slack-threads-refactor-uptime", # Enable new slack threads logic for uptime issues
            "organizations:insights-uptime", # Enable access to insights uptime view
            "organizations:uptime-automatic-hostname-detection",
            "organizations:uptime-automatic-subscription-creation",
            "organizations:uptime-create-issues", # Enable creating issues via the issue platform
            "organizations:uptime-settings", # Enables uptime related settings for projects and orgs
            # Below these are for issues platform
            "organizations:uptime-domain-failure-visible",
            "organizations:uptime-domain-failure-ingest",
            "organizations:uptime-domain-failure-post-process-group",
        )
    }
)

On your sentry/config.yml, add this line on the very bottom:

uptime.snuba_uptime_results.enabled: true

Then, run ./install.sh script and you're done.

Just trust me on that image, you can also build it yourself but why even bother. I provide both linux/amd64 and linux/arm64 image. Here's how I build it for future reference (which probably just for me):

sudo docker buildx build --platform linux/arm64,linux/amd64 --tag "aldy505/sentry-uptime-checker:$(git log -1 --format=%H | xargs)" . && \
sudo docker image tag "aldy505/sentry-uptime-checker:$(git log -1 --format=%H | xargs)" "aldy505/sentry-uptime-checker:latest" && \
sudo docker image push "aldy505/sentry-uptime-checker" --all-tags

Custom CA Certificates

If your URL contains self-signed TLS certificate, you should do some modifications after applying all that above. Store your CA certificates on the certificates directory (the directory already exists, with gitignore in it).

On your docker-compose.yml file, modify uptime-checker like so:

  uptime-checker:
    <<: *restart_policy
    image: uptime-checker-local
    build:
      context: .
      dockerfile: ./uptime-checker/Dockerfile
      args:
        - UPTIME_IMAGE
    command: run
    environment:
      UPTIME_CHECKER_SENTRY_DSN: "$SENTRY_DSN" # Set this to your internal SENTRY_DSN to monitor this container
      UPTIME_CHECKER_RESULTS_KAFKA_CLUSTER: "kafka:9092"
      UPTIME_CHECKER_CONFIGS_KAFKA_CLUSTER: "kafka:9092"
      UPTIME_CHECKER_REDIS_HOST: "redis://redis:6379"
      UPTIME_CHECKER_LOG_LEVEL: "debug" # Valid options: info, warn, error, debug
      # If you are using statsd to monitor metrics and stuff, you can uncomment this:
      # UPTIME_CHECKER_STATSD_ADDR: "127.0.0.1:8125"
      # If you want to enable HTTP connection reuse to save up connection time, you can set this to true, otherwise, just comment it out:
      # UPTIME_CHECKER_DISABLE_CONNECTION_REUSE: true
    volumes:
      - ./certificates:/ca-certificates:ro # This!
    depends_on:
      kafka:
        <<: *depends_on-healthy
      redis:
        <<: *depends_on-healthy
    profiles:
      - feature-complete

Create a directory uptime-checker, create a file Dockerfile inside it. On your uptime-checker/Dockerfile, put this:

ARG UPTIME_IMAGE
FROM ${UPTIME_IMAGE}

USER root
COPY ./certificates /ca-certificates
RUN cat /ca-certificates/* >> /etc/ssl/certs/ca-certificates.crt
USER app