Distributed Deployment Configuration

[mapl]

Assume you have Gatus deployed in several security subnets (or zones) to monitor individual services because one single Gatus instance is not able to reach those services (administratively prohibited due to firewall rules, etc.)

But you want one main Gatus instance which is capable of retrieving health information of services from all those other Gatus instances to display them in one unified Gatus Dashboard.

Can we start a short discussion on this?

PS. Thank you for this great project!

[TwiN]

Hello @mapl.

This is quite an edge case, so honestly, I'm not sure I'll be personally working on this unless more people show interest for this feature (based on the number of 👍 on the issue).

The easiest way to implement this that I can think of is by having a way to allow pushing data into Gatus, as opposed to the current behavior, which only allows retrieving data from Gatus.
By leveraging this as well as a configuration that allows specifying whether a Gatus instance is the primary instance or a secondary instance, the latter required to specify the endpoint of the primary instance, it would be possible to have a "global" dashboard and multiple Gatus instances configured independently.

Fortunately, the easiest way is also the most convenient one, because the other ones would likely involve persistence.

There's an even easier solution, but it assumes that the users accessing the dashboard has access to all "security subnets/zones", which I'm not sure is the case based on your explanation. The only work required for to send a request from the Gatus' dashboard frontend to each backends and merge the statuses.

[mapl]

I made a quick diagram about a simple distributed Gatus Deployment where the main Gatus Instance just pulls data from remote Gatus instances.

The Data from each remote Gatus instance is just embedded into the Main Gatus Dashboard.

[sefaphlvn]

I think this is a good idea. I would like it if there was

[TwiN]

That looks good.

@mapl What do you think would be the appropriate behavior when there are overlapping service names?

Also, how about we do the opposite: the remote Gatus instances push their data to the main Gatus instance?
I think that would allow a lot more flexibility, especially if, for instance, one of the remote Gatus instances is running in an environment completely inaccessible from the main Gatus instance (i.e. locally).

Of course, this would require a layer of security, but I built something oddly relevant to this specific use case: https://github.com/TwiN/g8

We'd also need to add something that periodically cleans up services that haven't been refreshed in a long time (i.e. in case a remote instance is taken offline, we don't want to keep the outdated service health checks on the dashboard forever).

[mapl]

Overlapping service names

Hmm... not sure. Is this actually supposed to be a problem? A UUID would solve the most of the issues I think.

---

Clean up services

A steady Health Status of the remote Gatus instance would be cool, like the last reached Status, just to know if its still alive.

---

There is and always been an ongoing debate about Pulling VS Pushing when it comes to monitoring.

For example, if you have 100 remote Gatus instances and each of those instances would push data to the main Gatus instance, the main instance is quickly subject to an overload of metric data.

I am wondering if it is enough to just proxy through data from the remote instances to the main instance, or is caching needed?

The goal should always be a simple as possible design.

https://dave.cheney.net/2019/07/09/clear-is-better-than-clever

Security

An API Token is commonly used to restrict permissions. What's your opinion on this?

---

Anyway, a very goof reference is Prometheus and why the devs decide to go with pulling rather than pushing.

However, that's not a one way road, so it depends on the scenario. For the most part Pulling is the better choice.

https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/

https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push

[TwiN]

Fair enough, though if we're pulling, then there's no need for a token since the endpoint needs to be public for the dashboard to be shown.

Sounds good!

[mapl]

If you can ensure that the main Gatus instance can reach its remote instances via http(s) to fetch the json data, it should be fine

What do you think of this config idea snippet to be deployed on the main instance? I thought an Api key could be handy but not required as you mentioned.

remote-services:
  - name: Gatus Remote Instance 1	
    url: http://10.10.10.10/api/v1/statuses
    api-key: XXXXXXXXXXXXXX
    interval: 10s
  - name: Gatus Remote Instance 2 	
    url: http://10.10.10.20/api/v1/statuses
    api-key: XXXXXXXXXXXXXX
    interval: 5s
  - name: Gatus Remote Instance 3
    url: http://10.10.10.30/api/v1/statuses
    api-key: XXXXXXXXXXXXXX
    interval: 10s

[TwiN]

I was thinking something along the lines of

remote:
  instances:
    - name: "gatus-internal"
      url: "http://10.10.10.10/api/v1/statuses"
      interval: 30s

because in the future, we may have to add other parameters specific to the remote configuration, such as:

• strategy: merge: merge all statuses fetched from the remote instances with existing, overlapping service names, or create new statuses if they already exist
  

• strategy: prefix-service: Prepend each retrieved service names with the configured name of the remote instance
  

• strategy: prefix-group: Prepend each retrieved group names with the configured name of the remote instance
  

Anyways, this isn't for right now, but I still think it's good to make the configuration as extensible as possible to prevent future breaking changes.

[mapl]

Absolutely, the config should be as extensible as possible and future proof where possible. Good point.
Your concerns about possible service names clashes when the main and remote instances have the same names could be an actual issue if you put them in one flat view.
I think, if every remote instance is moved into its own, lets says "container" box, then you could clearly derive where its origin is.
So if you have duplicate names in this case, it wouldn't matter as they are displayed in their own container, in a way you added this nice grouping feature.
Additionally, you could also check the health state of the entire remote instance, about last time reached for instance, latency, etc.

[TwiN]

I've been thinking a bit more about the implementation for this, and one issue I can think of is how to handle alerting.

Assuming that the purpose is to monitor internal applications within a network that isn't publicly accessible by other users, but is accessible by a "remote" Gatus instance, how would alerting for a "remote" Gatus instance be handled, assuming that the alerting configuration as well as the individual alerts for each services are not exposed through Gatus' main endpoint (/api/v1/statuses)?

I think it's safe to assume that each remote instances are expected to deal with their own alerts and that the only difference between the remote instance(s) and the main instance is that the main instance's dashboard must display the statuses from the remote instances as well.

All in all, I don't think this is a problem, but felt like specifying this through a comment would be worthwhile for the sake of traceability.

[martincollado]

Maybe would be a good idea to have different systems?
One implemented with a check-in system, and another pulling data from intranet networks.

Check-in system: Could give the control to developers to implement maybe some logic into the check (for example, in our systems we use healthchecks to run certain tests and if everything goes well, we send the ok to the healthchecks api endpoint, so we know that we are not only receiving that the server responds and is alive, plus adding complementary logic to the checks. For example, knowing that in our databases we're receiving data or our auth service is working properly, for example)

• Statping: https://github.com/statping/statping
• Healthchecks: https://github.com/healthchecks/healthchecks

About the alerting on that mode of check-in, it is necessary to setup the expectation in time when the check-in must be in, and if is delayed, starts alerting.