Webhook Sentry is a proxy that helps you send webhooks to your customers securely.
Sending webhooks appears simple on the surface -- they're just HTTP requests after all. But sending them securely is hard. If your application sends webhooks, does your implementation
- Prevent SSRF attacks?
- Protect against DNS rebinding attacks?
- Support mutual TLS?
- Validate SSL certificate chains correctly?
- Use an updated CA certificate bundle?
- Specify reasonable idle socket and connection timeouts?
By proxying webhooks through Webhook Sentry, you get all of these for free.
Sending webhooks involves making connections to untrusted and possibly malicious servers on the public internet. Maintaining an audit trail is essential for forensics and compliance. Limiting the set of instances that send such requests to a single proxy layer makes auditing simpler and more manageable.
Many customers require webhook requests to be sent from a list or range of static IPs in order to configure their firewalls. In a cloud environment with autoscaling, you may not want to allocate static IPs to your application instances. In other situations, like serverless applications, it may be impossible to assign static IPs. With a centralized egress proxy layer, you only need to assign static IPs to your proxy instances.
Webhook Sentry runs on port 9090 by default. You can configure the address and port in the listeners section of the config.
The simplest way to run Webhook Sentry is to use the latest binary:
- Download the latest release for your platform
- Run the downloaded binary:
whsentry
We also have a docker image:
docker run -p 9090:9090 juggernaut/webhook-sentry:latest
You can also pin a tagged release:
docker run -p 9090:9090 juggernaut/webhook-sentry:v1.0.8
If you need to override settings, you can mount a configuration file, pass in command line flags or set environment variables. See configuration for details.
If you need prometheus metrics for the service, allow access on port 2112 with something like -p 2112:2112.
curl -x http://localhost:9090 http://www.google.com
HTTP clients create a CONNECT tunnel when a proxy is configured and the target is a https URL. This does not give us the benefits of initiating TLS from the proxy. To get around this behavior, Webhook Sentry supports a unique way of proxying to HTTPS targets. Pass a X-WhSentry-TLS header and change the protocol to http:
curl -v -x http://localhost:9090 --header 'X-WhSentry-TLS: true' http://www.google.com
Although CONNECT is supported, I strongly recommend using the header approach to take advantage of the TLS capabilities of Webhook Sentry, like mutual TLS and robust certificate validation.
Specify clientCertFile and clientKeyFile in the configuration to enable mutual TLS:
clientCertFile: /path/to/client.pem
clientKeyFile: /path/to/key.pem
Point your collector to :2112 for metrics.
E.g if the proxy is running on localhost, to verify metrics are correctly exposed:
curl http://localhost:2112/metrics
To deploy Webhook Sentry with a static egress IP addresses in AWS EKS, you'll need a node group with an Elastic IP address:
- Create a new NAT Gateway.
- Create an Elastic IP address and assign it to your NAT Gateway. This will be your egress IP.
- Create a subnet dividing your network space.
- Create a route table linking the subnet to the NAT Gateway.
- Create a custom node group for the Webhook Sentry pods and assign it to the new subnet.
- Finally, assign your Webhook Sentry pods to that node group, and ensure your existing pods do not get assigned to it.
For redundancy, you may want to create multiple NAT Gatways, subnets, and egress IP addressses.
Webhook Sentry blocks access to private/internal IPs to prevent SSRF attacks:
$ curl -i -x http://localhost:9090 http://127.0.0.1:3000
HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:15:20 GMT
Content-Length: 24
IP 127.0.0.1 is blocked
Unlike naive implementations, it also correctly checks the IP after DNS resolution. This example makes use of the 1u.ms service which can serve up DNS records using any IP we want:
$ curl -i -x http://localhost:9090 http://make-127-0-0-1-rr.1u.ms
HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:21:58 GMT
Content-Length: 24
IP 127.0.0.1 is blocked
A malicious attacker can set up their DNS such that it first resolves to a valid public IP adddress, but subsequent resolutions point to private/internal IP addresses. This can be used to exploit webhook implementations that validate the resolved IP using getaddrinfo() or equivalent, then pass the original URL to a HTTP client library which resolves the host a second time. Again, let's use 1u.ms to first return a valid public IP and then the loopback IP:
$ curl -i -x http://localhost:9090 http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 324
Content-Type: application/json
Date: Wed, 30 Sep 2020 07:38:47 GMT
Server: gunicorn/19.9.0
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms",
"User-Agent": "Webhook Sentry/0.1",
"X-Amzn-Trace-Id": "Root=1-5f743607-afdf257ca619f90a14fc92b8"
},
"origin": "73.189.176.226",
"url": "http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get"
}
Webhook Sentry uses the latest Mozilla CA certificate bundle instead of relying on CA certificates bundled with the OS. This avoids the problem of out-of-date root CA certificates on older OS versions. See this blog post for why this is important. Notably, Stripe's webhooks were affected by this issue and took hours to fix.
On startup, Webhook Sentry checks if there is a newer version of the Mozilla CA certificate bundle than on disk, and if so, downloads it.
Additionally, by virtue of being written in Go, Webhook Sentry does not rely on OpenSSL or GnuTLS for certificate validation.
webhook-sentry uses viper for configuration. You can use a yaml file, environment variables or command line flags to provide configuration parameters.
By default, webhook-sentry looks for a file named config.yaml in the current working directory. You can specify a different file using the --config <filename flag.
The parameters documented below are in YAML, but most can also be provided as environment variables or command line flags:
listener: An HTTP/HTTPS endpoint the proxy listens on. For HTTPS endpoints, also specifycertFileandkeyFile.
Example:
listener:
type: https
address: 127.0.0.1:9091
certFile: /path/to/cert
keyFile: /path/to/key
connectTimeout: Timeout for the TCP connection to the destination host.
Default: 10s
connectionLifetime: Maximum time a connection to the destination can be alive.
Default: 60s
readTimeout: Maximum time a connection to the destination can remain idle.
Default: 10s
maxResponseBodySize: Maximum size of the HTTP response body in bytes. IfContent-Lengthis specified in the response and it is greater than this value, the connection is torn down and the response is discarded. The client receives a 502.
Default: 1048576
-
clientCertFile: Path to the client certificate to present to the destination (if enabling mutual TLS) -
clientKeyFile: Path to the private key of the client certificate (if enabling mutual TLS) -
accessLog: Specifiestypeandfileof the proxy access log.typecan be eithertextorjson. By default,textis output to stdout.
Example
accessLog:
type: json
file: /path/to/access.log
-
proxyLog: Specifiestypeandfileof the proxy application log. This log includes warnings and info messages related to handling proxy requests. By default,textis output to stdout. -
metrics.address: Listening address of the Prometheus metrics endpoint.
Default: :2112
- No IPv6 support
- No TLSv1.3 support
- No Proxy authentication
- Proxy does not check client certificates (not to be confused with proxy presenting client certificate to the remote host)