Skip to content

Commit d2bed33

Browse files
bhussain91bhussain91
authored
feat: Add support for tracing (#1021)
Signed-off-by: Bilal Hussain <bilal.hussain@10xbanking.com> Signed-off-by: bhussain91 <161487948+bhussain91@users.noreply.github.com>
1 parent d891e6e commit d2bed33

File tree

18 files changed

+920
-30
lines changed

18 files changed

+920
-30
lines changed

‎.github/workflows/linkinator.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,4 @@ jobs:
2222
paths: "**/*.md"
2323
markdown: true
2424
retry: true
25-
linksToSkip: "https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-interceptor, https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-operator, https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-scaler, http://opentelemetry-collector.open-telemetry-system:4318, https://www.gnu.org/software/make/"
25+
linksToSkip: "https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-interceptor, https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-operator, https://github.com/kedacore/http-add-on/pkgs/container/http-add-on-scaler,http://opentelemetry-collector.open-telemetry-system:4318,http://opentelemetry-collector.open-telemetry-system:4318/v1/traces, https://www.gnu.org/software/make/"

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ This changelog keeps track of work items that have been completed and are ready
2525

2626
### New
2727

28+
- **General**: Add configurable tracing support to the interceptor proxy ([#1021](https://github.com/kedacore/http-add-on/pull/1021))
2829
- **General**: TODO ([#TODO](https://github.com/kedacore/http-add-on/issues/TODO))
2930

3031
### Improvements

‎config/interceptor/e2e-test/otel/deployment.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,11 @@ spec:
1919
value: "http://opentelemetry-collector.open-telemetry-system:4318"
2020
- name: OTEL_METRIC_EXPORT_INTERVAL
2121
value: "1"
22+
- name: OTEL_EXPORTER_OTLP_TRACES_ENABLED
23+
value: "true"
24+
- name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
25+
value: "http/protobuf"
26+
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
27+
value: "http://opentelemetry-collector.open-telemetry-system:4318/v1/traces"
28+
- name: OTEL_EXPORTER_OTLP_TRACES_INSECURE
29+
value: "true"

‎docs/operate.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,30 @@ For setting multiple TLS certs, set `KEDA_HTTP_PROXY_TLS_CERT_STORE_PATHS` with
2929
* `XYZ.pem` + `XYZ-key.pem`
3030

3131
The matching between certs and requests is performed during the TLS ClientHelo message, where the SNI service name is compared to SANs provided in each cert and the first matching cert will be used for the rest of the TLS handshake.
32+
# Configuring tracing for the KEDA HTTP Add-on interceptor proxy
33+
34+
### Supported Exporters:
35+
* **console** - The console exporter is useful for development and debugging tasks, and is the simplest to set up.
36+
* **http/protobuf** - To send trace data to an OTLP endpoint (like the collector or Jaeger >= v1.35.0) you’ll want to configure an OTLP exporter that sends to your endpoint.
37+
* * **grpc** - To configure exporter to send trace data over gRPC connection to an OTLP endpoint (like the collector or Jaeger >= v1.35.0) you’ll want to configure an OTLP exporter that sends to your endpoint.
38+
39+
### Configuring tracing with console exporter
40+
41+
To enable tracing with the console exporter, the `OTEL_EXPORTER_OTLP_TRACES_ENABLED` environment variable should be set to `true` on the interceptor deployment. (`false` by default).
42+
Secondly set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` to `console` (`console` by default). Other protocols include (`http/protobuf` and `grpc`).
43+
Finally set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to `"http://localhost:4318/v1/traces"` (`"http://localhost:4318/v1/traces"` by default).
44+
45+
46+
### Configuring tracing with OTLP exporter
47+
When configured, the interceptor proxy can export metrics to a OTEL HTTP collector.
48+
49+
To enable tracing with otlp exporter, the `OTEL_EXPORTER_OTLP_TRACES_ENABLED` environment variable should be set to `true` on the interceptor deployment. (`false` by default).
50+
Secondly set `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` to `otlphttp` (`console` by default). Other protocols include (`http/protobuf` and `grpc`)
51+
Finally set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to the collector to send the traces to (e.g. http://opentelemetry-collector.open-telemetry-system:4318/v1/traces) (`"http://localhost:4318/v1/traces"` by default).
52+
NOTE: full path is required to be set including <scheme><url><port><path>
53+
54+
55+
Optional variables
56+
`OTEL_EXPORTER_OTLP_HEADERS` - To pass any extra headers to the spans to utilise your OTEL collector e.g. authentication details (`"key1=value1,key2=value2"`)
57+
`OTEL_EXPORTER_OTLP_TRACES_INSECURE` - To send traces to the tracing via HTTP rather than HTTPS (`false` by default)
58+
`OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` - The batcher timeout in seconds to send batch of data points (`5` by default)

‎go.mod‎

Lines changed: 20 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,13 @@ require (
1111
github.com/onsi/ginkgo/v2 v2.23.3
1212
github.com/onsi/gomega v1.36.3
1313
github.com/stretchr/testify v1.10.0
14+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0
15+
go.opentelemetry.io/contrib/propagators/b3 v1.35.0
1416
go.opentelemetry.io/otel v1.35.0
1517
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.35.0
18+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0
19+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.35.0
20+
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.35.0
1621
go.opentelemetry.io/otel/sdk v1.35.0
1722
go.uber.org/mock v0.5.0
1823
golang.org/x/sync v0.12.0
@@ -48,6 +53,20 @@ require (
4853
github.com/beorn7/perks v1.0.1 // indirect
4954
github.com/blang/semver/v4 v4.0.0 // indirect
5055
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
56+
github.com/felixge/httpsnoop v1.0.4 // indirect
57+
github.com/go-logr/stdr v1.2.2 // indirect
58+
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
59+
github.com/gorilla/websocket v1.5.1 // indirect
60+
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
61+
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
62+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0 // indirect
63+
go.opentelemetry.io/proto/otlp v1.5.0 // indirect
64+
go.uber.org/zap v1.27.0 // indirect
65+
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
66+
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
67+
)
68+
69+
require (
5170
github.com/cespare/xxhash/v2 v2.3.0 // indirect
5271
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
5372
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
@@ -56,12 +75,10 @@ require (
5675
github.com/fsnotify/fsnotify v1.7.0 // indirect
5776
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
5877
github.com/go-errors/errors v1.5.1 // indirect
59-
github.com/go-logr/stdr v1.2.2 // indirect
6078
github.com/go-logr/zapr v1.3.0 // indirect
6179
github.com/go-openapi/jsonpointer v0.21.0 // indirect
6280
github.com/go-openapi/jsonreference v0.21.0 // indirect
6381
github.com/go-openapi/swag v0.23.0 // indirect
64-
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
6582
github.com/gogo/protobuf v1.3.2 // indirect
6683
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
6784
github.com/golang/protobuf v1.5.4 // indirect
@@ -70,8 +87,6 @@ require (
7087
github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
7188
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
7289
github.com/google/uuid v1.6.0 // indirect
73-
github.com/gorilla/websocket v1.5.1 // indirect
74-
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
7590
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
7691
github.com/imdario/mergo v0.3.16 // indirect
7792
github.com/inconshreveable/mousetrap v1.1.0 // indirect
@@ -85,7 +100,6 @@ require (
85100
github.com/modern-go/reflect2 v1.0.2 // indirect
86101
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
87102
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
88-
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
89103
github.com/pkg/errors v0.9.1 // indirect
90104
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
91105
github.com/prometheus/client_golang v1.21.1
@@ -101,10 +115,8 @@ require (
101115
go.opentelemetry.io/otel/exporters/prometheus v0.57.0
102116
go.opentelemetry.io/otel/metric v1.35.0
103117
go.opentelemetry.io/otel/sdk/metric v1.35.0
104-
go.opentelemetry.io/otel/trace v1.35.0 // indirect
105-
go.opentelemetry.io/proto/otlp v1.5.0 // indirect
118+
go.opentelemetry.io/otel/trace v1.35.0
106119
go.uber.org/multierr v1.11.0 // indirect
107-
go.uber.org/zap v1.27.0 // indirect
108120
golang.org/x/exp v0.0.0-20241108190413-2d47ceb2692f
109121
golang.org/x/mod v0.23.0 // indirect
110122
golang.org/x/net v0.37.0 // indirect
@@ -115,9 +127,7 @@ require (
115127
golang.org/x/time v0.8.0 // indirect
116128
golang.org/x/tools v0.30.0 // indirect
117129
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
118-
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
119130
google.golang.org/genproto/googleapis/rpc v0.0.0-20250218202821-56aae31c358a // indirect
120-
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
121131
gopkg.in/inf.v0 v0.9.1 // indirect
122132
gopkg.in/yaml.v2 v2.4.0 // indirect
123133
gopkg.in/yaml.v3 v3.0.1 // indirect

‎go.sum‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ github.com/evanphx/json-patch v5.9.0+incompatible h1:fBXyNpNMuTTDdquAq/uisOr2lSh
1919
github.com/evanphx/json-patch v5.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
2020
github.com/evanphx/json-patch/v5 v5.9.0 h1:kcBlZQbplgElYIlo/n1hJbls2z/1awpXxpRi0/FOJfg=
2121
github.com/evanphx/json-patch/v5 v5.9.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ=
22+
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
23+
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
2224
github.com/expr-lang/expr v1.17.0 h1:+vpszOyzKLQXC9VF+wA8cVA0tlA984/Wabc/1hF9Whg=
2325
github.com/expr-lang/expr v1.17.0/go.mod h1:8/vRC7+7HBzESEqt5kKpYXxrxkr31SaO8r40VO/1IT4=
2426
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
@@ -154,12 +156,24 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
154156
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
155157
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
156158
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
159+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 h1:TT4fX+nBOA/+LUkobKGW1ydGcn+G3vRw9+g5HwCphpk=
160+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0/go.mod h1:L7UH0GbB0p47T4Rri3uHjbpCFYrVrwc1I25QhNPiGK8=
161+
go.opentelemetry.io/contrib/propagators/b3 v1.35.0 h1:DpwKW04LkdFRFCIgM3sqwTJA/QREHMeMHYPWP1WeaPQ=
162+
go.opentelemetry.io/contrib/propagators/b3 v1.35.0/go.mod h1:9+SNxwqvCWo1qQwUpACBY5YKNVxFJn5mlbXg/4+uKBg=
157163
go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
158164
go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
159165
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.35.0 h1:0NIXxOCFx+SKbhCVxwl3ETG8ClLPAa0KuKV6p3yhxP8=
160166
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.35.0/go.mod h1:ChZSJbbfbl/DcRZNc9Gqh6DYGlfjw4PvO1pEOZH1ZsE=
167+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0 h1:1fTNlAIJZGWLP5FVu0fikVry1IsiUnXjf7QFvoNN3Xw=
168+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.35.0/go.mod h1:zjPK58DtkqQFn+YUMbx0M2XV3QgKU0gS9LeGohREyK4=
169+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 h1:qFffATk0X+HD+f1Z8lswGiOQYKHRlzfmdJm0wEaVrFA=
170+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0/go.mod h1:MOiCmryaYtc+V0Ei+Tx9o5S1ZjA7kzLucuVuyzBZloQ=
171+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.35.0 h1:xJ2qHD0C1BeYVTLLR9sX12+Qb95kfeD/byKj6Ky1pXg=
172+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.35.0/go.mod h1:u5BF1xyjstDowA1R5QAO9JHzqK+ublenEW/dyqTjBVk=
161173
go.opentelemetry.io/otel/exporters/prometheus v0.57.0 h1:AHh/lAP1BHrY5gBwk8ncc25FXWm/gmmY3BX258z5nuk=
162174
go.opentelemetry.io/otel/exporters/prometheus v0.57.0/go.mod h1:QpFWz1QxqevfjwzYdbMb4Y1NnlJvqSGwyuU0B4iuc9c=
175+
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.35.0 h1:T0Ec2E+3YZf5bgTNQVet8iTDW7oIk03tXHq+wkwIDnE=
176+
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.35.0/go.mod h1:30v2gqH+vYGJsesLWFov8u47EpYTcIQcBjKpI6pJThg=
163177
go.opentelemetry.io/otel/metric v1.35.0 h1:0znxYu2SNyuMSQT4Y9WDWej0VpcsxkuklLa4/siN90M=
164178
go.opentelemetry.io/otel/metric v1.35.0/go.mod h1:nKVFgxBZ2fReX6IlyW28MgZojkoAkJGaE8CpgeAU3oE=
165179
go.opentelemetry.io/otel/sdk v1.35.0 h1:iPctf8iprVySXSKJffSS79eOjl9pvxV9ZqOWT0QejKY=

‎interceptor/config/tracing.go‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
package config
2+
3+
import (
4+
"github.com/kelseyhightower/envconfig"
5+
)
6+
7+
// Tracing is the configuration for configuring tracing through the interceptor.
8+
type Tracing struct {
9+
// States whether tracing should be enabled, False by default
10+
Enabled bool `envconfig:"OTEL_EXPORTER_OTLP_TRACES_ENABLED" default:"false"`
11+
// Sets what tracing export to use, must be one of: console,http/protobuf, grpc
12+
Exporter string `envconfig:"OTEL_EXPORTER_OTLP_TRACES_PROTOCOL" default:"console"`
13+
}
14+
15+
// Parse parses standard configs using envconfig and returns a pointer to the
16+
// newly created config. Returns nil and a non-nil error if parsing failed
17+
func MustParseTracing() *Tracing {
18+
ret := new(Tracing)
19+
envconfig.MustProcess("", ret)
20+
return ret
21+
}

‎interceptor/handler/upstream.go‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,12 @@ import (
55
"net/http"
66
"net/http/httputil"
77

8+
"go.opentelemetry.io/otel"
9+
"go.opentelemetry.io/otel/attribute"
10+
"go.opentelemetry.io/otel/propagation"
11+
"go.opentelemetry.io/otel/trace"
12+
13+
"github.com/kedacore/http-add-on/interceptor/config"
814
"github.com/kedacore/http-add-on/pkg/util"
915
)
1016

@@ -14,11 +20,13 @@ var (
1420

1521
type Upstream struct {
1622
roundTripper http.RoundTripper
23+
tracingCfg *config.Tracing
1724
}
1825

19-
func NewUpstream(roundTripper http.RoundTripper) *Upstream {
26+
func NewUpstream(roundTripper http.RoundTripper, tracingCfg *config.Tracing) *Upstream {
2027
return &Upstream{
2128
roundTripper: roundTripper,
29+
tracingCfg: tracingCfg,
2230
}
2331
}
2432

@@ -28,6 +36,21 @@ func (uh *Upstream) ServeHTTP(w http.ResponseWriter, r *http.Request) {
2836
r = util.RequestWithLoggerWithName(r, "UpstreamHandler")
2937
ctx := r.Context()
3038

39+
if uh.tracingCfg.Enabled {
40+
p := otel.GetTextMapPropagator()
41+
ctx = p.Extract(ctx, propagation.HeaderCarrier(r.Header))
42+
43+
p.Inject(ctx, propagation.HeaderCarrier(w.Header()))
44+
45+
span := trace.SpanFromContext(ctx)
46+
defer span.End()
47+
48+
serviceValAttr := attribute.String("service", "keda-http-interceptor-proxy-upstream")
49+
coldStartValAttr := attribute.String("cold-start", w.Header().Get("X-KEDA-HTTP-Cold-Start"))
50+
51+
span.SetAttributes(serviceValAttr, coldStartValAttr)
52+
}
53+
3154
stream := util.StreamFromContext(ctx)
3255
if stream == nil {
3356
sh := NewStatic(http.StatusInternalServerError, errNilStream)

0 commit comments

Comments
 (0)