Skip to content

Commit

Permalink
[DOC] Update Set up test app to use agent flow (#3228)
Browse files Browse the repository at this point in the history 
* Update Set up test app to use agent flow

* Apply suggestions from code review

* Correct a distributor reference to query frontend.

Also makes a creation step for `values.yaml` generic.

Signed-off-by: Heds Simons <hedley.simons@grafana.com>

* Apply suggestions from code review

* Apply suggestions from code review

---------

Signed-off-by: Heds Simons <hedley.simons@grafana.com>
Co-authored-by: Heds Simons <hedley.simons@grafana.com>
  • Loading branch information
@knylander-grafana @hedss
knylander-grafana and hedss authored Jan 16, 2024
1 parent d936515 commit e11ceb9
Showing 1 changed file with 110 additions and 91 deletions.
201 changes: 110 additions & 91 deletions docs/sources/tempo/setup/set-up-test-app.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,110 +15,98 @@ For example, if you [set up Tempo using the Kubernetes with Tanka procedure]({{<

You'll need:

* Grafana 9.0.0 or higher
* Microservice deployments require the Tempo querier URL, for example: `http://query-frontend.tempo.svc.cluster.local:3200`
* Grafana 10.0.0 or higher
* Microservice deployments require the Tempo querier URL, for example: `http://tempo-cluster-query-frontend.tempo.svc.cluster.local:3100/`
* [OpenTelemetry telemetrygen](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) for generating tracing data

Refer to [Deploy Grafana on Kubernetes](/docs/grafana/latest/setup-grafana/installation/kubernetes/#deploy-grafana-on-kubernetes) if you are using Kubernetes.
Otherwise, refer to [Install Grafana](/docs/grafana/latest/installation/) for more information.

## Set up `remote_write` to your Tempo cluster

To enable writes to your cluster:

1. Add a `remote_write` configuration snippet to the configuration file of an existing Grafana Agent.

If you do not have an existing traces collector, refer to [Set up with Grafana Agent](/docs/agent/latest/set-up/).
For Kubernetes, refer to the [Grafana Agent Traces Kubernetes quick start guide](/docs/grafana-cloud/kubernetes-monitoring/agent-k8s/k8s_agent_traces/).

The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

```yaml
kind: ConfigMap
metadata:
name: grafana-agent-traces
apiVersion: v1
data:
agent.yaml: |
traces:
configs:
- batch:
send_batch_size: 1000
timeout: 5s
name: default
receivers:
jaeger:
protocols:
grpc: null
thrift_binary: null
thrift_compact: null
thrift_http: null
opencensus: null
otlp:
protocols:
grpc: null
http: null
zipkin: null
remote_write:
- endpoint: <tempoDistributorServiceEndpoint>
insecure: true # only add this if TLS is not used
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
```
If you have followed the [Tanka Tempo installation example]({{< relref "../setup/tanka" >}}), then the `endpoint` value would be:

```bash
distributor.tempo.svc.cluster.local:4317
```

1. Apply the ConfigMap with:
## Configure Grafana Agent Flow to remote-write to Tempo

We'll use a [Grafana Agent Helm chart](/docs/agent/latest/flow/setup/install/kubernetes) deployment to send traces to Tempo.

To do this, you need to create a configuration that can be used by the Agent to receive and export traces in OTLP protobuf format.

1. Create a new `values.yaml` file which we'll use as part of the Agent install.

1. Edit the `values.yaml` file and add the following configuration to it:
```yaml
agent:
extraPorts:
- name: otlp-grpc
port: 4317
targetPort: 4317
protocol: TCP
configMap:
create: true
content: |-
// Creates a receiver for OTLP gRPC.
// You can easily add receivers for other protocols by using the correct component
// from the reference list at: https://grafana.com/docs/agent/latest/flow/reference/components/
otelcol.receiver.otlp "otlp_receiver" {
// Listen on all available bindable addresses on port 4317 (which is the
// default OTLP gRPC port) for the OTLP protocol.
grpc {
endpoint = "0.0.0.0:4317"
}
// Output straight to the OTLP gRPC exporter. We would usually do some processing
// first, most likely batch processing, but for this example we pass it straight
// through.
output {
traces = [
otelcol.exporter.otlp.tempo.input,
]
}
}
// Define an OTLP gRPC exporter to send all received traces to GET.
// The unique label 'tempo' is added to uniquely identify this exporter.
otelcol.exporter.otlp "tempo" {
// Define the client for exporting.
client {
// Send to the locally running Tempo instance, on port 4317 (OTLP gRPC).
endpoint = "http://tempo-cluster-distributor.tempo.svc.cluster.local:4317"
// Disable TLS for OTLP remote write.
tls {
// The connection is insecure.
insecure = true
// Do not verify TLS certificates when connecting.
insecure_skip_verify = true
}
}
}
```
Ensure that you use the specific namespace you've installed Tempo in for the OTLP exporter. In the line:
```yaml
endpoint = "http://tempo-cluster-distributor.tempo.svc.cluster.local:3100"
```
change `tempo` to reference the namespace where Tempo is installed, for example: `http://tempo-cluster-distributor.my-tempo-namespaces.svc.cluster.local:3100`.

```bash
kubectl apply --namespace default -f agent.yaml
```
1. Deploy the Agent using Helm:
```bash
helm install -f values.yaml grafana-agent grafana/grafana-agent
```
If you wish to deploy the agent into a specific namespace, make sure to create the namespace first and specify it to Helm by appending `--namespace=<grafana-agent-namespace>` to the end of the command.

1. Deploy Grafana Agent using the procedures from the relevant instructions above.

## Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

1. Navigate to **Configuration ≫ Data Sources**.
1. Navigate to **Connections** > **Data Sources**.

1. Click on **Add data source**.

1. Select **Tempo**.

1. Set the URL to `http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/`, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the [Tanka Tempo installation example]({{< relref "../setup/tanka.md" >}}), this will be: `http://query-frontend.tempo.svc.cluster.local:3200/`
1. Set the URL to `http://<TEMPO-QUERY-FRONTEND-SERVICE>:<HTTP-LISTEN-PORT>/`, filling in the path to Tempo's query frontend service, and the configured HTTP API prefix. If you have followed the [Deploy Tempo with Helm installation example]({{< relref "../setup/helm-chart.md" >}}), the query frontend service's URL will look something like this: `http://tempo-cluster-query-frontend.<namespace>.svc.cluster.local:3100`

1. Click **Save & Test**.

You should see a message that says `Data source is working`.

If you see an error that says `Data source is not working: failed to get trace with id: 0`, check your Grafana version.

To fix the error, [upgrade your Grafana to 9.0 or later](/docs/grafana/latest/setup-grafana/upgrade-grafana/).

## Visualize your data

Once you have created a data source, you can visualize your traces in the **Grafana Explore** page.
Expand All @@ -129,35 +117,66 @@ For more information, refer to [Tempo in Grafana]({{< relref "../getting-started
Next, you can use [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) to generate tracing data to test your Tempo installation.

In the following instructions we assume the endpoints for both the Grafana Agent and the Tempo distributor are those described above, for example:
* `grafana-agent-traces.default.svc.cluster.local` for Grafana Agent
* `distributor.tempo.svc.cluster.local` for the Tempo distributor
Replace these appropriately if you have altered the endpoint targets.
* `grafana-agent.grafana-agent.svc.cluster.local` for Grafana Agent
* `tempo-cluster-distributor.tempo.svc.cluster.local` for the Tempo distributor
Replace these appropriately if you have altered the endpoint targets for the following examples.

1. Install `telemetrygen` using the [installation procedure](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen).
**NOTE**: You do not need to configure an OpenTelemetry Collector as we are using the Grafana Agent.
**NOTE**: You don't need to configure an OpenTelemetry Collector as we are using the Grafana Agent.

1. Generate traces using `telemtrygen`:
```bash
telemetrygen traces --otlp-insecure --rate 20 --duration 5s grafana-agent-traces.default.svc.cluster.local:4317
telemetrygen traces --otlp-insecure --rate 20 --duration 5s grafana-agent.grafana-agent.svc.cluster.local:4317
```
This configuration sends traces to Grafana Agent for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send the trace directly to the Tempo database without using Grafana Agent as a collector by using the following:
```bash
telemetrygen traces --otlp-insecure --rate 20 --duration 5s distributor.tempo.svc.cluster.local:4317
telemetrygen traces --otlp-insecure --rate 20 --duration 5s tempo-cluster-distributor.tempo.svc.cluster.local:4317
```

### View tracing data in Grafana
If you're running `telemetrygen` on your local machine, ensure that you first port-forward to the relevant Agent or Tempo distributor service, eg:
```bash
kubectl port-forward services/grafana-agent 4317:4317 --namespace grafana-agent
```

To view the tracing data:

1. Go to Grafana and select the **Explore** menu item.
1. Go to Grafana and select **Explore**.

1. Select the **Tempo data source** from the list of data sources.

1. Copy the trace ID into the **Trace ID** edit field.
1. Select the `Search` Query type.

1. Select **Run query**.

1. Confirm that traces are displayed in the traces **Explore** panel. You should see 5 seconds worth of traces, 100 traces in total per run of `telemetrygen`.

### Test your configuration using the Intro to MLTP application

The Intro to MLTP application provides an example five-service appliation generates data for Tempo, Mimir, Loki, and Pyroscope.
This procedure installs the application on your cluster so you can generate meaningful test data.

1. Navigate to https://github.com/grafana/intro-to-mltp to get the Kubernetes manifests for the Intro to MLTP application.
1. Clone the repository using commands similar to the ones below:
```bash
git clone git+ssh://github.com/grafana/intro-to-mltp
cp intro-to-mltp/k8s/mythical/* ~/tmp/intro-to-mltp-k8s
```
1. Change to the cloned repository: `cd intro-to-mltp/k8s/mythical`
1. In the `mythical-beasts-deployment.yaml` manifest, alter each `TRACING_COLLECTOR_HOST` environment variable instance value to point to the Grafana Agent location. For example, based on the a Grafana Agent install in the default namespace called and with a Helm installation called `test`:
```yaml
- env:
...
- name: TRACING_COLLECTOR_HOST
value: grafana-agent.grafana-agent.svc.cluster.local
```
1. Deploy the Intro to MLTP application. It deploys into the default namespace.
```bash
kubectl apply -f mythical-beasts-service.yaml,mythical-beasts-persistentvolumeclaim.yaml,mythical-beasts-deployment.yaml
```
1. Once the application is deployed, go to Grafana Enterprise and select the **Explore** menu item.
1. Select the **Tempo data source** from the list of data sources.
1. Select the `Search` Query type for the data source.
1. Select **Run query**.
1. Traces from the application will be displayed in the traces **Explore** panel.

0 comments on commit e11ceb9

Please sign in to comment.