docs: otel metrics (#3096)

ericallam · web-flow · commit 676525279aa7 · 2026-02-20T16:51:48.000Z
diff --git a/docs/config/config-file.mdx b/docs/config/config-file.mdx
@@ -154,21 +154,30 @@ Some ones we recommend:
 
 ### Telemetry Exporters
 
-You can also configure custom telemetry exporters to send your traces and logs to other external services. For example, you can send your logs to [Axiom](https://axiom.co/docs/guides/opentelemetry-nodejs#exporter-instrumentation-ts). First, add the opentelemetry exporter packages to your package.json file:
+You can also configure custom telemetry exporters to send your traces, logs, and metrics to other external services. For example, you can send your logs to [Axiom](https://axiom.co/docs/guides/opentelemetry-nodejs#exporter-instrumentation-ts). First, add the opentelemetry exporter packages to your package.json file:
 
 ```json package.json
 "dependencies": {
   "@opentelemetry/exporter-logs-otlp-http": "0.52.1",
-  "@opentelemetry/exporter-trace-otlp-http": "0.52.1"
+  "@opentelemetry/exporter-trace-otlp-http": "0.52.1",
+  "@opentelemetry/exporter-metrics-otlp-proto": "0.52.1"
 }
 ```
 
+<Note>
+  Axiom's `/v1/metrics` endpoint only supports protobuf (`application/x-protobuf`), not JSON. Use
+  `@opentelemetry/exporter-metrics-otlp-proto` instead of
+  `@opentelemetry/exporter-metrics-otlp-http` for metrics. Traces and logs work fine with the
+  `-http` (JSON) exporters.
+</Note>
+
 Then, configure the exporters in your `trigger.config.ts` file:
 
 ```ts trigger.config.ts
 import { defineConfig } from "@trigger.dev/sdk";
 import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
 import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
 
 // Initialize OTLP trace exporter with the endpoint URL and headers;
 export default defineConfig({
@@ -196,18 +205,28 @@ export default defineConfig({
         },
       }),
     ],
+    metricExporters: [
+      new OTLPMetricExporter({
+        url: "https://api.axiom.co/v1/metrics",
+        headers: {
+          Authorization: `Bearer ${process.env.AXIOM_API_TOKEN}`,
+          "x-axiom-metrics-dataset": process.env.AXIOM_METRICS_DATASET,
+        },
+      }),
+    ],
   },
 });
 ```
 
-Make sure to set the `AXIOM_API_TOKEN` and `AXIOM_DATASET` environment variables in your project.
+Make sure to set the `AXIOM_API_TOKEN`, `AXIOM_DATASET`, and `AXIOM_METRICS_DATASET` environment variables in your project. Axiom requires a separate, dedicated dataset for metrics — you cannot reuse the same dataset for traces/logs and metrics.
 
-It's important to note that you cannot configure exporters using `OTEL_*` environment variables, as they would conflict with our internal telemetry. Instead you should configure the exporters via passing in arguments to the `OTLPTraceExporter` and `OTLPLogExporter` constructors. For example, here is how you can configure exporting to Honeycomb:
+It's important to note that you cannot configure exporters using `OTEL_*` environment variables, as they would conflict with our internal telemetry. Instead you should configure the exporters via passing in arguments to the `OTLPTraceExporter`, `OTLPLogExporter`, and `OTLPMetricExporter` constructors. For example, here is how you can configure exporting to Honeycomb:
 
 ```ts trigger.config.ts
 import { defineConfig } from "@trigger.dev/sdk";
 import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
 import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
 
 // Initialize OTLP trace exporter with the endpoint URL and headers;
 export default defineConfig({
@@ -235,6 +254,15 @@ export default defineConfig({
         },
       }),
     ],
+    metricExporters: [
+      new OTLPMetricExporter({
+        url: "https://api.honeycomb.io/v1/metrics",
+        headers: {
+          "x-honeycomb-team": process.env.HONEYCOMB_API_KEY,
+          "x-honeycomb-dataset": process.env.HONEYCOMB_DATASET,
+        },
+      }),
+    ],
   },
 });
 ```
@@ -465,8 +493,9 @@ export default defineConfig({
 ```
 
 <Note>
-  Any packages that install or build a native binary or use WebAssembly (WASM) should be added to external, as they
-  cannot be bundled. For example, `re2`, `sharp`, `sqlite3`, and WASM packages should be added to external.
+  Any packages that install or build a native binary or use WebAssembly (WASM) should be added to
+  external, as they cannot be bundled. For example, `re2`, `sharp`, `sqlite3`, and WASM packages
+  should be added to external.
 </Note>
 
 ### JSX
diff --git a/docs/insights/metrics.mdx b/docs/insights/metrics.mdx
@@ -9,6 +9,12 @@ In the Trigger.dev dashboard we have built-in dashboards and you can create your
 
 Metrics dashboards are powered by [TRQL queries](/insights/query) with widgets that can be displayed as charts, tables, or single values. They automatically refresh to show the latest data.
 
+### Available metrics data
+
+Trigger.dev automatically collects process metrics (CPU, memory) and Node.js runtime metrics (event loop, heap) for all deployed tasks -- no configuration needed. Requires SDK version **4.4.1 or later**. You can also create custom metrics using the `otel.metrics` API from the SDK.
+
+All of this data is available in the `metrics` table for use in dashboard widgets. See [Logging, tracing & metrics](/logging#metrics) for the full list of automatic metrics and how to create custom ones, or the [Query page](/insights/query#metrics-table-columns) for the `metrics` table schema.
+
 ![The built-in Metrics dashboard](/images/metrics-built-in.png)
 
 ### Visualization types
diff --git a/docs/insights/query.mdx b/docs/insights/query.mdx
@@ -6,7 +6,43 @@ description: "Query allows you to write custom queries against your data using T
 ### Available tables
 
 - `runs`: contains all task run data including status, timing, costs, and metadata
-- `metrics`: contains metrics data for your runs including CPU, memory, and your custom metrics.
+- `metrics`: contains metrics data for your runs including CPU, memory, and your custom metrics
+
+### `metrics` table columns
+
+| Column | Type | Description |
+| :--- | :--- | :--- |
+| `metric_name` | string | Metric identifier (e.g., `process.cpu.utilization`) |
+| `metric_type` | string | `gauge`, `sum`, or `histogram` |
+| `value` | number | The observed value |
+| `bucket_start` | datetime | 10-second aggregation bucket start time |
+| `run_id` | string | Associated run ID |
+| `task_identifier` | string | Task slug |
+| `attempt_number` | number | Attempt number |
+| `machine_id` | string | Machine that produced the metric |
+| `machine_name` | string | Machine preset (e.g., `small-1x`) |
+| `worker_version` | string | Worker version |
+| `environment_type` | string | `PRODUCTION`, `STAGING`, `DEVELOPMENT`, `PREVIEW` |
+| `attributes` | json | Raw JSON attributes for custom data |
+
+See [Logging, tracing & metrics](/logging#automatic-system-and-runtime-metrics) for the full list of automatically collected metrics and how to create custom metrics.
+
+### `prettyFormat()`
+
+Use `prettyFormat()` to format metric values for display:
+
+```sql
+SELECT
+  timeBucket(),
+  prettyFormat(avg(value), 'bytes') AS avg_memory
+FROM metrics
+WHERE metric_name = 'process.memory.usage'
+GROUP BY timeBucket
+ORDER BY timeBucket
+LIMIT 1000
+```
+
+Available format types: `bytes`, `percent`, `duration`, `durationSeconds`, `quantity`, `costInDollars`.
 
 ## Using the Query dashboard
 
@@ -191,7 +227,7 @@ SELECT
   task_identifier,
   avg(value) AS avg_memory
 FROM metrics
-WHERE metric_name = 'system.memory.usage'
+WHERE metric_name = 'process.memory.usage'
 GROUP BY task_identifier
 ORDER BY avg_memory DESC
 LIMIT 20
@@ -500,14 +536,14 @@ LIMIT 1000
 
 ### Memory usage by task (past 7d)
 
-Average memory usage per task identifier over the last 7 days.
+Average process memory usage per task identifier over the last 7 days.
 
 ```sql
 SELECT
   task_identifier,
   avg(value) AS avg_memory
 FROM metrics
-WHERE metric_name = 'system.memory.usage'
+WHERE metric_name = 'process.memory.usage'
 GROUP BY task_identifier
 ORDER BY avg_memory DESC
 LIMIT 20
diff --git a/docs/logging.mdx b/docs/logging.mdx
@@ -1,6 +1,6 @@
 ---
-title: "Logging and tracing"
-description: "How to use the built-in logging and tracing system."
+title: "Logging, tracing & metrics"
+description: "How to use the built-in logging, tracing, and metrics system."
 ---
 
 ![The run log](/images/run-log.png)
@@ -77,3 +77,119 @@ export const customTrace = task({
   },
 });
 ```
+
+## Metrics
+
+Trigger.dev collects system and runtime metrics automatically for deployed tasks, and provides an API for recording custom metrics using OpenTelemetry.
+
+You can view metrics in the [Metrics dashboards](/insights/metrics), query them with [TRQL](/insights/query), and export them to external services via [telemetry exporters](/config/config-file#telemetry-exporters).
+
+### Custom metrics API
+
+Import `otel` from `@trigger.dev/sdk` and use the standard OpenTelemetry Metrics API to create custom instruments.
+
+Create instruments **at module level** (outside the task `run` function) so they are reused across runs:
+
+```ts /trigger/metrics.ts
+import { task, logger, otel } from "@trigger.dev/sdk";
+
+// Create a meter — instruments are created once at module level
+const meter = otel.metrics.getMeter("my-app");
+
+const itemsProcessed = meter.createCounter("items.processed", {
+  description: "Total number of items processed",
+  unit: "items",
+});
+
+const itemDuration = meter.createHistogram("item.duration", {
+  description: "Time spent processing each item",
+  unit: "ms",
+});
+
+const queueDepth = meter.createUpDownCounter("queue.depth", {
+  description: "Current queue depth",
+  unit: "items",
+});
+
+export const processQueue = task({
+  id: "process-queue",
+  run: async (payload: { items: string[] }) => {
+    queueDepth.add(payload.items.length);
+
+    for (const item of payload.items) {
+      const start = performance.now();
+
+      // ... process item ...
+
+      const elapsed = performance.now() - start;
+
+      itemsProcessed.add(1, { "item.type": "order" });
+      itemDuration.record(elapsed, { "item.type": "order" });
+      queueDepth.add(-1);
+    }
+
+    logger.info("Queue processed", { count: payload.items.length });
+  },
+});
+```
+
+#### Available instrument types
+
+| Instrument | Method | Use case |
+| :--- | :--- | :--- |
+| Counter | `meter.createCounter()` | Monotonically increasing values (items processed, requests sent) |
+| Histogram | `meter.createHistogram()` | Distributions of values (durations, sizes) |
+| UpDownCounter | `meter.createUpDownCounter()` | Values that go up and down (queue depth, active connections) |
+
+All instruments accept optional attributes when recording values. Attributes let you break down metrics by dimension (e.g., by item type, status, or region).
+
+### Automatic system and runtime metrics
+
+Trigger.dev automatically collects the following metrics for deployed tasks. No configuration is needed. Requires SDK version **4.4.1 or later**.
+
+| Metric name | Type | Unit | Description |
+| :--- | :--- | :--- | :--- |
+| `process.cpu.utilization` | gauge | ratio | Process CPU usage (0-1) |
+| `process.cpu.time` | counter | seconds | CPU time consumed |
+| `process.memory.usage` | gauge | bytes | Process memory usage |
+| `nodejs.event_loop.utilization` | gauge | ratio | Event loop utilization (0-1) |
+| `nodejs.event_loop.delay.p95` | gauge | seconds | Event loop delay p95 |
+| `nodejs.event_loop.delay.max` | gauge | seconds | Event loop delay max |
+| `nodejs.heap.used` | gauge | bytes | V8 heap used |
+| `nodejs.heap.total` | gauge | bytes | V8 heap total |
+
+<Note>
+In dev mode (`trigger dev`), only `process.*` and custom metrics are available.
+</Note>
+
+### Context attributes
+
+All metrics (both automatic and custom) are tagged with run context so you can filter and group them:
+
+- `run_id` — the run that produced the metric
+- `task_identifier` — the task slug
+- `attempt_number` — the attempt number
+- `machine_name` — the machine preset (e.g., `small-1x`)
+- `worker_version` — the deployed worker version
+- `environment_type` — `PRODUCTION`, `STAGING`, `DEVELOPMENT`, or `PREVIEW`
+
+### Querying metrics
+
+Use [TRQL](/insights/query) to query metrics data. For example, to see average CPU utilization over time:
+
+```sql
+SELECT
+  timeBucket(),
+  avg(value) AS avg_cpu
+FROM metrics
+WHERE metric_name = 'process.cpu.utilization'
+GROUP BY timeBucket
+ORDER BY timeBucket
+LIMIT 1000
+```
+
+See the [Query page](/insights/query#metrics-table-columns) for the full `metrics` table schema.
+
+### Exporting metrics
+
+You can send metrics to external observability services (Axiom, Honeycomb, Datadog, etc.) by configuring [telemetry exporters](/config/config-file#telemetry-exporters) in your `trigger.config.ts`.
