Add anonymous metadata based product analytics for local and self-host (#1498)

* Add anonymous product analytics for local and self-host

* Track artifact usage in local and self-host analytics

* Drop integration slugs from analytics events

* Document telemetry in TELEMETRY.md

* Ship platform-node as a runtime dependency of analytics

Author: RhysSullivan

.changeset/heavy-planets-tell.md

@@ -0,0 +1,9 @@
+---
+"@executor-js/analytics": patch
+"@executor-js/sdk": patch
+"@executor-js/api": patch
+"@executor-js/host-selfhost": patch
+"executor": patch
+---
+
+Add anonymous product analytics to the local daemon (CLI + desktop) and self-host: execution counts split by MCP/API plane, toolkit usage, integration add/remove, and artifact usage (created/viewed/updated/deleted, attributed to agent tools vs the console UI), filed under a persisted per-install anonymous id. Opt out with DO_NOT_TRACK or EXECUTOR_DISABLE_ANALYTICS.

TELEMETRY.md

@@ -0,0 +1,101 @@
+# Telemetry
+
+Executor's local products (the CLI, the desktop app) and self-hosted
+deployments send a small set of anonymous usage events. This document is the
+complete account of what is sent, what is deliberately not sent, why the
+feature exists, and how to turn it off.
+
+The hosted cloud product has its own browser-side analytics, disclosed
+separately in its terms; this document covers the software that runs on your
+machines.
+
+## Why
+
+Executor is used mostly outside our infrastructure. Without some signal from
+local and self-hosted installs, every product decision about them is a guess:
+we cannot tell whether a feature ships broken, whether anyone uses toolkits,
+whether executions fail at unusual rates after a release, or whether the
+product is growing anywhere except cloud.
+
+The events exist to answer exactly one kind of question: **how are product
+features being used?** They are metadata about the product, not about you.
+Anything that would answer "what is this user doing" — which APIs you call,
+what your tools are named, what your code does — is out of scope by design,
+not by omission.
+
+## What is sent
+
+Each event carries its named properties plus, on every event: a random
+per-install id, the product surface (`cli`, `desktop`, or `selfhost`), the
+release channel, and the app version.
+
+| Event                 | Properties                                       | Meaning                                                                                                                                         |
+| --------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| `execution_completed` | `ok`, `plane` (`mcp`/`api`), `toolkit` (boolean) | A code execution finished, split by whether an agent (MCP) or a human-facing API triggered it, and whether a toolkit-scoped endpoint served it. |
+| `integration_added`   | `plugin_key`                                     | An integration was added, by kind (`openapi`, `mcp`, `graphql`, ...).                                                                           |
+| `integration_removed` | `plugin_key`                                     | An integration was removed, by kind.                                                                                                            |
+| `artifact_created`    | `via` (`agent`/`ui`)                             | A generative-UI artifact was saved.                                                                                                             |
+| `artifact_viewed`     | `via`                                            | An artifact was opened for its content.                                                                                                         |
+| `artifact_updated`    | `via`                                            | An artifact was overwritten or renamed.                                                                                                         |
+| `artifact_deleted`    | `via`                                            | An artifact was deleted.                                                                                                                        |
+
+That table is exhaustive. The typed catalog the code compiles against is
+[`packages/core/analytics/src/events.ts`](packages/core/analytics/src/events.ts);
+an event that is not in that file cannot be sent.
+
+## What is never sent
+
+- No code, tool arguments, tool results, or error messages.
+- No secrets, tokens, or credentials.
+- No names you typed: no integration slugs, connection names, toolkit slugs,
+  tool names, or artifact titles. The **kind** of integration (`openapi`,
+  `mcp`, ...) is a product question; **which** service you connected it to is
+  your business.
+- No identity: no emails, usernames, hostnames, IP-derived location, or org
+  names. Events are marked so the analytics backend builds no person profile.
+
+## The anonymous id
+
+A random UUID is minted the first time the daemon or server starts and stored
+as `analytics-id` in the data directory (`~/.executor` for local installs, the
+configured data dir for self-host). It exists so that ten events from one
+install count as one install, not ten. It is not derived from your machine,
+account, or network, and deleting the file resets it. When telemetry is
+disabled the file is never created.
+
+## Opting out
+
+Set either environment variable to `1`, `true`, or `yes`:
+
+- `DO_NOT_TRACK` — the [cross-tool convention](https://consoledonottrack.com),
+  which Executor also honors for its other outbound calls (crash reporting,
+  the integrations.sh catalog fetch).
+- `EXECUTOR_DISABLE_ANALYTICS` — telemetry only, if you want the catalog
+  fetch and crash reporting to keep working.
+
+Opting out is total: the analytics service becomes a no-op, nothing is
+buffered, nothing is sent, and no id file is written. The CLI's managed
+service (launchd/systemd) forwards both variables into the supervised
+daemon's environment, so an opted-out install stays opted out.
+
+## Delivery mechanics
+
+Events buffer in memory (bounded) and flush in batches on a fixed cadence,
+plus once at shutdown. Delivery is best-effort: a failed flush re-queues and
+retries later, failures are swallowed, and no user-facing operation ever
+waits on — or can be failed by — analytics. The ingest endpoint is PostHog
+(`us.i.posthog.com`); the project key in the source identifies the project
+and grants no read access.
+
+## Where the line is enforced
+
+Structurally, not by convention:
+
+- The event catalog is a closed, typed interface — adding a property means
+  editing [`events.ts`](packages/core/analytics/src/events.ts) in a reviewed
+  change, next to the property rules at the top of that file.
+- Attribution (`plane`, `via`) is bound where each serving surface is
+  composed, so events cannot carry request-controlled labels.
+- Observers hang off neutral seams (an engine wrapper, post-commit hooks); the
+  core SDK carries no analytics vocabulary and hosts that do not opt in — like
+  every test — compose with no analytics at all.

apps/cli/src/service.ts

@@ -177,6 +177,10 @@ const serviceEnvironment = (
     "EXECUTOR_SENTRY_RELEASE",
     "EXECUTOR_SENTRY_ENVIRONMENT",
     "EXECUTOR_RUN_ID",
+    // Analytics opt-out must survive into the supervised unit's minimal env,
+    // or an opted-out install would silently re-enable analytics under launchd.
+    "DO_NOT_TRACK",
+    "EXECUTOR_DISABLE_ANALYTICS",
   ] as const;
   const passThrough = Object.fromEntries(
     passThroughKeys.flatMap((key) => {

apps/host-selfhost/package.json

@@ -22,6 +22,7 @@
     "@cloudflare/worker-bundler": "0.2.1",
     "@effect/atom-react": "catalog:",
     "@effect/platform-bun": "catalog:",
+    "@executor-js/analytics": "workspace:*",
     "@executor-js/api": "workspace:*",
     "@executor-js/app": "workspace:*",
     "@executor-js/execution": "workspace:*",

apps/host-selfhost/src/analytics.ts

@@ -0,0 +1,83 @@
+import { Effect, Layer, ManagedRuntime } from "effect";
+
+import {
+  Analytics,
+  defaultLayer as analyticsDefaultLayer,
+  withExecutionAnalytics,
+  type AnalyticsService,
+} from "@executor-js/analytics";
+import { EngineDecorator } from "@executor-js/api/server";
+
+import packageJson from "../package.json" with { type: "json" };
+import { resolveDataDir } from "./config";
+
+// ---------------------------------------------------------------------------
+// Self-host product analytics: one anonymous per-install service for the
+// server lifetime. The anonymous id persists next to the instance's other
+// first-boot state (secret.key) under the data dir, so an instance counts
+// once across restarts. Module-singleton runtime; dispose flushes at shutdown.
+// ---------------------------------------------------------------------------
+
+// Deployed self-host builds are published releases; the prerelease heuristic
+// mirrors apps/local/src/installation.ts.
+const VERSION: string = packageJson.version;
+const CHANNEL = VERSION.includes("-") ? ("beta" as const) : ("stable" as const);
+
+let analyticsRuntime: ManagedRuntime.ManagedRuntime<Analytics, never> | null = null;
+
+const getAnalyticsRuntime = (): ManagedRuntime.ManagedRuntime<Analytics, never> => {
+  if (analyticsRuntime) return analyticsRuntime;
+  analyticsRuntime = ManagedRuntime.make(
+    analyticsDefaultLayer({
+      surface: "selfhost",
+      version: VERSION,
+      channel: CHANNEL,
+      dataDir: resolveDataDir(),
+    }),
+  );
+  return analyticsRuntime;
+};
+
+/**
+ * Lazy facade over the instance's shared service. `record` forks into the
+ * module runtime (fire-and-forget) so no request path ever waits on the
+ * analytics layer; ordering within the runtime preserves event order.
+ */
+export const selfHostAnalytics: AnalyticsService = {
+  record: (name, properties) =>
+    Effect.sync(() => {
+      getAnalyticsRuntime().runFork(
+        Effect.flatMap(Analytics.asEffect(), (analytics) => analytics.record(name, properties)),
+      );
+    }),
+  flush: Effect.promise(() =>
+    getAnalyticsRuntime().runPromise(
+      Effect.flatMap(Analytics.asEffect(), (analytics) => analytics.flush),
+    ),
+  ),
+};
+
+/** Flush and dispose the analytics runtime (server shutdown). */
+export const disposeAnalytics = async (): Promise<void> => {
+  const runtime = analyticsRuntime;
+  if (!runtime) return;
+  analyticsRuntime = null;
+  await Effect.runPromise(Effect.promise(() => runtime.dispose()).pipe(Effect.ignoreCause()));
+};
+
+/**
+ * The execution-analytics `EngineDecorator`: every engine the shared stack
+ * builds gets wrapped once, with the plane derived from what the stack was
+ * built to serve — an MCP session carries its `mcpResource`, the HTTP
+ * executions plane has none. The wrap point IS the plane, so misattribution
+ * is structurally impossible.
+ */
+export const SelfHostAnalyticsEngineDecorator: Layer.Layer<EngineDecorator> = Layer.succeed(
+  EngineDecorator,
+)({
+  decorate: (engine, _identity, context) =>
+    withExecutionAnalytics(engine, selfHostAnalytics, {
+      plane: context.mcpResource === undefined ? "api" : "mcp",
+      toolkit: context.mcpResource?.kind === "toolkit",
+    }),
+});

apps/host-selfhost/src/app.ts

@@ -2,7 +2,12 @@ import { HttpApiSwagger } from "effect/unstable/httpapi";
 import { HttpEffect, HttpRouter } from "effect/unstable/http";
 import { Effect, Layer } from "effect";
 
-import { composePluginApi, ExecutorApp, textFailureStrategy } from "@executor-js/api/server";
+import {
+  ArtifactUsageObserver,
+  composePluginApi,
+  ExecutorApp,
+  textFailureStrategy,
+} from "@executor-js/api/server";
 
 import { runSqliteDataMigrations } from "@executor-js/sdk";
 
@@ -14,6 +19,7 @@ import { makeSelfHostSystemApiLayer } from "./system/handlers";
 import { selfHostAccountMiddleware } from "./account";
 import { loadConfig, SELF_HOST_NAMESPACE, SELF_HOST_SCHEMA_VERSION } from "./config";
 import { createSelfHostDb, SelfHostDb, SelfHostDbProvider } from "./db/self-host-db";
+import { selfHostAnalytics, SelfHostAnalyticsEngineDecorator } from "./analytics";
 import {
   SelfHostCodeExecutorProvider,
   SelfHostHostConfig,
@@ -98,7 +104,12 @@ export const makeSelfHostApp = async (options: MakeSelfHostAppOptions = {}) => {
       identity: identityLayer,
       account: selfHostAccountMiddleware(betterAuth),
       db: SelfHostDbProvider,
-      engine: { codeExecutor: SelfHostCodeExecutorProvider }, // decorator defaults to no-op (no metering)
+      engine: {
+        codeExecutor: SelfHostCodeExecutorProvider,
+        // Anonymous execution analytics (this seam is the HTTP plane; the MCP
+        // plane's decorator is wired in mcp/session-store.ts's stack layer).
+        decorator: SelfHostAnalyticsEngineDecorator,
+      },
       mcp: { auth: mcp.auth, sessions: mcp.sessions, reporter: mcp.reporter },
       plugins: { provider: SelfHostPluginsProvider, config: SelfHostHostConfig },
       errorCapture: ErrorCaptureLive,
@@ -131,8 +142,16 @@ export const makeSelfHostApp = async (options: MakeSelfHostAppOptions = {}) => {
     config: { mountPrefix: "/api", failure: textFailureStrategy },
     // The boot-scoped context provideMerge'd under everything: the long-lived DB
     // handle (read by the DbProvider seam, Better Auth, and the MCP store) + the
-    // resolved identity (captured once by the execution middleware + MCP auth).
-    boot: Layer.merge(Layer.succeed(SelfHostDb)(dbHandle), identityLayer),
+    // resolved identity (captured once by the execution middleware + MCP auth)
+    // + the artifact-usage observer (this HTTP plane is the console UI's data
+    // layer, so operations it serves file as `via: "ui"`).
+    boot: Layer.mergeAll(
+      Layer.succeed(SelfHostDb)(dbHandle),
+      identityLayer,
+      Layer.succeed(ArtifactUsageObserver)((action) =>
+        selfHostAnalytics.record(`artifact_${action}`, { via: "ui" }),
+      ),
+    ),
   });
 
   return {

apps/host-selfhost/src/execution.ts

@@ -4,13 +4,13 @@ import {
   CodeExecutorProvider,
   DbProvider,
   EngineDecorator,
-  EngineDecoratorNoop,
   HostConfig,
   PluginsProvider,
 } from "@executor-js/api/server";
 import { makeQuickJsExecutor } from "@executor-js/runtime-quickjs";
 
 import executorConfig from "../executor.config";
+import { selfHostAnalytics, SelfHostAnalyticsEngineDecorator } from "./analytics";
 import { SelfHostDb, SelfHostDbProvider } from "./db/self-host-db";
 import { loadConfig } from "./config";
 
@@ -21,7 +21,7 @@ import { loadConfig } from "./config";
 //   makeScopedExecutor -> createExecutionEngine -> EngineDecorator.decorate.
 // Self-host just supplies the five seam Layers it reads from. Differences from
 // cloud: the QuickJS in-process code substrate (vs the Cloudflare dynamic
-// worker) and a NO-OP engine decorator (no usage metering).
+// worker) and an analytics engine decorator instead of cloud's usage metering.
 //
 //   - DbProvider          -> SelfHostDbProvider: projects the long-lived
 //                            libSQL handle (built once at boot, see db/). The
@@ -33,7 +33,8 @@ import { loadConfig } from "./config";
 //   - HostConfig           -> `{ allowLocalNetwork, webBaseUrl }` from
 //                            `loadConfig()`.
 //   - CodeExecutorProvider -> `makeQuickJsExecutor()`.
-//   - EngineDecorator      -> no-op (self-host does not meter executions).
+//   - EngineDecorator      -> execution analytics (anonymous per-install
+//                             counters; see ./analytics.ts).
 // ---------------------------------------------------------------------------
 
 export { makeExecutionStack } from "@executor-js/api/server";
@@ -54,6 +55,11 @@ export const SelfHostHostConfig: Layer.Layer<HostConfig> = Layer.sync(HostConfig
     allowLocalNetwork: config.allowLocalNetwork,
     webBaseUrl: config.webBaseUrl,
     oauthCallbackPath: "/api/oauth/callback",
+    onIntegrationChange: (event) =>
+      selfHostAnalytics.record(
+        event.kind === "added" ? "integration_added" : "integration_removed",
+        { plugin_key: event.pluginKey },
+      ),
   };
 });
 
@@ -82,4 +88,8 @@ export const SelfHostExecutionStackLayer: Layer.Layer<
   DbProvider | PluginsProvider | HostConfig | CodeExecutorProvider | EngineDecorator,
   never,
   SelfHostDb
-> = Layer.mergeAll(SelfHostScopedExecutorSeams, SelfHostCodeExecutorProvider, EngineDecoratorNoop);
+> = Layer.mergeAll(
+  SelfHostScopedExecutorSeams,
+  SelfHostCodeExecutorProvider,
+  SelfHostAnalyticsEngineDecorator,
+);

apps/host-selfhost/src/mcp/session-store.ts

@@ -8,6 +8,7 @@ import {
   type InMemoryMcpSessionStore,
 } from "@executor-js/host-mcp/in-memory-session-store";
 
+import { selfHostAnalytics } from "../analytics";
 import { ErrorCaptureLive } from "../observability";
 import { SelfHostDb, type SelfHostDbHandle } from "../db/self-host-db";
 import { SelfHostExecutionStackLayer } from "../execution";
@@ -39,7 +40,13 @@ export const makeSelfHostMcpSessionStore = (
   makeInMemoryMcpSessionStore(
     makeMcpBuildServer(
       SelfHostExecutionStackLayer.pipe(Layer.provide(Layer.succeed(SelfHostDb)(db))),
-      { loadAppShellHtml: loadMcpAppsShellHtml, smokeRenderArtifact },
+      {
+        loadAppShellHtml: loadMcpAppsShellHtml,
+        smokeRenderArtifact,
+        // Artifact operations on the MCP plane come from an agent's tools.
+        onArtifactUsage: (action) =>
+          selfHostAnalytics.record(`artifact_${action}`, { via: "agent" }),
+      },
     ),
     { webBaseUrl },
   );

apps/host-selfhost/src/serve.ts

@@ -26,6 +26,7 @@ import {
 import { BunFileSystem, BunHttpServer, BunPath, BunRuntime } from "@effect/platform-bun";
 import { Effect, Layer } from "effect";
 
+import { disposeAnalytics } from "./analytics";
 import { makeSelfHostApp } from "./app";
 import { loadConfig } from "./config";
 import type { BetterAuthHandle } from "./auth";
@@ -120,6 +121,11 @@ export const startServer = async (): Promise<void> => {
     cacheControl: "no-cache",
   }).pipe(Layer.provide(BunFileSystem.layer), Layer.provide(BunPath.layer));
 
+  // Server-scope finalizer: flush buffered analytics on graceful shutdown.
+  const AnalyticsFlushLive = Layer.effectDiscard(
+    Effect.addFinalizer(() => Effect.promise(() => disposeAnalytics())),
+  );
+
   const ServerLive = HttpRouter.serve(Layer.mergeAll(AppLayer, AssetsLive, SpaLive), {
     middleware: selfHostHttpMiddleware(betterAuth),
   }).pipe(
@@ -128,7 +134,7 @@ export const startServer = async (): Promise<void> => {
     ),
   );
 
-  await BunRuntime.runMain(Layer.launch(ServerLive));
+  await BunRuntime.runMain(Layer.launch(Layer.merge(ServerLive, AnalyticsFlushLive)));
 };
 
 if (import.meta.main) {

apps/local/package.json

@@ -22,6 +22,7 @@
   "dependencies": {
     "@effect/atom-react": "catalog:",
     "@effect/platform-node": "catalog:",
+    "@executor-js/analytics": "workspace:*",
     "@executor-js/api": "workspace:*",
     "@executor-js/app": "workspace:*",
     "@executor-js/config": "workspace:*",