Feat/146 implement file system arrays

.claude/skills/run-cc-backend/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: run-cc-backend
+description: Build, start, run and drive cc-backend end to end — an isolated server with the internal metric store, ingest line protocol, start jobs, query GraphQL/REST, smoke-test metric queries and scopes. Use when asked to run, start, launch, smoke-test or live-check cc-backend, or to verify a change against the running server rather than only the test suite.
+---
+
+cc-backend is a Go HTTP server (REST + GraphQL + embedded Svelte UI). Agents drive it
+through `.claude/skills/run-cc-backend/driver.sh`, which builds the binary, runs the
+server in the background in an isolated run directory, and wraps `curl` for line-protocol
+ingestion, job start, GraphQL and metric-store debugging.
+
+All paths are relative to the repo root. The driver never touches the repo's `./var`
+(it may hold the user's real data): binary, config, DB, job archive and checkpoints all
+live in `$CC_RUN_DIR` (default `$TMPDIR/cc-backend-run`).
+
+## Prerequisites
+
+Verified on macOS (darwin/arm64) with Go 1.27.1; `go.mod` needs Go ≥ 1.26.3. Also needs
+`python3`, `curl` and `bash` on `PATH`. The frontend does **not** need to be built: the
+server, REST and GraphQL run fine without `web/frontend/public/build` (only the web UI
+pages lack their bundles).
+
+## Run (agent path)
+
+One command, end to end (~13 s; exit code 0 means healthy):
+
+```bash
+.claude/skills/run-cc-backend/driver.sh smoke
+```
+
+It runs `setup`, `start`, ingests 11 one-minute samples of `fs_read_bw` for `/home` (1) and
+`/scratch` (2) on host `h1`, starts running job 4242, waits for it to be committed,
+queries `jobMetrics(scopes: [node, filesystem])`, asserts per-mount series 1 and 2 and a
+node series 3 without an id, all with no gaps, prints `SMOKE OK`, and stops the server. On a failed
+assertion it prints `FAIL: ...`, exits 1, and still stops the server.
+
+Step by step (a running job on `h1` with ten minutes of per-mount data):
+
+```bash
+D=.claude/skills/run-cc-backend/driver.sh
+$D setup                       # build, config, archive, DB, user tester/testpw, JWT
+$D start                       # background server on 127.0.0.1:18080, waits for ready
+
+NOW=$(python3 -c 'import time; print(int(time.time()) // 60 * 60)'); START=$((NOW - 600))
+python3 -c "
+for ts in range($START, $NOW + 1, 60):
+    for mnt, v in (('/home', 1), ('/scratch', 2)):
+        print(f'fs_read_bw,cluster=devcluster,hostname=h1,type=filesystem,type-id={mnt} value={v} {ts}')
+" | $D write -                 # line protocol from stdin (or: $D write FILE)
+
+cat > "${TMPDIR:-/tmp}/job.json" <<JSON
+{"cluster":"devcluster","subCluster":"a","jobId":4243,"jobState":"running","numNodes":1,
+ "numHwthreads":4,"resources":[{"hostname":"h1"}],"startTime":$START,"submitTime":$START,
+ "user":"tester","project":"p","walltime":86400}
+JSON
+$D start-job "${TMPDIR:-/tmp}/job.json"   # POST /api/jobs/start_job/
+ID=$($D wait-job 4243)                    # db id once the job is committed
+$D gql "{ jobMetrics(id: \"$ID\", metrics: [\"fs_read_bw\"], scopes: [node, filesystem]) { scope metric { series { id data } } } }"
+$D debug h1                    # metric store levels and buffers of host h1
+$D reset-store                 # stop, wipe checkpoints, start with an empty store
+$D stop                        # waits for the process to exit
+```
+
+| command | what it does |
+|---|---|
+| `setup` | rebuilds the run dir from scratch (refuses to delete a dir it did not create) |
+| `start` / `stop` | background launch with readiness poll on `/login`; stop waits for the PID to exit |
+| `write FILE\|-` | `POST /api/write?cluster=devcluster`, Bearer JWT |
+| `start-job FILE` | `POST /api/jobs/start_job/` with a JSON payload |
+| `wait-job JOBID` | polls GraphQL `jobs` until the job appears, prints its `id` |
+| `gql 'QUERY'` | raw GraphQL query text, JSON-wrapped and posted to `/query` |
+| `debug HOST` | `GET /api/debug?selector=devcluster:HOST` |
+| `reset-store` | empty metric store, keeps DB and jobs |
+| `smoke` | the full flow above, with assertions |
+
+Overrides: `CC_RUN_DIR=/some/dir` and `CC_PORT=18081` (both verified). Log:
+`$CC_RUN_DIR/server.log`; token: `$CC_RUN_DIR/token`; last smoke result:
+`$CC_RUN_DIR/result.json`.
+
+The fixture cluster `devcluster` is `.claude/skills/run-cc-backend/cluster.json`:
+subcluster `a` = host `h1` (filesystems `/home`, `/scratch`, accelerators `0`,`1`,
+network `ib0`), `b` = `h2` (`/work`), `c` = `h3` (no devices). Metrics: `flops_any`
+(hwthread), `fs_read_bw` (filesystem, sum), `acc_util` (accelerator, avg). Edit it to
+test other topologies; `setup` copies it into the archive.
+
+## Run (human path)
+
+`./startDemo.sh` and `./cc-backend -server -dev` run against the repo's own
+`./config.json` and `./var`, which may hold the user's real data. This skill does not
+exercise them; agents use the driver.
+
+## Test
+
+```bash
+go test ./...
+```
+
+All packages pass. The run rewrites `internal/repository/testdata/job.db`; restore it
+with `git checkout -- internal/repository/testdata/job.db` before committing.
+
+## Gotchas
+
+- **Device data must be tagged `type=<device>,type-id=<id>`** (as for accelerators), e.g.
+  `fs_read_bw,cluster=devcluster,hostname=h1,type=filesystem,type-id=/home value=1 <ts>`.
+  `type=node,stype=filesystem,stype-id=…` lands in the host buffer itself, not on a
+  device level, so filesystem-scope queries return nothing.
+- **Running jobs are invisible to GraphQL until committed.** `start_job` writes to the job
+  cache; the commit service moves it every `cron.commit-job-worker` (default 2 min). The
+  driver config sets `5s`; `wait-job` polls for it.
+- **Secrets in the caller's env override `config.json`.** An exported `JWT_PUBLIC_KEY` /
+  `JWT_PRIVATE_KEY` (or `*_FILE`) would make the minted token invalid; the driver unsets
+  them for every invocation.
+- **The final checkpoint is written after the port closes.** Deleting `var/checkpoints`
+  as soon as the port is free resurrects old samples on the next start, and earlier
+  timestamps are then rejected with `cannot write value to buffer from past`. `stop` and
+  `reset-store` wait for the process itself.
+- **Samples older than a buffer's start are dropped.** Ingest each series in time order,
+  and use `reset-store` before backfilling a window that starts earlier.
+- **A running job's query window ends at "now".** If a minute boundary passes after the
+  last ingested sample, the series gains a trailing `null` for the open minute. That is
+  correct, not data loss; `smoke` tolerates trailing nulls but not gaps.
+- **Fresh DB needs `-migrate-db` before `-init-db`**; `setup` does both.
+
+## Troubleshooting
+
+- **`unsupported database version 0, need 13`**: the DB was never migrated. Run `setup`
+  again (it migrates first).
+- **Pipeline hangs after `start`** (e.g. `driver.sh start | tail`): a background process
+  is holding the pipe. The driver redirects the server's stdio; if you launch the binary
+  yourself, use `( exec ./cc-backend ... ) > log 2>&1 < /dev/null &`.
+- **macOS `seq` prints epoch seconds as `1.79084e+09`**: every line gets the same broken
+  timestamp and only one sample lands. Generate timestamps with `python3` (as `smoke` does).
+- **`refusing to delete it`** from `setup`: `CC_RUN_DIR` points at a directory the driver
+  did not create. Pick another path.