Skip to content

fix(docker): stop released images reporting dirty builds - #333

Merged
tarakanof merged 2 commits into
mainfrom
fix/dirty-build
Oct 8, 2026
Merged

tarakanof merged 2 commits into
mainfrom
fix/dirty-build

Conversation

@tarakanof

Copy link
Copy Markdown
Owner

Closes #332

What was dirty and why

Every released image reported "dirty": true at /version (e.g. v0.46.4 at c03f29f). The binary was fine; the build context wasn't the checkout.

  • .dockerignore excluded the whole .claude directory.
  • Three files in it are tracked: the project Claude settings and the awtrix-berry-app skill (SKILL.md, references/awtrix-api.md).
  • The Dockerfile copies .git into the build stage and builds with -buildvcs=true. Go runs git status --porcelain, sees those three files as deleted, and stamps vcs.modified=true.
  • version (ldflags) and revision were correct; only dirty was wrong.

Fix

  • .dockerignore now excludes only the untracked parts of that directory (worktrees, settings.local.json), so every tracked file reaches the context. A real edit in the context still stamps dirty.
  • Guard: docker-publish.yml builds an amd64 image (loaded, from the gha cache) before the multi-arch push, and scripts/check-image-vcs.sh refuses to publish unless ember version shows the tagged $GITHUB_SHA without +dirty. It also fails on an unstamped binary (no .git in the context).
  • CI: a new image-vcs job runs the same check on every PR, so a future .dockerignore change or new tracked file can't regress silently.
  • scripts/image-smoke.sh fails on +dirty when the host checkout is clean.

Also checked, not affected:

  • release-producers.yml: builds from a clean tag checkout. Output goes to dist/, which is gitignored. A local run of package-producers.sh leaves git status empty, and the producer has vcs.modified=false. Producers don't expose VCS state anyway.
  • macOS app: its version comes from MARKETING_VERSION/CURRENT_PROJECT_VERSION in project.yml, not from git. The xcodegen output (*.xcodeproj) is gitignored, and there's no dirty concept to fix.

Evidence

Docker daemon not available locally, so reproduced with go build in a copy of the checkout filtered by .dockerignore (rsync with the same root-anchored excludes):

# c03f29f, old .dockerignore
 D .claude/settings.json
 D .claude/skills/awtrix-berry-app/SKILL.md
 D .claude/skills/awtrix-berry-app/references/awtrix-api.md
vcs.modified=true
ember dev (c03f29f41847e979988d20291339a976152ee7cf+dirty, go1.27.1)

# this branch, with untracked settings.local.json, worktrees/ and build.log present in the source
git status --porcelain   -> (empty)
vcs.modified=false
ember dev (700969192a53289e04b9f710d193dc317ec04a54, go1.27.1)

# same context, one tracked file edited
ember dev (700969192a53289e04b9f710d193dc317ec04a54+dirty, go1.27.1)

The new image-vcs CI job on this PR checks the real Docker build end to end.

Checks: gofmt, go vet ./..., go test ./... -race pass. hadolint (Dockerfile unchanged; only pre-existing DL3018/DL3059/DL3066 findings) and yamllint -d relaxed on both workflows (only pre-existing line-length warnings) report nothing new. shellcheck is clean on both scripts.

.dockerignore excluded the whole .claude directory, but .claude/settings.json
and the awtrix-berry-app skill are tracked. The build context therefore lacked
three tracked files, `git status` inside the build stage saw them as deleted,
and Go stamped vcs.modified=true, so /version and `ember version` reported
dirty for every release (v0.46.4 at c03f29f included).

Exclude only the untracked parts (.claude/worktrees, settings.local.json) so
every tracked file reaches the context. A real local edit still stamps dirty.

Guard: docker-publish builds an amd64 image first and refuses to push unless
`ember version` reports the tagged commit without +dirty
(scripts/check-image-vcs.sh); CI runs the same check on every PR, and
image-smoke.sh fails on +dirty from a clean checkout.
@tarakanof

Copy link
Copy Markdown
Owner Author

Opus review

No blocking findings. The root cause is real and the fix is complete for today's tree. Findings below, most severe first.

Low

  1. Docs not updated. docs/RUNBOOK.md §CI (around line 318) lists the CI jobs and doesn't mention image-vcs. The publish section (around line 1217) doesn't mention the pre-push VCS guard. docs/ARCHITECTURE.md gotchas (around line 3569) already has the worktree/buildvcs bullet, so it's the natural place for a sibling: ".dockerignore must never exclude a tracked file, or every image is stamped +dirty; scripts/check-image-vcs.sh enforces this in CI and before publish."
  2. cache-from: type=gha in the new image-vcs job never hits. The PR run's log has importing cache manifest, then zero CACHED steps, and go build takes 26s cold. PR runs can only restore caches from their own ref or the default branch. The only cache-to is in docker-publish.yml, which runs on release tags, and nothing writes the cache on main. Either drop the line or accept the cold build; 53s on a parallel job is fine. The publish job's cache-from has the same problem (pre-existing). The new guard there still costs only seconds, because the job-local Buildx builder reuses the amd64 layers in the following multi-arch Build and push (same context and VERSION build arg).

Nit

  1. .claude/settings.local.json is ignored only by a global git excludes file (**/.claude/settings.local.json in the author's ~/.config/git/ignore), not by the repo .gitignore. A contributor without that global rule sees ?? .claude/settings.local.json in git status, so the new +dirty check in image-smoke.sh silently skips. Add it to .gitignore next to .claude/worktrees/. Related: the container has no global excludes. Any untracked file that the host ignores only globally and .dockerignore doesn't exclude stays in the context and makes a local image +dirty on a "clean" host. That's correct to flag, but the message could mention it.

Verified, no issues

  • Root cause. git ls-files .claude lists exactly the three files from the PR body. None of the other .dockerignore patterns (.gocache, *.log, *.test, *.out, config.json, config.local.json, producer.env; root-anchored, and checked at any depth too) matches a tracked file. The repo has no submodules, symlinks or LFS/.gitattributes filters that could differ inside the container.
  • Build context contents. It adds 77 KB: settings.json (908 B, only permissions.deny for mcp__unraid__*), the skill SKILL.md and references/awtrix-api.md. No tokens, credentials, hostnames or IPs; only placeholders like awtrixng-xxxxxx.local and curl -u user:pass. All of it is in the build stage only; the final distroless image copies just /out/ember and the data dir.
  • Workflow. The guard runs before Build and push in the same job, and a non-zero exit stops the push. Multi-arch is unaffected: arm64 is cross-compiled from the same context and .git, so its stamp is the same. permissions: contents: read is enough, and nothing new is needed. Action refs follow the repo convention: actions/* and docker/* on major tags, only third-party actions SHA-pinned, and the versions match the existing steps. GITHUB_SHA equals checkout HEAD for release, workflow_dispatch and pull_request (the merge ref), so the ($SHA, match holds.
  • CI cost. The job took 53s on this PR, parallel with the 2m41s Go job, so it doesn't lengthen the wall clock. Running it on every PR without a path filter is right, since any new tracked file can trip it.
  • Scripts. shellcheck is clean on both. check-image-vcs.sh: set -euo pipefail, the usage check exits 2, a docker run failure aborts through set -e, $SHA is quoted in the case pattern, and an unstamped (unknown) binary fails the revision match. The PR run printed ember dev (f5411f4…, go1.26.8) and vcs: clean build.
  • Lint. actionlint: clean. yamllint -d relaxed: only line-length warnings on lines this PR doesn't touch. hadolint: only DL3018/DL3059/DL3066, all pre-existing (the Dockerfile is unchanged).

Review of #333: RUNBOOK lists the image-vcs CI job and the pre-push check in docker-publish; ARCHITECTURE gotchas say .dockerignore must never exclude a tracked file. Drop cache-from gha in image-vcs: PR runs can't restore a cache only release tags write, so it never hit. Ignore .claude/settings.local.json in the repo .gitignore so contributors without a global rule keep a clean status and image-smoke's dirty check still runs.
@tarakanof

Copy link
Copy Markdown
Owner Author

Review fixes in b21275a:

  1. Docs not updated: fixed. RUNBOOK §CI now lists the image-vcs job, and §Docker Hub release describes the pre-push check-image-vcs.sh guard. ARCHITECTURE gotchas has a new bullet next to the worktree/buildvcs one: .dockerignore must never exclude a tracked file.
  2. cache-from: type=gha in image-vcs never hits: fixed, the line is dropped from that job. The publish job's cache settings are unchanged; that problem predates this PR.
  3. .claude/settings.local.json only globally ignored: fixed, it's now in the repo .gitignore next to .claude/worktrees/.

actionlint is clean on both workflows. yamllint (relaxed) gives only line-length warnings on lines this PR doesn't touch. shellcheck is clean on both scripts.

@tarakanof
tarakanof merged commit 23dedec into main Oct 8, 2026
7 checks passed
@tarakanof
tarakanof deleted the fix/dirty-build branch October 8, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Released Docker image reports dirty: true at /version

1 participant