Skip to content

The reaper Makefile's computed VERSION never reaches CloudFormation, so deploys stamped 'unknown' #869

Closed

Description

@scttfrdmn

Found while running the production reaper deploy (2026-10-12).

lambda/ttl-reaper/Makefile appears to stamp the deployed version automatically:

VERSION?=$(shell git describe --tags --always --dirty 2>/dev/null | sed 's/^v//' || echo unknown)

It does not. Parameters are only sent when EMIT_EXPLICIT emits them, and that filters on origin:

define EMIT_EXPLICIT
$(if $(filter command line environment,$(origin $(2))),printf '%s\n' '$(1)=$($(2))' >> .deploy-explicit.txt;,)
endef

A ?= assignment in the Makefile has origin file, not command line or environment, so the computed value is silently dropped and the parameter falls back to the template default. For Version that default is unknown.

Confirmed two ways. Statically, by $(origin):

VERSION origin (no override):  file
VERSION origin (VERSION=1.2.3): command line

And empirically: the spawn-ttl-reaper-production stack — created 2026-06-08, last updated 2026-10-08 — had no Version parameter at all until today's deploy passed VERSION= explicitly. Version and LogRetentionDays were the only two parameters that changed; the other 15 were already set.

Why it matters

#768's stated purpose is that the deployed bytes can be traced to a commit. That is defeated for every deploy which did not pass VERSION= by hand — which, judging by the live stack, is all of them until now. The failure is silent and the artifact reads as authoritative: an operator running aws cloudformation describe-stacks sees a Version of unknown, or in this case no Version key, with nothing to suggest the Makefile had computed the right answer and discarded it.

LOG_RETENTION_DAYS?=30 has the same defect. It is harmless today only because 30 happens to equal the template default — change the Makefile default and it would silently not apply.

⚠️ The obvious fix would disarm the production reaper

Do not "fix" this by emitting every parameter regardless of origin. DRY_RUN?=true is also file origin, and for that parameter the current behaviour is load-bearing and correct: not emitting it is exactly what lets the merge preserve the live DryRun=false. Emitting file-origin defaults would push DryRun=true onto the stack and disarm the live reaper — the precise #650 failure, which the Makefile's own comments say was already reintroduced once by a fix.

So the mechanism is not simply broken. It is right for parameters whose live value must win, and wrong for parameters the repo should assert on every deploy. Those are different classes and need distinguishing, e.g. an explicit list of "always assert" parameters (Version, LogRetentionDays) emitted unconditionally, with everything else keeping today's origin-gated behaviour.

Why no gate caught it

TestEveryTemplateParameterIsInThePARAMMAP passes: Version:VERSION is in PARAM_MAP. The gate checks that a mapping exists, which is a correlate of "this parameter is settable" — not of "this parameter is actually sent". Being mappable and being sent come apart exactly here.

This is adjacent to the defect already recorded in the audit plan, that the same gate "matches the CFN key but never the make-variable, so DryRun:DRYRUN passes while make deploy DRY_RUN=true is silently ignored". Same gate, same blind spot, different half.

A gate with teeth would assert on the merged parameter file rather than on PARAM_MAP: run the merge with no explicit overrides and require that Version is not the template default. That fails today and passes after the fix.

  • Distinguish always-assert parameters from live-wins parameters, without emitting DRY_RUN.
  • Gate on the merged output, not on PARAM_MAP membership.
  • Check the other deploy scripts for the same shape — scripts/deploy-scheduler-handler.sh and scripts/deploy-cost-history-collector.sh compute SPAWN_VERSION with git describe and pass it to aws lambda tag-resource directly, so they are probably fine, but that is unverified.

Live state after today's deploy, for reference: Version=0.127.0-34-g5610712, LogRetentionDays=30, and DryRun=false / NetResources=reap preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions