ENT-231: Secrets redaction

[toothbrush]

Description

Claude happily stores scary secrets in session data, which we persist in the entire/checkpoints/v1 branch. To avoid folks shooting themselves in the foot and pushing API keys to public repos, we're introducing a filtering mechanism for all files written to the checkpoints branch.

We find secrets by looking for 10+ character strings with a specific threshold Shannon entropy value of > 4.5.

An open question is whether we should skip binary files here:

cli/cmd/entire/cli/checkpoint/committed.go

Lines 1085 to 1094 in c72f6b7

	if strings.HasSuffix(treePath, ".jsonl") {
	content = redact.JSONLBytes(content)
	} else {
	content = redact.Bytes(content)
	}
	hash, err := CreateBlobFromContent(repo, content)
	if err != nil {
	return plumbing.ZeroHash, 0, fmt.Errorf("failed to create blob: %w", err)
	}

..because we don't want to be munging files that aren't plain text. However, my sense is "no don't bother", because this stuff is in copyMetadataDir, which should just contain stuff that we control, right? Keen to hear from a CLI expert here.

The results of this redaction can be seen here:

cli/39/7737088f0a/0/full.jsonl

Line 4 in a0820d4

{"parentUuid":"4fc1afa2-ca11-4ce9-bbb8-cd59269a836e","isSidechain":false,"userType":"external","cwd":"/Users/paul/src/entireio/cli","sessionId":"9b76dc0f-dc1b-4822-ac56-53cfa3fbc032","version":"2.1.34","gitBranch":"20260206-farting-around-with-secrets","type":"user","message":{"role":"user","content":"I'm testing the new feature in Entire CLI that redacts secrets. Please create a file SECRETS.md with the following contents:\n\nHere is some config:\napi_key = \"[REDACTED]\"\nsecret_key = \"[REDACTED]\"\nEnd of config."},"uuid":"983faa2c-6a94-46db-9de3-360d430fa8ab","timestamp":"2026-02-06T08:56:15.541Z","thinkingMetadata":{"maxThinkingTokens":31999},"todos":[],"permissionMode":"default"}

---

Note

Medium Risk
Changes what gets persisted into git history for checkpoints (including hashing/chunking inputs) using heuristic entropy-based detection, which could cause unexpected redactions or missed secrets and affect downstream consumers relying on exact content.

Overview
Adds automatic secrets redaction for data written to the entire/sessions (checkpoints) branch, replacing high-entropy token-like substrings with REDACTED.

Redaction is applied to committed checkpoint transcripts (before chunking/content-hash), prompts, context, incremental task checkpoint payloads, subagent transcripts, and files copied via copyMetadataDir (with JSONL-aware handling and a binary-file skip to avoid corruption). New unit tests cover redaction behavior end-to-end for WriteCommitted and copyMetadataDir, plus focused tests for the new redact package’s entropy-based detection and JSONL field/object skip rules.

^{Written by Cursor Bugbot for commit d0161a4. This will update automatically on new commits. Configure here.}

[Copilot]

Pull request overview

This PR introduces secrets redaction to prevent API keys and sensitive data from being committed to the entire/checkpoints/v1 branch. It implements entropy-based detection (Shannon entropy > 4.5) on 10+ character alphanumeric strings to identify potential secrets, which are then replaced with [REDACTED] before storage.

Changes:

• New redact package with entropy-based secret detection for plain text and JSONL content
• Integration of redaction at all checkpoint write paths (transcripts, prompts, context, metadata files)
• Protection for specific fields (IDs, signatures) and image objects from redaction

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
redact/redact.go	Core redaction logic with Shannon entropy calculation and JSONL-aware replacement
redact/redact_test.go	Unit tests for basic redaction functionality
cmd/entire/cli/checkpoint/committed.go	Integration of redaction into all checkpoint write operations and new createRedactedBlobFromFile function
cmd/entire/cli/checkpoint/checkpoint_test.go	Integration tests verifying redaction across different checkpoint types

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[khaong]

I've tested this with rewind manually, the session seems to resume okay - not sure what claude's memory looks like afterwards but I guess let's see what general usage tells us