Skip to content

docs(env): env-file operations and encrypted .env values - #676

Open
kriszyp wants to merge 4 commits into
mainfrom
kris/env-secrets
Open

kriszyp wants to merge 4 commits into
mainfrom
kris/env-secrets

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Documents the component .env surface of Harper's secret handling, which had no page anywhere in the docs. The secrets store itself was already covered by reference/security/secrets.md and the Secrets section of the operations reference; this fills in the file-based half that shares its enc:v1: envelope.

What is documented

reference/operations-api/operations.md — new ### Environment File Operations section (in Components, before Deployment Operations), plus three rows in the Components summary table.

  • get_env_keys, set_env_value, delete_env_value: request and response shapes, project + optional file (defaults to .env, must be .env or .env.<suffix>), the [A-Za-z0-9_.-]+ key character set, and the fact that the two writers replicate.
  • set_env_value's key+value vs values forms, in-place preservation of comments/formatting/untouched keys, file creation, and the one unrepresentable quoting combination.
  • A note covering the masking behavior that was also undocumented: get_component_file on a protected .env returns protected: true, keys, and a KEY=******** body; get_components flags such files; .env.example / .sample / .template are not masked; set_component_file is not blocked and writes verbatim.

reference/environment-variables/encrypted-values.md — new page (sidebar entry added under Environment Variables).

  • The enc:v1: marker, per-value opt-in, coexistence with plaintext.
  • The Harper Pro custody requirement, and the four load-time outcomes (decrypt / skip-and-defer with no custody / replay when custody registers late / skip on decrypt failure), including that a skipped value is absent rather than set to ciphertext and that it never crashes the node.
  • The fetch-encrypt-write client flow, with the get_secrets_public_key response shape.
  • The threat model, and a note that set_env_value does not validate the envelope the way set_secret does.

reference/environment-variables/overview.md — a short "Managing .env Files Remotely" section pointing at both, plus Related links.

reference/security/secrets.md — one line: its existing mention of encrypted .env values now links to the new page.

Envelope internals and the reference Node.js client stay in secrets.md and are linked rather than duplicated.

Verified against code

All behavior was read from HarperFast/harper origin/main (the working checkout's relevant files are byte-identical to origin/main — confirmed by diff), not from the prior internal write-up:

  • Operation handlers and response shapes: components/operations.js (getEnvKeys, setEnvValue, deleteEnvValue, getComponentFile), components/operationsValidation.js (getEnvKeysValidator, setEnvValueValidator, deleteEnvValueValidator).
  • .env parsing, masking, quoting rules: utility/envFile.ts (isEnvFile, isProtectedEnvFile, isExampleEnvFile, formatEnvValue, upsertEnvValues, removeEnvKeys, ENV_KEY_REGEX, ENV_ENCRYPTED_PREFIX).
  • Load-time decrypt / skip / defer: resources/loadEnv.ts, resources/secretDecryptor.ts.
  • Envelope format and kid semantics: utility/secretEnvelope.ts.
  • get_secrets_public_key response is { public_key, fingerprint }: components/secretOperations.ts.
  • Pro custody (cluster-shared RSA-4096 keypair, leader generates on first boot, clone fetches it): harper-pro security/keyCustody.ts, security/fileKeyCustody.ts.
  • Response shapes and masking cross-checked against unitTests/components/envOperations.test.js.

Version badge. v5.2.0 on both new sections. Verified by feature presence at tags rather than ancestry: the three operation names in utility/hdbTerms.ts, components/secretOperations.ts, utility/envFile.ts, isProtectedEnvFile in components/operations.js, and isEncryptedEnvValue + deferEncryptedEnvValue in resources/loadEnv.ts are all present at v5.2.0 and all absent at v5.1.28.

Corrections to the prior internal write-up (docs/env-secret-encryption.md, commit 56511db95), which this PR does not reuse verbatim:

  • It documents get_secrets_public_key as returning { publicKey, fingerprint, scheme, algorithm }. The code returns { public_key, fingerprint } — snake_case, and no scheme or algorithm field. The new page uses the real shape.
  • Its claim that the keypair is "distributed the same way the JWT keypair is" is not stated here; the new page says custody is a Harper Pro component without describing the distribution mechanism, since the docs already cover custody at that level in secrets.md.

For the human reviewer

Review round on 2026-10-01 (Ethan's requested changes plus the pre-push review). What changed and what was left alone:

  • When an edit takes effect. An edit changes the file, not the running components: an edit to a file loadEnv has already loaded only sets restartRequired (resources/loadEnv.ts → requestRestart). Running workers keep the old values, including a deleted key, until they restart. restart_service with "replicated": true is the documented way to apply an edit. Only harper dev running an application directory restarts on its own (autoReload is set only for RUN_HDB_APP). The get_status restartRequired row now mentions .env edits.
  • Replication outcome. replicateOperation (harper-pro replication/replicator.ts) sends to each peer once and does not retry. Each peer appears in the response's replicated array, and a peer that could not be reached or answered with an error is "status": "failed". This replaces the earlier "an edit reaches every node", and settles the replication-response question the first draft listed as unverified. The restart-semantics question from that list is settled by the bullet above.
  • Encrypted values. A skipped value is never written to process.env: a skipped value leaves an already-set variable in place. A decrypted value, live or replayed, follows the plaintext override rule. A replay does not rerun code that already read the variable.
  • Carriage returns. A value containing \r is accepted but reads back with \r\n or \r turned into \n (dotenv normalizes line endings before it parses). The page now says so.

Declined here as core defects. All four are reproduced or traced in HarperFast/harper, and the fix belongs in core, not in a docs caveat. Deciding whether to add a temporary caveat until core is fixed is yours:

  1. utility/envFile.ts closesQuote/valueEndLine disagree with dotenv about where a quoted value ends. Given WIN_PATH="C:\tools\" followed by API_KEY=abc, or an unterminated A="foo, set_env_value on that key deletes the following keys on every node. Reproduced with the production helper. This contradicts the page's "every other line … is left exactly as it was".
  2. ASSIGNMENT_LINE matches only KEY=value, but dotenv also accepts KEY: value. On such a line, delete_env_value reports success and leaves the key, and set_env_value appends a duplicate. Reproduced.
  3. set_env_value and delete_env_value read, modify, and write the file with no lock and no ordering across nodes. Overlapping edits can drop a key or leave nodes with different files.
  4. The deferred replay in resources/secretDecryptor.ts skips loadEnv's config-shaping guard. An encrypted HARPER_SET_CONFIG loaded before custody registers ends up in process.env.

Kept as written: set_component_file stays documented as an unguarded verbatim write to .env, and set_env_value as not validating enc:v1: envelopes. Both are accurate descriptions of current core behavior. Whether core should block either one is a product call.

Merge conflict. This branch conflicts with main in reference/operations-api/operations.md, and the conflict is positional only. main added #### Checking what each node installed at the end of the deploy section, at the same spot where this PR inserts ### Environment File Operations. The resolution is to keep main's subsection, then the new section. It was not rebased from this run, because another checkout holds the branch.

Verification

  • npm run format:check — clean.
  • npm run build — succeeds, no broken-link errors (onBrokenLinks: 'throw').
  • Both cross-page anchors confirmed against the built HTML: id="environment-file-operations" in build/reference/v5/operations-api/operations.html and id="reference-client-nodejs" in build/reference/v5/security/secrets.html.
  • New page renders at build/reference/v5/environment-variables/encrypted-values.html and appears in the sidebar.

Uncertain / not verified

  • enc:v1: value size. set_secret caps value/envelope at 256 KB; set_env_value has no such cap in its validator. I did not document a limit for env values, but I also did not test a large one against the file writer.
  • Studio coverage. secrets.md says Studio provides a UI for the secrets store. I did not claim anything about Studio's .env editing surface, having no way to check it here.
  • Release notes. release-notes/v5-lincoln/5.2.md does not mention the env operations or .env masking. I left it alone — adding a retroactive entry felt out of scope for this PR, but it may be worth a follow-up.

— Claude Opus 5.5 (review round)

🤖 Generated with Claude Code

Review-Coverage: authored=claude; ran=gemini,cursor-composer,codex,cursor-muse; adjudicated=domain; declined=cursor-grok,cursor-kimi; rounds=4; full=1 @ 9110a07

Human-Review-Need: 3 (decisions: docs-caveat-for-known-writer-defects, document-set-component-file-unmasked-overwrite) @ 9110a07

The three operations that edit a component's `.env` file - `get_env_keys`,
`set_env_value`, `delete_env_value` - had no page anywhere in the docs, and
neither did the `enc:v1:` encrypted-value form that `loadEnv` decrypts at
startup. The secrets store was already documented; this fills in the file-based
half that shares its envelope format.

Adds an Environment File Operations section to the operations reference,
covering request and response shapes, the `.env`-only `file` constraint, the key
character set, replication, and the masking `get_component_file` and
`get_components` apply to protected `.env` files (while `set_component_file`
still writes them verbatim).

Adds a sibling reference page for encrypted values: what the `enc:v1:` marker
means, the Harper Pro custody requirement and the deferral behavior when no
decryptor is registered yet, the fetch-encrypt-write client flow, and the threat
model. Envelope internals and the reference client stay in the secrets page and
are linked rather than duplicated.

All of this shipped in v5.2.0, verified by the presence of the operations and
the loader path at the v5.2.0 tag and their absence at v5.1.28.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for Encrypted Environment Values and remote .env file management operations (v5.2.0), including new Operations API endpoints such as get_env_keys, set_env_value, and delete_env_value. The feedback suggests improving readability in the Operations API documentation by presenting critical fallback behaviors in separate, distinct sentences rather than combining them with semicolons.

Comment thread reference/operations-api/operations.md Outdated
@github-actions
github-actions Bot temporarily deployed to pr-676 September 22, 2026 00:18 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-676

This preview will update automatically when you push new commits.

@kriszyp
kriszyp marked this pull request as ready for review September 29, 2026 13:27
@kriszyp
kriszyp requested a review from a team as a code owner September 29, 2026 13:27

@Ethan-Arrowood Ethan-Arrowood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Reviewed with Codex

Comment thread reference/operations-api/operations.md
Comment thread reference/environment-variables/encrypted-values.md Outdated
Comment thread reference/environment-variables/encrypted-values.md Outdated
kriszyp and others added 3 commits October 1, 2026 11:42
… leaves behind

An edit through set_env_value or delete_env_value changes the file only:
loadEnv flags restartRequired on a change to a loaded file and the running
workers keep the old values until they restart. A skipped encrypted value
leaves an already-set variable in place, a decrypted one follows the same
override rule as plaintext, and a late decryptor's replay does not rerun
code that already read the variable.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a carriage return reads back as a newline

The replicated response lists each peer, and an unreachable one is marked
failed and not retried. dotenv normalizes \r\n and \r to \n, so a value
with a carriage return does not round-trip. The harper dev auto-restart
applies only when it runs an application directory.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… listed as failed too

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-676

This preview will update automatically when you push new commits.

@Ethan-Arrowood Ethan-Arrowood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - fix the merge conflict however you best see fit and you're welcome to squash merge without another review (I trust you wont dramatically change this PR when dealing with the conflict). Thanks!

🤖 Reviewed with Codex

This branch was successfully deployed

1 active deployment
pr-676 — 9110a070 Deployed Oct 1, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants