fix(thought-enrichment): three silent failures — metadata corruption,… - #462
Open
chriscfellows wants to merge 1 commit into
Open
Conversation
… env loading, uuid cursor
Each of these fails without surfacing an actionable error: the script prints
success while corrupting data, writing nothing, or scanning zero rows. Found
running the recipe against a stock install (Supabase, uuid `thoughts.id`, 2,448
rows).
1. patchThought() double-encoded the `metadata` jsonb column
`body.metadata` was passed through JSON.stringify() before the whole body was
serialized again, so Postgres received a JSON string and stored a jsonb scalar
string rather than an object.
Every enriched row's metadata became unqueryable:
metadata->>'source' -> NULL
metadata ? 'topics' -> false
jsonb_object_keys(metadata) -> ERROR: cannot call jsonb_object_keys on a scalar
2,439 of 2,448 rows were affected; the only intact rows were ones whose
enrichment had failed. The run still printed ENRICHMENT COMPLETE. Downstream it
presented as "this brain has 8 gmail-sourced thoughts" when the real count was
1,817 — the data was present but unreadable.
2. parseEnvFile() read only .env.local, ignoring process.env
Any runner that injects credentials as environment variables (CI, systemd, a
wrapper that fetches the service-role key at runtime instead of persisting it to
disk) received empty strings. With supabaseUrl == "" the request URL is
relative, so the failure was:
TypeError: Failed to parse URL from /rest/v1/thoughts?select=id&enriched=eq.true
That names a PostgREST path and never mentions env loading, which points
debugging in the wrong direction. Now layers .env.local over process.env; file
entries still take precedence, so existing setups are unaffected.
3. backfill-sensitivity.mjs seeded its uuid cursor with the integer 0
The first request was `?id=gt.0`, which PostgREST rejects when `thoughts.id` is
a uuid (the default):
400 {"code":"22P02","message":"invalid input syntax for type uuid: \"0\""}
The scanner then reported a false all-clear — "Scanned: 0, Errors: 1" — which
reads as "nothing to do". This script's job is to tier sensitive content before
a synthesis or digest step ships it to an LLM, so a silent no-op is the worst
available failure mode.
After the fix, on the same brain: 2,215 scanned, 18 -> personal, 5 ->
restricted. The restricted rows held an API key, an SSN pattern, and three
passport patterns, all previously eligible for outbound synthesis.
Also adds a Troubleshooting section to the recipe README covering all three
symptoms, including recovery SQL for installs whose metadata is already
double-encoded (no data is lost — the original object survives inside the string).
Bumps metadata.json version to 1.0.1.
Verified: node --check clean on both files; --help still runs; a subsequent
enrichment run keeps jsonb_typeof(metadata) = 'object' across all 2,448 rows;
backfill-sensitivity.mjs --dry-run paginates the full table; metadata.json parses.
|
Hey @chriscfellows — welcome to Open Brain Source! 👋 Thanks for submitting your first PR. The automated review will run shortly and check things like metadata, folder structure, and README completeness. If anything needs fixing, the review comment will tell you exactly what. Once the automated checks pass, a human admin will review for quality and clarity. Expect a response within a few days. If you have questions, check out CONTRIBUTING.md or open an issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution Type
/recipes)/schemas)/dashboards)/integrations)/skills)What does this do?
Fixes three bugs in
recipes/thought-enrichmentthat all fail silently — thescripts print success while corrupting data, writing nothing, or scanning zero
rows. One of them (
patchThought()double-encoding themetadatajsonb column)corrupts metadata on every enriched row of a stock install, so anyone who has
run this recipe is affected and has no reason to suspect it. Also adds a
Troubleshooting section to the recipe README with recovery SQL, and bumps
metadata.jsonto 1.0.1.Requirements
No new requirements. Unchanged from the recipe's existing prerequisites:
Node.js 18+, Supabase, and an OpenRouter or Anthropic API key. No new
dependencies, no schema changes, no
requires_skillsadditions.Checklist
README.mdwith prerequisites, step-by-step instructions, and expected outcomemetadata.jsonhas all required fieldsThe three bugs
1.
metadatadouble-encoded — data corruptionenrich-thoughts.mjs→patchThought()