Anonify

AI-assisted document redaction. Upload a PDF, Word document, spreadsheet, deck, email, CSV, text file or image; the system proposes what looks sensitive; you decide; the export removes it.

The governing rule: AI proposes, the application applies, and only what a person accepted is removed. A beautiful editor that leaves the original text under a black rectangle is not a redaction system.

Upload, review what was proposed, export a file the value is actually gone from.

Run with Docker

No clone, no Node and no build: the image is published to Docker Hub as nabeelwasif/anonify2.0 for linux/amd64 and linux/arm64, so it runs natively on Apple silicon and ARM servers too. You need Docker with Compose v2.

mkdir anonify && cd anonify
curl -LO https://github.com/Anonify-v2-0/Anonify2.0/releases/latest/download/docker-compose.yml
[ -e .env ] || docker run --rm nabeelwasif/anonify2.0 keys > .env
docker compose up -d

Then open http://localhost:3000. That starts Postgres and RustFS (S3-compatible storage), creates the bucket, applies the migrations and starts Anonify, with local OCR, no accounts anywhere, and every port bound to localhost.

Keep .env. anonify keys prints a new ENCRYPTION_KEY every time, and it encrypts every stored document: a new key cannot read what the old one wrote. That is why the command above only writes .env when there is none.

Choose a version. The file follows latest, which moves with every release. For anything you keep, pin one in .env:

ANONIFY_VERSION=1.17.0   # exactly this release; 1.17 takes its patches; 1 every minor release

Upgrading is a pull. The migrate service runs the new image's migrations before the app starts, so there is nothing else to run:

docker compose pull && docker compose up -d

Read the changelog before a minor or major version. Every setting is in .env.example, also attached to each release as env.example; add what you need to .env and run docker compose up -d again. For other platforms, replicas and production settings see docs/deploy/image.md.

Documentation

Document	What it covers
docs/architecture.md	The four layers, request paths, identity, encryption, deliberate limits
docs/workflow.md	The durable pipeline, its steps, streaming, expiry, failure modes
docs/ai-engine.md	Detection order, cost discipline, prompts, suggestion → decision → removal
docs/pipelines.md	Why each format's pipeline is built the way it is
docs/presets.md	The five shipped redaction presets and their detector/category memberships
docs/deploy/README.md	Where to run it: the contract every platform relies on, and a recipe per platform
docs/deploy/vercel.md	Deploying to Vercel on the Hobby plan, and the limits that shape it
docs/operations.md	Metrics, autoscaling workers on the queue, traces and alerts
docs/deploy/capacity.md	What one replica takes on, and how to size it
CONTRIBUTING.md	The invariants, the roadmap, and the benchmarks we would like

What it can redact

Ten formats today — the register that defines them lives in one place (lib/documents/formats.ts), and the table below mirrors it. Each format's pipeline is argued through in docs/pipelines.md.

Format	What gets removed	How it is removed
PDF	Text on any page	Redacted pages rasterized with boxes burned in; untouched pages keep selectable text. Scanned pages OCR'd so they are reviewable.
Word (.docx)	Text nodes, in place	Byte-identical in-place edit — styles, numbering and relationships survive. Headers, footers, footnotes and comments swept, not just the body.
Excel (.xlsx)	Cells	Cells rewritten; any formula still referencing a redacted address is dropped, because a cached result is a second copy. Hidden sheets read.
PowerPoint (.pptx)	Slides, speaker notes, layouts, master	The same OOXML surgery across all four. Three of those four are never on screen, and a deck that redacts only its slides ships the other three.
Email (.eml)	Headers, every body, HTML, quoted replies, attachment filenames, nested messages	Byte-range replacement in the original so untouched parts come out identical. Attachments in supported formats become child documents of their own.
CSV	Cells	Parsed into a grid and rewritten cell by cell, so a value next to a comma inside a quoted field cannot shift every row after it.
TSV	Cells	Same grid model as CSV.
Plain text (.txt)	Offsets	Addressed by offset into the source.
Rich text (.rtf)	Parsed text	RTF is parsed rather than searched, because a word processor splits a value across formatting groups and the string is often not in the file at all.
Mailbox (.mbox)	Nothing — it is a container	Split into one document per message, each reviewed and exported on its own. A decision taken on the first message is carried to the nine hundredth. The mailbox is never redacted as a file: it comes back as a mailbox, rebuilt from its messages' verified exports and verified again as a whole — or as the processed files, in folders that mirror where each came from.
Images (PNG, JPEG, WebP)	Pixels	Pixels replaced and the file re-encoded. EXIF and GPS go too.

Every export is then re-opened and read the way an adversary would. A surviving value fails the export rather than shipping (lib/redaction/validation.ts).

What it looks for

People's names, addresses, phone numbers, email addresses, government IDs, bank accounts, financial details, dates of birth, customer and case numbers, confidential business information, credentials, identifying URLs, faces, a person's health, and anything else that identifies someone. Each category is defined in one place, lib/redaction/categories.ts, with what it is not: an amount on an invoice is not a financial detail unless it is what a named person earns, owes or holds, and a date is not a date of birth because a name is nearby. The model's instructions and the benchmark corpus read the same definitions (#197).

Health has its own category. A diagnosis, a medication or a blood pressure reading about a named person is suggested as health, and it can only be masked: a stand-in for a diagnosis still says something about the person. Before this, the same details were suggested as confidential, which is meant for codenames and trade secrets.

How it works

Four layers, kept deliberately separate:

Layer	What it is	Where
A. Source	The uploaded file. Never mutated.	Blob storage, AES-256-GCM sealed
B. Normalized	Pages, spans with geometry, runs, sheets, regions	lib/documents/*, types/document.ts
C. Redactions	The record of what should be removed	types/redaction.ts, lib/redaction/*
D. Output	A new document generated from A + accepted C	lib/redaction/export.ts

flowchart TD
    U["Upload<br/>browser → blob storage"]
    I["Ingest<br/>sniff, checksum, seal, delete the plaintext"]
    E["Extract<br/>pages, spans, geometry"]
    N["Normalize<br/>encrypted model artifact"]
    D["Detect<br/>regex first, the model for context only"]
    R["Review<br/>accept · reject · manual · global rules"]
    X["Export<br/>remove, verify, checksum, signed download"]
    P["Report<br/>counts, styles and both checksums"]
    U --> I --> E --> N --> D --> R --> X --> P

Loading

Redaction is removal, not concealment

• PDF — a page with accepted redactions is rendered to pixels with the boxes burned in and rebuilt from that raster. There is no way to paint over text in a PDF and have it be gone. Pages without redactions are copied through and keep their selectable text. Scanned pages are read with OCR so they are reviewable like any other.
• DOCX — the text nodes that carry the characters are edited in place, so styles, numbering and relationships survive byte-identical. Headers, footers, footnotes and comments are extracted and reviewable, not merely swept.
• XLSX — cells are rewritten, and any formula still referencing a redacted address is dropped, because a cached result is a second copy of the value.
• PPTX — the same OOXML surgery, across the slides, the speaker notes, the layouts and the master. Three of those four are never on screen, and a deck that redacts only its slides ships the other three.
• EML — a message is a tree, not a body with a header on it. Every header, every text part, the visible text of every HTML part, quoted replies, attachment filenames and nested messages are all reviewable, and the export replaces byte ranges in the original so untouched parts come out identical. Attachments in supported formats become child documents of their own, are redacted, and are substituted back into the message.
• MBOX — a mailbox is not a document, it is hundreds of them. It expands into a batch: one document per message, each indistinguishable from the same .eml uploaded off a desktop, and the decisions a reviewer makes carry across all of them. That is the point of it — "this recurring name is a colleague, not a subject" answered once and applied to nine hundred messages is work nobody would do by hand. Splitting is bounded and fails closed, and a From line in a message body does not fracture the message.
• CSV / TSV — parsed into a grid and rewritten cell by cell, so a value next to a comma inside a quoted field cannot shift every row after it.
• TXT / RTF — addressed by offsets into the source. RTF is parsed rather than searched, because a word processor splits a value across formatting groups and the string is often not in the file at all.
• Images — pixels are replaced and the file re-encoded. EXIF and GPS go too.

Each of these is argued through in docs/pipelines.md, which also carries the table of what each format's redaction model is and how each is verified.

Human review, and verification as a gate

The two additions that matter most are not in the detection layer at all.

Human review. Detection produces suggestions; only an accepted suggestion reaches the exporter. This is the difference between a tool that redacts for you and a tool you redact with — and it is what makes a wrong suggestion a non-event instead of a data loss. Nothing is removed that a person did not accept (lib/redaction/model.ts, isAccepted).

Verification as a gate. v1 was correct by construction: rebuild from text and the old text is necessarily gone. v2 preserves the original document, which means that guarantee has to be earned rather than assumed. So every export is re-opened and read the way an adversary would — extracted PDF text, every OOXML part, every sheet including hidden ones — and refused if an accepted value survived. tests/security.test.ts runs twenty-one such recovery attempts against produced artifacts, all of which must fail.

Batches

Several files at once become a batch: one upload, one review pass, and one archive at the end. The half that matters is not the upload — it is that a decision made once is not made again on the next file. "This recurring name is a colleague, not a subject" is answered in the document where it came up and carried to the others, including the ones still being analyzed when it was answered.

A batch owns decisions, not processing. Each document keeps its own run, its own quota accounting, its own failure and its own expiry, so one document failing leaves the rest exactly where they were — and the batch export delivers every document that succeeded, naming the ones it could not include and why.

Presets, and the one thing they must not say

A preset is a named set of detectors and categories — "Names and contact details", "Payment and account numbers", "Credentials and keys" — chosen at upload. It changes what is looked for and nothing else.

That sentence is the whole design. A preset called "HIPAA" that somebody applies and then believes they have a compliant document is a worse outcome than having no presets at all: it turns a tool that helps into one that misleads, on exactly the question where being misled is most expensive. So presets are named for what they search for, never for what they achieve, and the rule is enforced rather than documented — a preset whose id, label or description contains a regulation's name or a compliance claim fails validation at import (lib/redaction/presets.ts).

Presets are data (lib/redaction/presets/presets.json), so changing what one covers is a reviewable diff. A narrowed search is carried forward everywhere it matters: the caveat sits under the chooser, the editor says which preset the document was analyzed with, and the export report records it — because a short list of removals means either a clean document or a narrow search, and those are not the same thing.

The five presets that ship with the app:

Preset	What it looks for
Everything we can detect	Every detector, contextual pass unrestricted.
Names and contact details	People's names, email/phone, postal addresses, faces and signatures.
Identifiers and dates	SSNs, customer/case/patient references, dates of birth.
Payment and account numbers	Luhn-valid cards, IBANs, labelled account/sort codes, financial facts tied to a named party.
Credentials and keys	API keys/tokens, links carrying a token or reset path, text marked confidential.

The detector and category memberships for each are in docs/presets.md.

The export report

Every export produces a second artifact, downloadable beside the file: what was removed by category and count, how each removal was applied — solid removal and blur are not the same guarantee and the record says which — what the reviewer rejected or never decided, and both checksums, so a third party can tie the statement to a specific source and a specific output.

It carries counts and never content. A report that lists what was removed, verbatim, is a leak with a covering letter, so the report is verified before it is stored the same way the document is (lib/redaction/report.ts), and pnpm smoke reads the delivered bytes and fails if a redacted value appears in them.

Cost discipline

Deterministic detectors run first: email, phone, Luhn-checked cards, structurally valid SSNs, IBANs, credentials, and label-gated dates and account numbers. The model is asked only the contextual question, once per chunk. A value judged sensitive once is expanded to all its occurrences by local search — occurrence 2..n costs a string scan, not a request.

What this version makes better

	v1	v2
Formats	PDF	PDF, DOCX, XLSX, PPTX, EML, MBOX, CSV, TSV, TXT, RTF, images
Fidelity	Document rebuilt as plain text	Format-native: DOCX edited in place byte-identically, XLSX cells rewritten, only redacted PDF pages rasterized
Who decides	High / Med / Low, applied globally	Every suggestion accepted or rejected individually; nothing is removed that a person did not accept
Manual control	None	Click a word, drag a region, redact a cell/row/column, apply a rule everywhere
Detection	Fixed NER labels	Validated patterns + contextual model pass + local expansion
Proof	Trusted by construction	Every export re-opened and read adversarially; a surviving value fails the export
Processing	In-request, Socket.io	Durable workflow — retryable steps, resumable stream, a timeout costs a retry not the upload
At rest	Files on disk in the container	Per-document AES-256-GCM, TTL, scheduled purge of every artifact
Access	Path-based	Session ownership on every read; signed, short-lived, checksum-verified downloads

Where this came from

This is the successor to nabeel-w/Anonify — a Flask + React service that extracted PDF text with pdfminer.six, ran a spaCy NER model fine-tuned on a Kaggle PII dataset over it, masked the detected entities, and rebuilt the document with reportlab. Three redaction levels (High / Medium / Low) selected which entity labels to act on, and Socket.io streamed progress back to the browser.

It got the hardest thing right. Because it rebuilt the PDF from text rather than drawing over the original, the redacted content genuinely was not in the output file. That is more than most redaction tools manage, and this version inherits the principle rather than replacing it.

What it could not do is what drove the rewrite.

The constraints

Rebuilding the document destroyed it. reportlab wrote a fresh PDF of word-wrapped text. Layout, fonts, tables, images, columns, headers and footers did not survive. The old README listed formatting preservation as future work, which is the right diagnosis: a redaction tool people route around because it mangles the document has not solved their problem.

Detection was a fixed vocabulary. A NER model recognizes the labels it was trained on — NAME_STUDENT, EMAIL, PHONE_NUM, ID, URL_PERSONAL. Adding IBANs, API keys, or "customer reference" meant assembling labelled data and retraining model-best. And NER answers a categorical question: it tags Apple as an organization whether the document is a fruit invoice or a customer file. It cannot weigh whether a given mention is sensitive here.

Levels are a blunt instrument. High / Medium / Low is one global dial across the whole document. There was no way to keep one name and remove another, and no review step between detection and output — the model's judgement was the final judgement. Manual annotation was, again, listed as future work.

One format. PDF only; Word and spreadsheets were on the roadmap.

Weight. en_core_web_lg plus a fine-tuned model is hundreds of megabytes resident in every container instance, for a workload that is idle most of the time.

Why language models — and where they are not used

The shift is narrower than "replace NER with an LLM", and the boundary is the interesting part.

A language model is genuinely better at exactly one thing here: contextual judgement. Customer: Apple Inc. / Account manager: John Smith needs a reader that understands roles, not a tagger that emits entity types. Extending coverage becomes a prompt and a schema rather than a labelling effort.

But a language model is worse at everything a regular expression already does well. It is non-deterministic, it costs money per token, and it can produce a value that was never in the document. So detection runs deterministic detectors first, and those are strictly stronger than the NER they replace for structured data:

	v1 (spaCy NER)	v2 (deterministic first)
Card numbers	Tagged if the model learned the shape	Luhn checked — an order number is not a card
Social Security numbers	Pattern-shaped	Structurally validated — area ≠ 000/666
Account numbers	ID label, context-free	Only when labelled as an account
Dates of birth	DATE, indistinguishable from an invoice date	Only when preceded by DOB / born
API keys, IBANs	Not in the label set	Recognized by shape

The model is then asked only what the detectors cannot settle, once per chunk — and its answers are located, never trusted. It returns text it claims to have seen; the application finds that text in the source itself and discards anything it cannot locate. A model that paraphrases or invents a plausible value cannot inject it into your document.

A value judged sensitive once is expanded to its other occurrences by local string search. Seventeen mentions of a name cost one model call, not seventeen — and cannot come back with a different answer the second time.

Build from source

Two supported setups. pnpm setup asks which you want and writes a working .env; everything below is what it does, in case you would rather do it by hand.

pnpm install
pnpm setup

It asks five things: how to run it, who can reach it, which services, what the limits should be, and whether to keep the secrets already in your .env. The middle two are the ones worth slowing down for.

Who can reach it is a separate question from which services you use, because they are separate facts — a self-hosted instance on Neon is still your instance and gets your allowances. Answer "just me" and there are no daily quotas at all; answer "public and shared" and one visitor's workbook stops being everyone's budget.

Limits covers the groups that decide what this instance will accept: daily quotas, rate limits, email parser limits, how far an email's attachments are expanded into documents of their own, and how far a mailbox is. Every default it prints is read from the code that enforces it, so what you see is what is in force, and everything you leave alone is written into .env as a commented line — so the file says what the default is rather than leaving it to be discovered.

Anything that is a size is asked for and written as a size — 32MB, 512KB, 1.5GB — rather than as a byte count. Nobody types 33554432 correctly, and the way that goes wrong is not a rejected answer but a plausible number off by a factor of a thousand, quietly in force. A plain number is still read as bytes, so an existing .env keeps working.

pnpm setup --local --defaults --yes   # no questions: local, profile defaults
pnpm setup --public                   # a shared instance: strict limits
pnpm setup --help                     # every flag

Re-running it is safe. It backs up the previous .env, and it offers to reuse the secrets and keys already in it — which matters more than it sounds: documents are sealed with per-document keys wrapped by ENCRYPTION_KEY, so a new one does not reset anything, it makes everything already stored permanently unreadable.

Fully local — no accounts, nothing leaves your machine

Postgres and RustFS run in Docker, OCR runs through Tesseract, and no external service is involved at any point.

The local stack moved because the MinIO server repository was archived and made read-only in April 2026, after Community Edition binary releases had already become source-only; the mc repository was archived in July 2026 as well. RustFS is a better fit for this maintained development stack: it is actively released under Apache-2.0, exposes the same standard S3 operations the app already uses through the AWS SDK, publishes a maintained rc client for bucket setup, and has a tested one-way import path for the existing object volume. See the migration guide for compatibility limits and rollback requirements.

Prerequisites: Node 22+, pnpm 11+, Docker.

git clone <this repo> && cd Anonify2.0
pnpm install
pnpm setup --local              # or just `pnpm setup` and choose; generates the secrets
docker compose up -d --build    # Postgres, RustFS, migrations, then Anonify itself
                                # http://localhost:3000

docker compose up -d --build runs the whole application, with the app image built from this checkout and tagged as the published one. Without --build Compose runs the published image instead (Run with Docker), which is not your code. The schema is applied before the app starts, and nothing needs to be installed on the host beyond Docker itself — pnpm install and pnpm setup are there to write .env, which is where the encryption key and the fingerprint secret come from.

To develop against those services with the app on the host, start only the dependencies so port 3000 stays free:

docker compose up -d postgres rustfs rustfs-init
pnpm db:migrate         # apply the schema
pnpm ocr:warm           # optional: fetch the OCR model now rather than later
pnpm dev                # http://localhost:3000

Tesseract downloads a ~5 MB English model the first time it reads a scanned document, cached in .cache/tesseract (gitignored). pnpm ocr:warm fetches it during setup instead, so the wait lands at a moment when waiting is expected. Set TESSERACT_CACHE_PATH to a mounted volume if you run the app in a container.

What docker compose up -d starts:

Service	Port	Credentials	Purpose
app	3000	—	Anonify: the published image, or this checkout with --build
Postgres 17	5432	anonify / anonify	The database, in place of Neon
RustFS	9000 (API), 9001 (console)	anonify / anonify-dev-secret	S3-compatible storage, in place of Vercel Blob
rustfs-init	—	—	Idempotently creates the anonify bucket, then exits
migrate	—	—	The app image running anonify migrate: migrations and the workflow schema, then exits
scheduler	—	—	Deprecated, removed in 1.18.0: the app sweeps on its own. See Expiring documents

Every published port binds to 127.0.0.1 only, and the data lives in named volumes across restarts. The credentials are development defaults — do not reuse them anywhere reachable from outside your machine. If something on your machine already holds one of these ports, set APP_PORT, POSTGRES_PORT, RUSTFS_PORT or RUSTFS_CONSOLE_PORT in .env; the addresses used inside the compose network are fixed and unaffected.

The app starts only after migrate exits successfully, so a fresh up never serves against a schema that has not been applied. The container image is a Next.js standalone build on node:22-slim — Debian rather than Alpine, because sharp, @napi-rs/canvas and the pdf.js renderer all ship native binaries. The same image migrates its own database (docker run IMAGE migrate) and runs the expiry sweep (docker run IMAGE cleanup); see docs/deploy/image.md.

Durable runs need somewhere to live. On Vercel that is provided; in the container it is Postgres, through @workflow/world-postgres, and instrumentation.ts starts the worker that polls for jobs. Without that worker a self-hosted install would accept uploads and never process them.

The resulting configuration:

ANONIFY_PROFILE=self-hosted
DATABASE_URL=postgresql://anonify:anonify@localhost:5432/anonify
DATABASE_DRIVER=postgres
STORAGE_DRIVER=s3
S3_ENDPOINT=http://localhost:9000
S3_BUCKET=anonify
S3_ACCESS_KEY_ID=anonify
S3_SECRET_ACCESS_KEY=anonify-dev-secret
OCR_PROVIDER=tesseract
ENCRYPTION_KEY=...       # generated by pnpm setup
FINGERPRINT_SECRET=...   # generated by pnpm setup

RustFS keeps object data in the existing anonify_minio-data Docker volume so upgrading does not silently switch the app to an empty store. RustFS 1.0.0-rc.6 imports the unencrypted MinIO on-disk format used by Anonify in place. The move is one-way after RustFS writes new metadata, so back up the volume before the first start. Installations that enabled storage-server encryption, tiering, or custom bucket features need the API-copy path described in Migrating object storage to RustFS.

Without even Docker. Leave STORAGE_DRIVER unset and uploads go to .anonify-storage/ on disk. You still need a Postgres somewhere, because the redaction record lives in it.

AI detection is optional. With the default Gateway and no AI_GATEWAY_API_KEY, pattern detection, manual redaction, global rules and export all work; only the contextual model pass is skipped.

Demo-compatible — the same services as the hosted demo

Prerequisites: Node 22+, pnpm 11+, and accounts for the services below.

pnpm install
pnpm setup --demo       # or just `pnpm setup` and choose
# fill in the three keys it lists
pnpm db:migrate
pnpm dev

Variable	Service	Needed for
DATABASE_URL	Neon	The database. Use the pooled string.
BLOB_READ_WRITE_TOKEN	Vercel Blob	Storage. The browser uploads directly to it.
MISTRAL_API_KEY	Mistral	OCR for scanned documents.
AI_GATEWAY_API_KEY	Vercel AI Gateway	Optional — the contextual detection pass.
CRON_SECRET	your host	Gates the expiry sweep.

ANONIFY_PROFILE=demo selects the restrictive rate limits the shared demo runs with.

What is configurable, and what it defaults to

Every backend is chosen by configuration and inferred when unset, so a fresh clone works before anything is decided.

	Options	Default
DATABASE_DRIVER	postgres, neon	Inferred from the connection string
STORAGE_DRIVER	s3, azure-blob, vercel-blob, local	Inferred from what is configured
OCR_PROVIDER	tesseract, mistral	tesseract — local, no account
ANONIFY_PROFILE	self-hosted, demo	self-hosted
ANONIFY_PUBLIC_URL	The address people reach the instance at	Vercel's own on Vercel; http://localhost:3000 elsewhere
ANONIFY_ROLE	all, web (serves, runs no step), worker (runs steps, serves no one)	all; see Process roles

ANONIFY_PUBLIC_URL is used only for absolute URLs in link previews. Set it on a self-hosted instance whose links get shared. It is read when a page is served, so a published image works at any address without a rebuild; a malformed value stops the server at start.

Reads pick their driver from the stored key rather than the current setting, so changing backends does not orphan documents that are already stored.

A note on the OCR choice: Tesseract locates text per word, Mistral per block — roughly a paragraph. Both are real geometry, but a redaction over a block covers the whole block, because a proportional slice of a wrapped paragraph would be a rectangle in the wrong place. Mistral reads difficult scans better; Tesseract redacts more precisely and needs no account.

The OCR model, not just the engine

Choosing the engine and choosing the model it reads with are separate decisions, and both are configuration. Every value below is checked against a fixed list when it is read, because these end up in a compose file, where free text turns a typo into a container that builds, starts, accepts an upload and then fails on the first scanned page.

Variable	Options	Default
OCR_TESSERACT_MODEL	fast (~2 MB), standard (~3 MB), best (~13 MB)	standard — what tesseract.js ships with
OCR_TESSERACT_LANGUAGE	eng, deu, fra, spa, ita, por, nld, pol, rus, tur, ara, hin, jpn, kor, chi_sim — join with +	eng
MISTRAL_OCR_MODEL	mistral-ocr-latest, mistral-ocr-2505	mistral-ocr-latest

fast trades accuracy on poor scans for speed and size; best is the float model and is the one to reach for when a scan is genuinely difficult. Each variant caches in its own directory under TESSERACT_CACHE_PATH, so switching actually switches rather than continuing to read with the model already on disk. pnpm ocr:warm downloads whatever is configured, up front, rather than in the middle of somebody's first redaction.

Only languages present in every variant are listed, so changing the variant can never leave you without the data for the language you chose. Detection reads German, French and Spanish as well as English (#43): the document's language is told from its text, the patterns and the model's instructions are tuned for it, and the editor says which language it was read as. Other scripts are not tuned yet (#195), so OCR in those languages makes the page readable, which is the half that has to work first.

AI detection, model, and cost

The contextual pass supports official AI SDK providers, local models through Ollama, LM Studio or llama.cpp, OpenRouter, Synthetic, any other OpenAI-compatible endpoint by URL, and a ChatGPT subscription signed in with pnpm ai login --provider openai. For that last option, read the caveat in the provider docs first. pnpm setup asks for the provider and credentials, discovers models, disables known incompatible choices, and verifies structured output and image input. pnpm ai verify runs the same verification without the rest of setup. To use OpenAI, follow Connecting to OpenAI. See AI providers for the complete provider list, cloud credentials, Docker and local-server setup, and text-only analysis. Existing Gateway configurations keep working unchanged.

Variable	What it does	Default
AI_GATEWAY_API_KEY	Enables the contextual model pass. Without it, pattern detection, manual redaction, global rules and export all still work.	unset (pass skipped)
AI_PROVIDER	Selects the provider. Credentials belong to this provider.	gateway
AI_MODEL	Provider model ID (Azure deployment name).	anthropic/claude-haiku-4.5 for Gateway; required for other providers
AI_MODEL_CAPABILITIES	Setup's verified structured-output/image declaration, bound to the chosen model and destination.	Legacy Gateway behavior when unset
AI_MODEL_PRICES	Optional JSON rates keyed by usage model ID; accounts for historical models separately. Setup offers to add the chosen model's list price, with confirmation.	unset
ANONIFY_MODEL_CACHE_PATH	Where setup and pnpm models:warm cache provider model lists. Not read by the app.	.cache/models
AI_PRICE_INPUT_PER_MTOK	USD per million input tokens. Turns recorded token counts into an estimated cost in the UI.	unset
AI_PRICE_OUTPUT_PER_MTOK	USD per million output tokens, same purpose.	unset

Prices change and differ per account, so they are not hardcoded — leave AI_PRICE_* unset and the UI reports tokens and duration only, never a dollar figure. Set them and the run report shows an estimated cost alongside the token count.

External service limits

Two of the services this app depends on belong to somebody else. Mistral meters OCR requests per second; the AI Gateway meters spend — a credit balance, not a requests-per-minute number — with the model provider behind it still free to return a 429 of its own. Neither ceiling is yours to raise, and the only thing you control is how hard this instance pushes at it.

This is the opposite direction from the rate limits below. Those ration callers arriving here, and being refused is the answer. These pace this instance arriving somewhere else, where refusing is not an option — the document is already uploaded — so a request that arrives too early waits instead.

Variable	What it bounds	Default
ANONIFY_AI_CONCURRENCY	Model calls in flight, across every document	4 hosted, 1 local server
ANONIFY_AI_REQUESTS_PER_MINUTE	Sustained model-call rate; 0 paces nothing	0
ANONIFY_AI_MAX_ATTEMPTS	Tries per call before it is given up	4
ANONIFY_OCR_CONCURRENCY	Hosted-OCR requests in flight	2
ANONIFY_OCR_REQUESTS_PER_MINUTE	Sustained hosted-OCR rate	60
ANONIFY_OCR_MAX_ATTEMPTS	Tries per page before the step fails	4
ANONIFY_AI_DAILY_SPEND_USD	USD per UTC day; 0 is no cap	0

The OCR pair applies only to a hosted engine — Tesseract runs on your machine and answers to no limit. Unlike every other limit here these are not per-profile: they describe someone else's service, and an account's ceiling is the same whether a shared demo or a laptop is calling it.

Two mechanisms sit behind them, and they are not the same thing. Pacing holds a request back until the bucket can afford it, so a limit you know about is never hit. Retrying handles the one you did not: a 429, a 5xx or a timeout is tried again with jitter, honouring Retry-After; a 401, 402 or 403 is not retried at all, because no number of tries fixes a bad key or an empty balance.

ANONIFY_AI_DAILY_SPEND_USD is the gateway's real ceiling, estimated from the token counts already recorded on every call and the AI_PRICE_* rates. At 80% the gateway drops to one call at a time; at 100% the contextual pass is skipped for the rest of the day and pattern detection, manual redaction, rules and export all carry on — and the document says so, in its usage panel, rather than looking finished. Without AI_PRICE_* no spend can be estimated, and the app says that in the log rather than pretending to enforce a cap it cannot compute.

Several replicas: one budget

By default these limits are per process, which is the whole deployment when there is one container. Four workers each allowed four calls send sixteen at an account that allows four, and the provider answers with 429s. With ANONIFY_SERVICE_LIMIT_SCOPE=cluster the numbers above mean the whole deployment, whatever the replica count:

Variable	What it does	Default
ANONIFY_SERVICE_LIMIT_SCOPE	process, or cluster to share the AI and OCR limits between replicas	process
ANONIFY_RATE_STORE	Where the shared state lives: postgres, or redis	postgres
REDIS_URL	redis:// or rediss://, with ANONIFY_RATE_STORE=redis	unset
REDIS_KEY_PREFIX	Prefix for every key written	anonify:
REDIS_CLUSTER	true for a cluster-mode endpoint	false

• Postgres needs nothing new: one statement per request for the rate and one per call for the concurrency slot. That is comfortable up to a few hundred provider calls a second across the deployment, far above what any provider allows.
• Redis or Valkey is for very high request rates, or a deployment that already runs one: Redis 6 or later, Valkey, ElastiCache, MemoryDB, Memorystore, Azure Cache for Redis, Upstash. Every operation is a single-key script, so cluster mode needs nothing beyond REDIS_CLUSTER=true. ANONIFY_RATE_STORE=redis moves the inbound rate limiter's buckets there too, which takes the busiest write off Postgres.
• When the store is unreachable, documents keep processing: outbound calls fall back to each process's own limits, which is less precise, and the inbound limiter falls back to Postgres. A warning is logged every 30 seconds while it lasts, and /api/ready lists redis under degraded without taking the replica out of rotation.
• A provider's Retry-After holds every replica back, not just the one that got the 429, when a rate is configured for that service.

Test database

TEST_DATABASE_URL points the database-backed integration suites (pnpm test:db) at a throwaway database. These suites create and delete rows, so it must never be DATABASE_URL. Unset means the database-backed suites skip, and pnpm test still passes. See CONTRIBUTING.md.

Rate limits

Defaults differ by profile — the shared demo is strict, a self-hosted install is not — and can be changed without touching source:

pnpm rate-limit show                            # what is in force, and from where
pnpm rate-limit set --requests 500 --window 60  # all limits
pnpm rate-limit set upload --requests 20 --window 60
pnpm rate-limit reset

Three layers, later winning: profile defaults → ANONIFY_RATE_LIMIT_<NAME>=100/60 → the CLI, which writes to the database and takes effect without a restart.

Daily quotas

Separate from rate limits, and counted per identity per UTC day: pages, cells, images, uploads, and the model tokens Hush spends. They exist so one anonymous visitor cannot spend the shared demo's whole budget.

A self-hosted install has none. There is nobody to ration against, and a quota here is indistinguishable from the software being broken — it arrives as "processing failed" on a spreadsheet that is in no way unusual. If you are running something shared, set them explicitly:

ANONIFY_QUOTA_XLSX_CELLS=500000   # 0, or unset on self-hosted, means unlimited
ANONIFY_QUOTA_PDF_PAGES=200
ANONIFY_QUOTA_DOCX_PAGES=200
ANONIFY_QUOTA_TEXT_PAGES=400
ANONIFY_QUOTA_PPTX_SLIDES=200
ANONIFY_QUOTA_EMAIL_KILOBYTES=4096
ANONIFY_QUOTA_IMAGES=100
ANONIFY_QUOTA_UPLOADS=200
ANONIFY_QUOTA_ASSISTANT_TOKENS=2000000

The demo profile ships with its own defaults — small allowances, since one visitor's workbook is everyone's budget. These are what the hosted demo runs with when none of the env vars above are set:

Quota	Demo default	Unit
ANONIFY_QUOTA_PDF_PAGES	10	pages
ANONIFY_QUOTA_DOCX_PAGES	10	pages
ANONIFY_QUOTA_XLSX_CELLS	10,000	filled cells
ANONIFY_QUOTA_IMAGES	3	images
ANONIFY_QUOTA_TEXT_PAGES	40	pages of extracted text
ANONIFY_QUOTA_EMAIL_KILOBYTES	512	KiB of decoded text
ANONIFY_QUOTA_PPTX_SLIDES	20	slides
ANONIFY_QUOTA_UPLOADS	20	uploads
ANONIFY_QUOTA_ASSISTANT_TOKENS	300,000	model tokens Hush spends

The self-hosted profile sets every one of these to 0 (unlimited). The values come from lib/security/quota-config.ts, and pnpm setup reads them straight from there when it prints what it will enforce.

The units are not all the same shape, because the work is not:

Quota	Unit	Formats
PDF_PAGES / DOCX_PAGES	pages	PDF, DOCX
XLSX_CELLS	filled cells	XLSX, CSV, TSV
TEXT_PAGES	pages of extracted text	TXT, RTF
PPTX_SLIDES	slides	PPTX
EMAIL_KILOBYTES	KiB of decoded text: headers, every body, every nested message	EML, and every message out of an MBOX
IMAGES / UPLOADS	one each	all
ASSISTANT_TOKENS	model tokens, input and output, as the provider reports them	Hush, whichever document it is asked about

An email is charged by the text it actually decoded rather than as a page, because counting it as a page would charge a one-line reply the same as a forwarded thread. A deck is charged by its slides; its notes, layouts and masters are processed with the slide they belong to.

Hush is charged by the token, step by step, because a conversation has no pages and every step resends it. A run that uses up the allowance stops before its next step. This is on top of ANONIFY_AI_DAILY_SPEND_USD, which is one number for the whole instance: without a per-visitor allowance, one visitor could spend the cap and stop analysis for everybody.

A message additionally has parser limits of its own — MIME depth, part count, decoded text, header size, attachment count, nesting — which are independent of the upload ceiling on purpose. Raising the size a file may be must never be the same decision as allowing unlimited complexity. See .env.example.

A mailbox is charged nothing of its own, because it is never extracted: it expands, and each message it produces costs exactly what the same .eml would have cost uploaded on its own. What it has instead is bounds on the expansion, since one upload becoming nine hundred documents is the amplification these exist for:

Limit	Demo	Self-hosted	Bounds
ANONIFY_MBOX_MAX_MESSAGES	200	1000	documents one mailbox may produce
ANONIFY_MBOX_MAX_TOTAL_BYTES	32MB	64MB	content across every message
ANONIFY_MBOX_MAX_MESSAGE_BYTES	12MB	25MB	the largest single message
ANONIFY_MBOX_MAX_DEPTH	2	2	a mailbox reached through a mailbox

Every size here is written as a size — 32MB, 512KB, 1.5GB — rather than as a byte count, and so are the email parser's and expansion's. The units are powers of two, so 1KB is 1024 bytes and MB and MiB mean the same thing, which is what ls -h and docker --memory already do. A plain number is still read as bytes, so an existing .env keeps working. pnpm setup asks for them in the same units and writes back what it would have shown.

MAX_MESSAGES is its own number rather than ANONIFY_BATCH_MAX_FILES — that one is how many files a person may drag in at once, and a mailbox is one file — but it is floored at it, so a mailbox can never produce a smaller batch than the same person could have assembled by hand.

Spreadsheet cells are counted as cells that hold something, not as the area of the used range — a sheet with three filled columns and one stray value out in column AN is charged for what it contains, not for the blanks between.

Expiring documents when you self-host

Documents are temporary, which is only true if something is actually deleting them. A self-hosted Anonify does that itself: every process that runs workers runs the sweep every five minutes, and one replica leads at a time, so one container or fifty, exactly one sweeps (#183). It deletes expired documents, restarts runs a crashed worker left behind, and starts documents stuck in the queue. Nothing to schedule and no secret to share.

Setting	Default
ANONIFY_SCHEDULER	on	off leaves the sweep to an external scheduler
ANONIFY_SCHEDULER_INTERVAL_SECONDS	300	10–86400, with ±10% jitter so replicas do not wake together

On Vercel, which has no long-lived worker, the cron entry in vercel.json calls /api/cron/cleanup instead. (On the Hobby plan that entry can only fire once a day, so a Hobby deployment wants another backstop. See docs/deploy/vercel.md.)

Where workers are not long-lived, run the sweep as a job. It needs no server and no secret, so it suits cron, a systemd timer, Task Scheduler or a Kubernetes CronJob:

pnpm cleanup     # one pass: mark expired, delete documents, prune rate limits

*/15 * * * *  cd /srv/anonify && pnpm cleanup

The internal timer, the route and the job share one lock, so running several is harmless: whichever comes second steps aside.

The Compose scheduler service, which called the endpoint on a timer, is deprecated and removed in 1.18.0. Stop starting it with --profile scheduler; the app does its job now.

Whichever you choose, the sweep deletes the source, the normalized model, every export and the database row — and is idempotent, so a failed run is retried rather than leaving bytes nothing is tracking.

Database changes

Migrations are the canonical workflow, and CI applies them to an empty database on every push:

pnpm db:migrate         # create and apply a migration in development
pnpm db:migrate:deploy  # apply existing migrations (CI, production)
pnpm db:push            # schema straight to the database, for scratch work only

Commands

pnpm setup             # choose a setup, set the limits, write .env
pnpm setup --help      # its flags, for a scripted install
pnpm dev               # development server
pnpm build             # production build
pnpm test              # unit and adversarial suites
pnpm test:db           # the Postgres-backed suites (needs TEST_DATABASE_URL)
pnpm typecheck         # next typegen && tsc --noEmit
pnpm lint              # eslint
pnpm db:migrate        # create and apply a migration
pnpm db:migrate:deploy # apply existing migrations
pnpm rate-limit show   # inspect the limits in force
pnpm ocr:warm          # pre-download the Tesseract model
pnpm models:warm       # refresh the model lists and list prices setup shows
pnpm ai status         # the AI provider and model in force, and whether it is verified
pnpm cleanup           # run the expiry sweep once
pnpm smoke             # every format, against a running instance
pnpm smoke --only=eml  # or one of them
pnpm bench             # extraction and export, timed over large documents
pnpm bench:models      # models on the labelled corpus: quality, cost, and what the patterns save
pnpm bench:charts      # redraw the benchmark charts from the committed results

Node 22+ and pnpm 11+ are required and enforced — engines plus engine-strict, so an unsupported runtime fails at install with a clear message rather than somewhere confusing later.

Tests worth knowing about

• tests/security.test.ts — takes each exported artifact and tries to get the sensitive value back out: PDF text extraction, raw byte scans with glyph spacing stripped, annotation objects, every OOXML part, hidden sheets and rows, cached formula results, the shared string table, sampled pixels, EXIF.
• tests/adversarial.test.ts — one pass over every format through the real export path, asserting three things each: the accepted value is gone read in that format's own terms, a rejected value is still there (a redactor that empties the file passes the first test and is useless), and the artifact still opens as what it claims to be.
• tests/export.test.ts — end-to-end redaction per format, including that a suggestion the user never accepted is still present in the output.
• tests/integration/ — the database-backed suites. Expiry cleanup's ordering guarantee (storage first, the row second, and only if the storage actually went) and quota accounting's atomicity and charge-once behaviour are properties of the row rather than of the function, and a fake would agree with whatever the code did.
• tests/detectors.test.ts — that an order number is not reported as a card, and a date is only a birth date when it is labelled as one.
• scripts/smoke.ts — not a unit test: it drives a running instance over HTTP, uploading a synthetic document of each supported format and opening the downloaded export the way an adversary would — reparsing a CSV as a grid, reparsing a message with an independent MIME library, unzipping a deck and reading the notes and the master, sampling an image's pixels. CI runs it against the compose stack on every pull request, which is how the container's assembly gets checked at all — a missing native library or an unreachable workflow world passes every test above and fails here.

Benchmarks

The architecture makes a claim that could be wrong: that detecting with patterns first, asking a model only the contextual question and expanding repeats by local search costs far fewer tokens than sending the document to a model, at comparable quality. pnpm bench:models measures it on a labelled corpus of synthetic documents, for any model the app can use, and pnpm bench:charts draws what it found. How it is measured, and what it does not measure: benchmarks/README.md.

Every chart, the numbers behind each, and what they do not measure: benchmarks/README.md.

Contributing

The project is meant to be cloned, read and changed. CONTRIBUTING.md covers the invariants that must not break, an audit of what is currently half-wired, and what the next phase of work is.

The standing ask: benchmark it. The architecture claims that deterministic detection plus a narrow model pass costs far fewer tokens than sending documents to a model, at comparable quality. Measured on 100 test documents, the quality half now holds and the cost half does not. With glm-5.3-flash, the patterns first reach 95.0% F1 against 93.4% for the model alone, but they still cost more tokens, and the verification call is the cost they add every time (what it found, #217). That is one corpus and two models. More models, a size pair from one family, a local model, and a different corpus would each test the claim further: #208 lists them, and CONTRIBUTING says what to submit.

There are no accounts and no auth, deliberately: you run this against your own database and your own storage. The hosted demo is anonymous so people can try the tool without setting it up.

Found a way to get a redacted value back out of an exported file? That is the one thing here worth reporting privately first — SECURITY.md says how, and what does and does not count.

Security notes

• Per-document random data keys, sealed under a server-held master key. AES-256-GCM throughout. SHA-256 is used for integrity only — it is not encryption, and nothing here pretends otherwise.
• Ownership comes from a server-issued session id; quotas and rate limits use a coarsened network hash as well, so clearing a cookie does not reset the demo allowance. Only salted hashes are stored. No browser fingerprinting, and no MAC addresses — a browser cannot expose one.
• Blob URLs are never handed to the client. Downloads go through a signed, short-lived, ownership-checked route.
• Documents expire (1h / 6h / 24h / 3 days) and a scheduled sweep deletes the source, the normalized model, every export and every row. Retention can be extended, but the new expiry is computed from creation and capped at 72 hours, so renewing repeatedly converges on the ceiling rather than moving it.
• Logs carry ids, stages, durations and error categories. Never document content, never prompts, never keys.

License

Apache-2.0. Use it, fork it, run it commercially, deploy it inside a company — the patent grant is there so nobody's legal team has to think about it.

Section 7 is worth reading rather than skimming, though: this software comes with no warranty, including no warranty that any particular value was removed from any particular file. That is not boilerplate here. Anonify verifies every export against the values you accepted and refuses to deliver one that fails — but verification can only look for what it was told to look for, and the decision about what to accept was yours. Check the file before you send it.