Skip to content

liangzid/GemFilter

BranchesTags

Repository files navigation

GemFilter

GemFilter 0.2.2 Local first privacy Agent integrations Python 3.11+

A local privacy firewall for coding agents.
GemFilter detects, masks, tracks, and sanitizes sensitive developer data before it reaches LLMs, tools, logs, or agent context.

Quick Start · Agent Setup Prompt · 中文版 · Agent Privacy Boundary · Masking Modes · Interfaces · Configuration

Copy-paste Agent Setup

Already using a coding agent? Copy this prompt and send it to the agent from the project where you want GemFilter enabled:

Please install and configure GemFilter for this coding-agent project.

What I want:
- Clone or inspect the GemFilter project from https://github.com/liangzid/GemFilter.
- Install GemFilter with pip.
- Read the relevant installation/configuration docs before changing my agent config.
- Configure GemFilter for the coding agent used in this project.
- Run a safe smoke test with fake secrets only.
- Do not print, copy, summarize, or expose any real secrets from my machine.

Steps:
1. Clone GemFilter somewhere temporary if the repo is not already available:
   git clone https://github.com/liangzid/GemFilter.git
2. Read these files from the GemFilter repo:
   - README.md
   - gemfilter/skill/README.md
   - docs/CONFIGURATION.md
3. Before installing, ask me which privacy level I want:
   - strict: maximum privacy; use typed placeholders such as <EMAIL_1>.
   - balanced: default; preserve useful syntax while hiding values, such as <EMAIL_LOCAL_1>@<EMAIL_DOMAIN_1>.
   - utility: preserve more task utility with plausible fake values, such as user1@example.test.
   If I do not answer, use balanced.
4. Install GemFilter:
   pip install gemfilter
5. Detect the coding-agent environment in this project:
   - Claude Code: .claude/ or .claude/settings.json
   - OpenCode: ~/.config/opencode/opencode.json or .opencode/
   - Codex/MCP: .codex/ or MCP config
6. Configure the matching integration according to the docs you read.
7. Configure the requested privacy level with masking_mode: strict, balanced, or utility where GemFilter config is used.
8. Run a safe local smoke test with fake values only:
   python -m gemfilter.cli filter "Contact user@example.com and OPENAI_API_KEY=sk-proj-abcdefghijklmnopqrstuvwxyz123456"
9. If configuring OpenCode, run one non-interactive opencode test with fake values and compare:
   - GemFilter enabled
   - GEMFILTER_OPENCODE_DISABLED=1
10. Report:
   - which agent integration was configured,
   - which privacy level was selected,
   - which config files changed,
   - how to disable or uninstall it,
   - whether the smoke test proves that fake email/API key values were filtered.

If multiple agent environments are present, ask me which one to configure before making changes.

Overview

GemFilter started as a sensitive-information redactor. In v0.2, it is moving toward a more practical role: a local privacy boundary for AI coding agents.

Coding agents do not only receive user prompts. They inspect repositories, read files, execute shell commands, consume MCP tool results, write transcripts, and echo model responses. Private data can cross any of those boundaries:

Boundary Example risk GemFilter protection
User prompt User pastes an API key into a request Pre-send filtering
Tool output Shell output prints .env values Tool-output filtering
Repository context Config files contain private endpoints Recursive payload filtering
Model response LLM echoes a surrogate or generates a new secret Post-receive sanitization
CLI / HTTP output The filter itself returns raw matches Safe serialization by default

GemFilter is local, rule-based, and LLM-independent. It is designed to be understandable, auditable, and easy to integrate into agent workflows.

What It Protects

GemFilter includes built-in rules for common developer privacy risks:

Category Examples
Credentials API keys, passwords, bearer tokens, JWTs
Provider tokens OpenAI, Anthropic, GitHub, npm, PyPI
Cloud secrets AWS access keys, AWS secret access keys
Local config .env secret assignments, database URLs
Contact data Email addresses, Chinese and US phone numbers
Personal identifiers Chinese ID cards, passports, credit cards
Network data URLs, IPv4, IPv6, MAC addresses

The default output is safe: serialized detections do not include raw sensitive matches unless an explicit unsafe debug flag is used.

Agent Privacy Boundary

GemFilter protects three main runtime paths:

User prompt / context
        |
        v
  pre_send hook
        |
        v
Masked context -----------------------> LLM / agent
        |                                  |
        |                                  v
        |                          model response
        |                                  |
        v                                  v
Tool output / MCP result -----> post_receive sanitizer
        |
        v
filter_tool_output hook

The same local session is used across these paths, so a surrogate generated during pre-send can be reused later when the same value appears in tool output.

Masking Modes

Different coding tasks need different privacy-utility tradeoffs. GemFilter provides three modes.

Mode Example Best for
strict john@example.com -> <EMAIL_1> Maximum privacy
balanced john@example.com -> <EMAIL_LOCAL_1>@<EMAIL_DOMAIN_1> Default coding-agent use
utility john@example.com -> user1@example.test Tests and examples that need plausible fake data

Secrets such as API keys, passwords, private keys, bearer tokens, and database URLs remain typed placeholders even in utility-oriented workflows.

Example:

from gemfilter.skill import GemMasker

strict = GemMasker(masking_mode="strict")
balanced = GemMasker(masking_mode="balanced")
utility = GemMasker(masking_mode="utility")

text = "Contact john@example.com with OPENAI_API_KEY=sk-proj-abcdefghijklmnopqrstuvwxyz123456"

print(strict.mask(text)[0])
# Contact <EMAIL_1> with OPENAI_API_KEY=<OPENAI_KEY_1>

print(balanced.mask(text)[0])
# Contact <EMAIL_LOCAL_1>@<EMAIL_DOMAIN_1> with OPENAI_API_KEY=<OPENAI_KEY_1>

print(utility.mask(text)[0])
# Contact user1@example.test with OPENAI_API_KEY=<OPENAI_KEY_1>

Quick Start

Install

pip install gemfilter

For local development:

git clone https://github.com/liangzid/GemFilter.git
cd GemFilter
pip install -e .

CLI

gemfilter filter "Contact user@example.com and OPENAI_API_KEY=sk-proj-abcdefghijklmnopqrstuvwxyz123456"

Output:

Contact [EMAIL] and OPENAI_API_KEY=[OPENAI_API_KEY]

JSON output is safe by default:

gemfilter filter "Contact user@example.com" --json
{
  "text": "Contact [EMAIL]",
  "detections": [
    {
      "rule": "email",
      "start": 8,
      "end": 24,
      "sensitive_type": "contact",
      "replacement": "[EMAIL]",
      "match_length": 16
    }
  ],
  "summary": {
    "email": 1
  }
}

Raw matches require an explicit unsafe opt-in:

gemfilter filter "Contact user@example.com" --json --unsafe-include-matches

Python SDK

from gemfilter import SandFilter

sf = SandFilter()
result = sf.filter("My email is test@example.com, phone 13800138000")

print(result.text)
# My email is [EMAIL], phone [PHONE_CN]

print(result.summary)
# {'email': 1, 'phone_cn': 1}

Agent Hook API

from gemfilter.skill import HookManager

manager = HookManager()

pre = manager.pre_send(
    "Send the report to john@example.com. The token is sk-proj-abcdefghijklmnopqrstuvwxyz123456.",
    session_id="demo",
)

print(pre.payload)
# Send the report to <EMAIL_LOCAL_1>@<EMAIL_DOMAIN_1>. The token is <OPENAI_KEY_1>.

tool = manager.filter_tool_output(
    {
        "tool": "shell",
        "stdout": "DATABASE_URL=postgres://user:pass@db.internal:5432/app",
        "exit_code": 0,
    },
    session_id="demo",
)

print(tool.payload["stdout"])
# DATABASE_URL=<DATABASE_URL_1>

post = manager.post_receive(
    "I saw <EMAIL_LOCAL_1>@<EMAIL_DOMAIN_1> in the logs.",
    session_id="demo",
)

print(post.payload)
# I saw [FILTERED] in the logs.

Interfaces

GemFilter can be used through several local interfaces.

Interface Command / API Use case
Python SDK SandFilter Library filtering
Skill API HookManager Agent pre-send, tool-output, post-receive hooks
CLI gemfilter filter Shell workflows and scripts
HTTP server gemfilter-server Local REST filtering
MCP / Codex schema gemfilter_filter_tool_output Tool-result filtering for agent contexts

HTTP Server

gemfilter-server --host localhost --port 8080

Endpoints:

Method Endpoint Description
GET /health Health check
GET /rules Enabled and disabled rules
POST /filter Filter one text field
POST /filter/batch Filter multiple text fields

Example:

curl -X POST http://localhost:8080/filter \
  -H "Content-Type: application/json" \
  -d '{"text": "Email user@example.com"}'

Agent Integrations

Install GemFilter into the coding-agent project where you want local privacy protection. The installer writes agent-specific hook configuration in the current working directory.

Agent Integration surface Hook coverage
Claude Code settings.json hooks pre-send, post-receive, tool-output
OpenCode plugin hooks chat message transform
Codex MCP-style tool/resource schema filter, restore, tool-output filter

Claude Code

From the root of your coding project:

pip install gemfilter
python -m gemfilter.skill.install --agent claude_code
python -m gemfilter.skill.install --agent claude_code --status

This creates or updates:

.claude/settings.json

Registered hooks:

onBeforeSend   -> gemfilter.skill.hooks.pre_send_hook
onAfterReceive -> gemfilter.skill.hooks.post_receive_hook
onToolOutput   -> gemfilter.skill.hooks.tool_output_hook

Uninstall:

python -m gemfilter.skill.install --agent claude_code --uninstall

OpenCode

OpenCode 1.14+ uses a real JavaScript plugin API. The recommended integration is a local plugin that filters text in experimental.chat.messages.transform before messages enter model context.

Minimal global plugin setup:

pip install gemfilter
mkdir -p ~/.config/opencode

Create ~/.config/opencode/gemfilter-plugin.mjs:

import { spawnSync } from "node:child_process";

const PYTHON = process.env.GEMFILTER_PYTHON || "python3";
const DISABLED = process.env.GEMFILTER_OPENCODE_DISABLED === "1";

function filterText(text) {
  if (DISABLED || typeof text !== "string" || text.length === 0) return text;
  const result = spawnSync(PYTHON, ["-m", "gemfilter.cli", "filter"], {
    input: text,
    encoding: "utf8",
    maxBuffer: 10 * 1024 * 1024,
  });
  if (result.status !== 0 || result.error) return text;
  return result.stdout.endsWith("\n") ? result.stdout.slice(0, -1) : result.stdout;
}

export default async function GemFilterPlugin() {
  return {
    "experimental.chat.messages.transform": async (_input, output) => {
      for (const message of output.messages ?? []) {
        for (const part of message.parts ?? []) {
          if (part?.type === "text" && typeof part.text === "string") {
            part.text = filterText(part.text);
          }
        }
      }
    },
  };
}

Then add the plugin path to ~/.config/opencode/opencode.json:

{
  "plugin": ["/home/YOUR_USER/.config/opencode/gemfilter-plugin.mjs"]
}

Test with a real non-interactive OpenCode call using fake secrets:

opencode run --format json \
  "Repeat exactly this one line and nothing else: Contact user@example.com and OPENAI_API_KEY=sk-proj-abcdefghijklmnopqrstuvwxyz123456"

Expected assistant text:

Contact [EMAIL] and OPENAI_API_KEY=[OPENAI_API_KEY]

Temporary disable:

GEMFILTER_OPENCODE_DISABLED=1 opencode

The legacy python -m gemfilter.skill.install --agent opencode adapter is kept for compatibility with older assumptions, but current OpenCode versions should use the plugin approach above.

Codex / MCP

From the root of your coding project:

pip install gemfilter
python -m gemfilter.skill.install --agent coodex
python -m gemfilter.skill.install --agent coodex --status

This creates or updates:

.codex/mcp_config.json

Registered resources and tools:

gemfilter://filter      -> pre-send filtering
gemfilter://restore     -> response sanitization
gemfilter://tool-output -> tool-output filtering
gemfilter_filter_tool_output

Uninstall:

python -m gemfilter.skill.install --agent coodex --uninstall

Check All Agents

python -m gemfilter.skill.install --status

Note: the Codex adapter is currently named coodex internally for backwards compatibility. The user-facing integration target is Codex/MCP.

Adapter behavior should still be validated against the exact live hook format of each host agent. The internal GemFilter APIs and tests are stable, but host-agent hook contracts can change.

Configuration

GemFilter looks for skill configuration in this order:

  1. GEMFILTER_SKILL_CONFIG
  2. ./config/skill.yaml
  3. ./gemfilter/skill/config.yaml
  4. ~/.gemfilter/skill.yaml

Example:

name: "gemfilter"
version: "1.0.0"
auto_activate: true

notification:
  style: "banner"
  show_types: true
  show_count: true

masking_mode: "balanced"  # strict | balanced | utility
preserve_format: true

filter:
  config_path: null
  auto_update: true
  enabled_types: []
  filter_tool_outputs: true

Tool-output filtering can affect coding-agent utility when public/example strings need to remain exact. Disable it globally:

filter:
  filter_tool_outputs: false

Or skip one structured payload:

manager.filter_tool_output({
    "gemfilter_skip": True,
    "stdout": "public example value that must stay exact",
})

Full details: Configuration Guide

Custom Rules

Add project-specific rules with regex patterns.

from gemfilter import DetectionRule, SandFilter

sf = SandFilter()

sf.add_rule(
    DetectionRule(
        name="student_id",
        pattern=r"STU\d{8}",
        priority=1,
        sensitive_type="education",
        group="custom",
    )
)

print(sf.filter("Student ID: STU20240001").text)
# Student ID: [STUDENT_ID]

YAML configuration:

settings:
  default_processor: rule_name

rules:
  - name: student_id
    pattern: "STU\\d{8}"
    priority: 10
    sensitive_type: education
    group: custom
    processor: replace
    processor_config:
      replacement: "[STUDENT_ID]"

Built-in Rules

Rule Description
email Email address
phone_cn, phone_us Chinese and US phone numbers
id_card_cn, passport Personal identifiers
credit_card, bank_account_cn Financial identifiers
password, dotenv_secret Passwords and .env secret assignments
api_key, api_key_generic API key assignments and generic sk-... keys
openai_api_key, anthropic_api_key Provider-specific LLM API keys
github_token, npm_token, pypi_token Developer platform tokens
bearer_token, jwt Bearer tokens and JWTs
aws_access_key, aws_secret_key AWS credentials
private_key Private key headers
database_url PostgreSQL, MySQL, MongoDB, Redis URLs
ipv4, ipv6, mac_address, url Network identifiers

Design Principles

Principle Meaning
Local first Sensitive text is processed before it leaves the machine.
Safe by default CLI and HTTP outputs do not reveal raw matches by default.
Agent-aware Prompt, tool-output, and response paths are treated separately.
Session-aware Surrogates are reused across a local multi-turn session.
Utility-conscious Strict, balanced, and utility modes make tradeoffs explicit.
LLM-independent The core engine is deterministic and rule-based.

Development

Run tests:

python -m pytest -q

Current v0.2 hardening coverage includes:

safe CLI/HTTP serialization
session mapping correctness
strict/balanced/utility surrogate modes
stronger developer secret detection
tool-output filtering
CLI and HTTP smoke tests

Project documentation:

License

MIT. See the repository license for details.

About

Filter and sanitize all your Gems (e.g., API tokens, passwords, emails, ...) before send them into Agents or LLM APIs.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 3

GemFilter v0.2.2 Latest
May 24, 2026
+ 2 releases

Packages

 
 
 

Contributors

Languages