KBPrep

KBPrep is a local CLI for turning source material into quality-checked, traceable Markdown deliverables.

The project boundary is the source-to-deliverable quality loop:

1. inspect the input and declare the selected capability
2. choose one deterministic conversion route
3. preserve source evidence in auditable artifacts
4. build toward a Canonical IR fact layer
5. run conversion quality checks before cleanup
6. classify document type from bounded evidence
7. compile deterministic cleanup rules and private user preferences
8. reject unsafe cleanup changes and preserve original text
9. publish source-side Markdown and assets only after hard gates pass
10. turn feedback into rule proposals before promotion

KBPrep does not maintain AI development agent adapters. Calling environments should use the CLI and documented JSON artifacts when they need their own skill, plugin, or host integration.

Current State

The current codebase already has a Python worker, a Node CLI bridge, conversion routes, quality reports, source-side publication, and proposal-first feedback. It is not yet the full target architecture.

Target design and current status are tracked in:

• docs/kbprep-core-flow-design.md
• docs/kbprep-full-flowchart.html
• docs/kbprep-development-implementation-plan.md
• docs/development/kbprep-implementation-status.json
• docs/capability-matrix.md
• docs/quality-loop.md
• docs/feedback-learning.md
• docs/standalone-cli.md

Do not claim a source format or target route is fully supported unless the capability matrix links it to named tests or fixtures.

CLI

Standalone commands:

kbprep-preflight --help
kbprep-analyze --input ./source.pdf --output ./.kbprep/analyze
kbprep-prepare --input ./source.pdf --output ./.kbprep/source --force
kbprep-prepare --input https://www.youtube.com/watch?v=ExampleVideo01 --output ./.kbprep/youtube --force
kbprep-prepare --youtube-video-id ExampleVideo01 --output ./.kbprep/youtube --allow-youtube-media-fallback --force
kbprep-apply-review --run-dir ./.kbprep/source/runs/<run-id> --patch-file review.patch.json
kbprep-feedback --run-dir ./.kbprep/source/runs/<run-id> --feedback-text "下次删除「关注公众号」这种污染"
kbprep-feedback --run-dir ./.kbprep/source/runs/<run-id> --plan-rerun
kbprep-feedback --run-dir ./.kbprep/source/runs/<run-id> --execute-rerun
kbprep-feedback --accept-proposal latest --confirm-rule-acceptance
kbprep-cleanup --output ./.kbprep/source --dry-run
kbprep-batch --input ./sources --output ./.kbprep/batch
kbprep-batch --playlist https://www.youtube.com/playlist?list=ExamplePlaylist01 --output ./.kbprep/playlist

The CLI prints JSON envelopes for worker results. Failures use the same shape with ok: false, an error code, warnings when available, and evidence paths when available. Batch runs write batch_manifest.json with parent status, per-file status, skipped unsupported files, source hashes, command defaults, and evidence-backed rerun scope. Explicit YouTube playlist runs first expand the playlist into bounded local .url child descriptors, then run each child through the same quality pipeline. This is the live batch status summary. Use kbprep-batch --rerun --batch-manifest <batch_manifest.json> to rerun failed or pending children without rerunning unrelated successful children; rerun uses the manifest command defaults, writes batch_rerun_manifest.json, and refuses missing or changed source files. After a batch output root is finalized with kbprep-cleanup --action finalize, cleanup writes kbprep_batch_manifest.json; that file is only the retention manifest proving final deliverables were preserved before temporary process artifacts were removed.

PDF Routing

PDF routing is diagnosis-selected: simple trusted text-layer PDFs use pymupdf4llm, complex trusted PDFs use MinerU txt or auto, and scanned or untrusted text-layer PDFs use MinerU ocr. conversion_report.json.route_decision records the selected tier, actual route, fallback or upgrade, and reason.

Optional Media And YouTube Routes

Local audio/video files can route through local ffmpeg + Whisper transcription before KBPrep runs the normal conversion and publication gates. The route is partial, not verified: mocked golden fixtures prove routing and error reporting, while real ASR quality evidence is still required before verified promotion.

YouTube currently accepts direct YouTube URLs, explicit --youtube-video-id values, local .url descriptor files, or explicit playlist input through kbprep-batch --playlist. KBPrep writes URL inputs into controlled local descriptors, tries subtitles first through yt-dlp, and falls back to the local media transcript route only when --allow-youtube-media-fallback is explicitly set. That fallback downloads video/media through the yt-dlp Python package, transcribes it with local ffmpeg + Whisper, then sends the transcript through the normal cleanup, quality, and Obsidian Markdown publication gates. The route is partial, not verified: recorded-equivalent subtitle report coverage plus mocked fixtures prove routing, playlist expansion, fallback behavior, and failure handling, while real-network sample breadth, timeout behavior, dependency variance, and transcript quality evidence are still required before verified promotion.

Output

The maintained standard profile publishes beside the source:

<source-folder>/<source-stem>.md
<source-folder>/<source-stem>.assets/

When the source is Markdown, KBPrep avoids overwriting it:

<source-folder>/<source-stem>.cleaned.md
<source-folder>/<source-stem>.assets/

Process artifacts remain under the chosen output directory or job directory. Failed runs must not update the previous successful deliverable.

Successful runs expose latest_outputs.publish_report. Runs blocked after quality checks keep publish_report.json in the run directory so the owner can see why final publication did not happen. Runs blocked earlier by the pre-clean conversion gate report conversion_quality_report.json and error_report.json in the error envelope details instead.

Runtime

KBPrep creates its own Python runtime at .kbprep/venv inside the package directory. It installs worker dependencies there instead of relying on system Python packages.

Current worker dependencies include converter and OCR tooling. KBPrep should use proven open-source tools where possible; it should not become a custom OCR project.

First-run setup is split into visible steps: create venv, upgrade packaging tools, install worker dependencies, and run the setup-env probe. Advanced operators can override the bootstrap Python with KBPREP_BOOTSTRAP_PYTHON or configured python_path; setup timeout failures include stderr evidence.

For direct Python worker development, use the managed project environment:

node scripts/python-venv.mjs --print-python
node scripts/python-venv.mjs -m kbprep_worker.cli --help

Build And Test

npm install
npm run build
npm test
npm run pack:check

Use the release gate before publishing:

npm run release:check

When F:\Obsidian-Vault is available, run the isolated real-document smoke suite as an additional local release check:

npm run vault:smoke

vault:smoke copies representative files to a temporary directory before running prepare or batch. It must not write final Markdown or assets back into the original Obsidian vault.

Agent Usage

Agents should treat KBPrep as a CLI tool plus documented protocols:

• call the CLI
• read quality and review artifacts
• return a validated patch or rule proposal
• let KBPrep apply changes and rerun gates

The repository intentionally does not ship AI development agent adapter logic.