feat(ai): add Cloudflare Workers AI as a second provider - #21
Open
cygmris wants to merge 2 commits into
Open
Conversation
Adds an opt-in "Local AI" settings section plus two entry points — a command palette action to convert pasted text into a new note, and a toolbar button to tidy the current note (or just the selection). Requests go straight from the browser to the user's own Ollama; note text never reaches the server. Because the Worker runs at the edge and cannot reach the user's loopback address, there is no server-side proxy option here; the CSP `connect-src` is widened to exactly two hardcoded loopback endpoints and nothing else. Two failure modes drove most of the design: 1. Ollama silently drops the front of an over-long prompt and the model then stops cleanly on the part it did see, reporting `finish_reason: "stop"`. Nothing in the API surfaces this. `detectLoss` therefore uses a second, independent heuristic — output far shorter than input on a first-turn conversion — and long input is chunked paragraph-first, then by line, then hard-cut. 2. Offsets captured when the command fires go stale while the model works. With realtime sync enabled the note can change underneath, so a selection edit would land in the wrong place and a whole-note edit would clobber a concurrent change — both silently. Writes now carry the original text and are refused if it no longer matches, keeping the result on screen so it can be copied instead. Browser support is Chrome/Firefox only, stated in the UI rather than only in docs: Safari blocks an HTTPS page from calling http://localhost. Chrome 138+ additionally gates loopback access behind a permission prompt, and an unanswered prompt leaves the request hanging rather than failing, so the connection test carries its own timeout and says so. Note content is treated as untrusted data: the system prompt tells the model to reformat rather than answer, and that instruction follows the user's UI language so weaker local models weight it properly. Tests: 138 passing. Assertions were reverse-verified by breaking each implementation in turn and confirming the matching test goes red.
Builds on the local Ollama provider: same engine, same loss guards, same
panel — only the endpoint differs. A thin same-origin route translates
Workers AI's stream into the OpenAI-compatible SSE shape the client already
speaks, so the client needs no branching beyond one URL and one header.
This is worth having alongside the local option because it works where the
local one cannot: Safari, phones, and any machine that is not the one running
Ollama. It also sidesteps the loopback permission prompt entirely, since the
request is same-origin.
No new credentials: the `AI` binding already exists for semantic search.
- `src/shared/ai-models.ts` — allow list of 7 models within the free
allowance, shared by the server (to reject anything else) and the client (to
fill the dropdown). One list, because two would drift into "selectable in the
UI, rejected by the server".
- `src/worker/lib/workers-ai-stream.ts` — pure-function translator, unit
tested. Accepts both upstream chunk shapes; observed traffic carries
`{response}` and `{choices[0].delta.content}` side by side. Unreadable
chunks are skipped rather than aborting the stream. Deliberately imports no
`cloudflare:` module so it stays loadable under the repo's vitest setup.
- `src/worker/routes/ai.ts` — `POST /api/ai/chat` behind `requireAuth`, with
allow-list validation. Without that check the route would proxy arbitrary
Workers AI models on the account's quota.
Two measured details shaped the implementation:
Every model tested spends most of its output budget on hidden reasoning — for
one prompt, qwen3-30b used 121 completion tokens to produce 12 visible
characters. Passing `chat_template_kwargs: {enable_thinking: false}` cuts that
to 3, roughly a twentyfold reduction. Mistral rejects the parameter
("chat_template is not supported for Mistral tokenizers") and emits no
reasoning anyway, so support is declared per model in the shared list rather
than branched on a model name at the call site.
The `X-Inkstone-Client` header is sent only in cloud mode. Adding it
unconditionally would break the local path, since Ollama's CORS preflight does
not allow that header through.
The default provider stays local Ollama, and the cloud option states plainly
in the UI that note text is sent to Cloudflare — that trade-off should be the
user's to make explicitly.
Tests: 155 passing, assertions reverse-verified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Cloudflare Workers AI as an optional second provider for the same feature. Same engine, same loss guards, same panel — only the endpoint differs.
It earns its place by working where the local option cannot: Safari, phones, and any machine that is not the one running Ollama. It also sidesteps the loopback permission prompt entirely, since the request is same-origin.
No new credentials — the
AIbinding already exists for semantic search.How it fits
A thin same-origin route translates Workers AI's stream into the OpenAI-compatible SSE shape the client already speaks, so the client needs no branching beyond one URL and one header.
src/shared/ai-models.tssrc/worker/lib/workers-ai-stream.tscloudflare:module so it stays loadable under the repo's vitest setup.src/worker/routes/ai.tsPOST /api/ai/chatbehindrequireAuth, with allow-list validation.The allow-list check is load-bearing, not ceremony: without it the route would proxy arbitrary Workers AI models on the deployment's own quota.
Two measured details
Upstream chunks carry both shapes at once. Observed traffic has
{"response": "..."}at the top level andchoices[0].delta.content, mirroring each other. The translator reads whichever is non-empty rather than betting on one, and skips unreadable chunks instead of aborting the stream.Hidden reasoning dominates the token budget. For one prompt,
qwen3-30bspent 121 completion tokens to produce 12 visible characters — roughly 90% on reasoning the user never sees. Passingchat_template_kwargs: {enable_thinking: false}cuts that to 3 tokens, about a twentyfold reduction in neurons. Measured across the list:qwen3-30b-a3b-fp8gemma-4-26b-a4b-itllama-3.3-70b-fp8-fastgpt-oss-20bchat_template is not supported for Mistral tokenizers) and emits no reasoning anyway, so support is declared per model in the shared list rather than branched on a model name at the call site. A test asserts every entry declares the flag explicitly, so a newly added model cannot silently inherit a default.One non-obvious constraint
X-Inkstone-Clientis sent only in cloud mode. Adding it unconditionally breaks the local path, because Ollama's CORS preflight does not allow that header through. There are tests pinning that difference in both directions.Privacy
The default provider stays local Ollama. The cloud option states plainly in the UI that note text is sent to Cloudflare, and that the free allowance is shared with this notebook's semantic search — that trade-off should be the user's to make explicitly, not a default they discover later.
Checks run
Assertions were reverse-verified by breaking each implementation in turn and confirming the matching test goes red.
npm run test:e2edoes not complete on my machine, and neither does it on an unmodified checkout ofmain— wrangler crashes partway through with its own "please create an issue at workers-sdk" message. I did not upgrade wrangler since that would be an unrelatedpackage-lock.jsonchange. The route was instead exercised against a real deployment: unauthenticated → 401, missing client header → 403, paid-only model → 400, arbitrary string → 400, empty messages → 400, malformed role → 400, valid request → 200text/event-stream.Not included
No AI Gateway integration, no usage dashboard, no per-user quota. The quota-exhausted path maps to 429 by standard semantics but I never actually hit the limit, so that branch is unverified — flagging it rather than implying it was tested.