ANEDVD — Apple Neural Engine Decoupled Vandal Driver

CoreML-free LLM inference on the Apple Neural Engine, in Zig.

anedvd runs a real transformer directly on the Apple Neural Engine from a Zig program. It never links CoreML: MIL programs are generated by this project and compiled through Apple's private AppleNeuralEngine.framework (_ANEInMemoryModelDescriptor / _ANEInMemoryModel), with IOSurfaces for zero-copy tensor I/O.

Decoupled — no CoreML, no mlmodelc, no CoreML compiler: the MIL is written by this program. Vandal — it drives the ANE through undocumented private frameworks, which Apple does not support and App Store review rejects.

It loads GGUF models (Q8_0, Q4_0/1, Q5_0/1, F16, F32, BF16, Q2_K–Q8_K) and HuggingFace directories (safetensors + config.json + tokenizer.json), generates text from the CLI, and serves an OpenAI- and Anthropic-compatible HTTP API with a built-in WebUI. Both tokenizer families work: byte-level BPE (Llama 3, Qwen2/3, SmolLM2/3) and SentencePiece (Llama-2/TinyLlama-era).

Verified on this machine

Hardware	Apple A18 Pro, Mac17,5, 6 cores, 8 GB
OS	macOS 27.0.1 (26A434), SIP enabled
Binary	unsigned/ad-hoc, no entitlements
Zig	0.17.0

$ zig build                       # defaults to ReleaseFast
$ ./zig-out/bin/anedvd run --model models/smollm2-135m-q8_0.gguf \
      --prompt "What is the capital of France?" --max-tokens 40
prompt (16 tokens): <|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

---
The capital of France is Paris. Paris is a city located in the northern part of
the country, and it is known for its historical landmarks, cultural institutions,
and cultural attractions. Paris is famous for
---
prefill: 16 tokens in 0.04 s (400 tok/s; ANE 0.02 s, CPU 0.02 s)
decode: 40 tokens in 1.26 s (31.7 tok/s; ANE 88%, CPU 12%)
ANE per-token split: qkv 0.20 ms, o 0.18 ms, ffn 0.24 ms, lm_head 1.65 ms

Five model families run on the ANE. Decode is the median of 9 repeats; the range column is min..max, because the ANE's own timing varies a lot run to run and a single measurement is not a result:

model	params / format	decode	range	peak RSS
SmolLM2-135M-Instruct	135 M, GGUF Q8_0	52.2 tok/s	32–57	304 MB
SmolLM2-135M-Instruct	135 M, HF safetensors F16	47.4 tok/s		~450 MB
Qwen2.5-0.5B-Instruct	494 M, GGUF Q8_0	22.7 tok/s	21–26	1.0 GB
Qwen3-0.6B	600 M, GGUF Q8_0	17.1 tok/s		~1.1 GB
Qwen2.5-1.5B-Instruct	1.5 B, GGUF Q4_K_M	9.5 tok/s	9.2–9.6	1.6 GB
TinyLlama-1.1B-Chat	1.1 B, GGUF Q8_0	17.0 tok/s		~1.5 GB

Long prompts prefill faster per token because the ANE is fed a whole chunk at a time: Qwen2.5-0.5B reaches 698 tok/s at 203 prompt tokens. Decode is at the ANE's weight-bandwidth ceiling, so it barely changes with model size in relative terms; docs/RESULTS.md has the full measurements, the per-kernel bandwidth table and the accuracy numbers.

Numerical correctness is checked at three levels:

• anedvd verify <model.gguf> runs the whole prompt through the ANE and through the pure-CPU reference and compares the final logits. This is the check that matters: check validates single kernels in isolation, and a transposed activation layout between kernels once turned Qwen2.5-1.5B into garbage while every per-kernel check still passed. All four GGUF models report a relative error of 1.2–1.7% (fp16 accumulation across ~30 layers).

• anedvd selftest builds a tiny random transformer, runs it through the ANE and compares against a pure-CPU reference of the same model: 4.3×10⁻⁴ relative error, i.e. fp16 rounding.

• anedvd cpu --model … --prompt … runs the same model entirely on the CPU, which is how the model-handling bugs (RoPE permutation, attention biases) were separated from ANE issues.

• anedvd check <model.gguf> compares every kernel of layer 0 against a CPU matmul with the same weights. Typical result:

    qkv          max|ANE-CPU| = 2.99788e-3  rel = 3.55246e-4  nan = 0  -> ok
    o            max|ANE-CPU| = 2.00653e-3  rel = 1.41123e-4  nan = 0  -> ok
    gate         max|ANE-CPU| = 1.13177e-3  rel = 4.73259e-4  nan = 0  -> ok
    up           max|ANE-CPU| = 1.01376e-3  rel = 4.65309e-4  nan = 0  -> ok
    lm_head      max|ANE-CPU| = 6.44226e-2  rel = 3.66178e-4  nan = 0  -> ok

Build

zig build                  # produces zig-out/bin/anedvd (ReleaseFast)
zig build test             # 120 tests; the ANE tests skip without an ANE
zig fmt src build.zig

Requirements: Apple Silicon, macOS 15+, Xcode command line tools (the Objective-C shim is compiled with /usr/bin/clang because Zig's bundled clang crashes on ARC Objective-C), Zig 0.17.

Getting models

Nothing is bundled; models are large and models/ is gitignored. These are the ones this project was developed against:

mkdir -p models && cd models

# GGUF (single file, includes the tokenizer)
curl -LO https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q8_0.gguf
curl -LO https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf

# HuggingFace directory (safetensors + config.json + tokenizer.json)
mkdir -p SmolLM2-135M-Instruct && cd SmolLM2-135M-Instruct
for f in config.json tokenizer.json model.safetensors; do
  curl -LO "https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/resolve/main/$f"
done

Rough memory guidance on an 8 GB machine: the fp16 weights are baked into the ANE's compiled programs, so a Q8_0 0.5B model peaks around 1 GB of RSS and a 135M model around 320 MB. A 1B model at Q8_0 works but leaves little headroom; larger models need a machine with more unified memory.

Commands

Command	Purpose
anedvd info	show whether the ANE is reachable and which private classes resolved
anedvd probe [--size N]	compile a 1×1-conv matmul, run it, compare against CPU
anedvd bench	matmul throughput sweep (128 … 4096)
anedvd selftest	tiny transformer: ANE engine vs CPU reference
anedvd check <model.gguf> [--split]	load a real GGUF, compile every kernel, per-kernel diff, sample predictions
anedvd verify <model.gguf> [--layers N]	whole-model check: every layer on the ANE vs the pure-CPU reference
anedvd layers <model.gguf> --upto N	run only the first N layers (bisect a model that produces garbage)
anedvd kernels <model.gguf>	per-kernel timing and achieved weight bandwidth
anedvd run --model <m.gguf|hf-dir> --prompt "…" [--max-tokens N] [--temp T] [--top-k K] [--top-p P] [--repeat-penalty R] [--chunk N] [--max-seq N] [--ab]	generate text on the ANE
anedvd chat --model <m.gguf|hf-dir> [--system "…"] [--max-seq N]	interactive multi-turn chat in the terminal (/help, /reset, /temp, /topp, /rep)
anedvd cpu --model <m.gguf> --prompt "…" [--chat]	pure-CPU reference generation (validates model handling without the ANE)
anedvd serve --model <m.gguf> [--host 127.0.0.1] [--port 8080] [--max-seq N]	HTTP server: OpenAI + Anthropic compatible API and a built-in WebUI

--model accepts either a .gguf file or a HuggingFace model directory (model.safetensors + config.json + tokenizer.json, multi-shard aware). The format is chosen by the path: anything ending in .gguf is read as GGUF, everything else as an HF directory.

Overrides for unusual models: --rope-hf / --rope-adjacent force the RoPE convention, --chunk N sets the activation width (default 64), --ab runs the prompt through both the batched and the per-token path and compares the logits, --fuse enables the experimental fused FFN.

Prompt batching: why a chunk costs the same as a token

Every kernel is compiled for a fixed activation width (--chunk, default 64). A decode step fills only column 0; a prefill step fills up to 64 columns. That sounds wasteful until you measure anedvd width:

ANE conv cost vs activation width (2048 -> 2048, fp16, weights read once)
  width      us/eval       vs w=1      GFLOP/s
      1       304.97        1.00x         27.5
      8       286.93        0.94x        233.9
     32       305.47        1.00x        878.8
     64       290.23        0.95x       1849.8
    128       321.40        1.05x       3340.8

The cost is flat: the ANE is reading weights, not doing arithmetic, so 128 columns cost 5% more than one. One kernel set therefore serves both paths, with no extra compiles and no measurable decode penalty. run --ab verifies that the batched path is bit-identical to the per-token one.

HTTP API and WebUI

./zig-out/bin/anedvd serve --model models/qwen2.5-0.5b-q8_0.gguf --port 8080
# → ANEDVD server listening on http://127.0.0.1:8080

Open http://127.0.0.1:8080/ for the built-in chat UI (single self-contained HTML file, no CDN, no build step, streaming replies, stop button, and settings for temperature, top-p, repetition penalty, max tokens and a system prompt — all persisted in localStorage).

Both API dialects are served from the same endpoint set, so existing clients work by pointing their base URL at http://127.0.0.1:8080/v1:

Endpoint	Dialect
POST /v1/chat/completions	OpenAI (streaming and non-streaming)
POST /v1/completions	OpenAI legacy
POST /v1/messages	Anthropic Messages (streaming and non-streaming)
GET /v1/models	OpenAI model list
GET /health	liveness probe
GET /	WebUI

$ curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":24}'
{"id":"chatcmpl-1791404097","object":"chat.completion","created":1791404097,
 "model":"smollm2-135m-q8_0","choices":[{"index":0,"message":{"role":"assistant",
 "content":"The capital of France is Paris. Paris is a city located in the northern part of the country, and it is known"},
 "finish_reason":"length"}],"usage":{"prompt_tokens":16,"completion_tokens":24,"total_tokens":40}}

$ curl -sN http://127.0.0.1:8080/v1/messages -H 'Content-Type: application/json' \
    -d '{"max_tokens":12,"stream":true,"messages":[{"role":"user","content":"Say hi."}]}'
event: message_start
data: {"type":"message_start","message":{"id":"msg_1791404130",...}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

What is implemented:

• SSE streaming with the exact chunk shapes each SDK expects (chat.completion.chunk + data: [DONE], and Anthropic's message_start / content_block_delta / message_delta / message_stop).
• stop sequences (string or array) with holdback, so a stop string is never emitted partially.
• usage / stop_reason reporting (length ↔ max_tokens, stop ↔ end_turn).
• Message content as a plain string or as an array of {type:"text"} blocks.
• Sampling controls: temperature, top_p, top_k, repetition_penalty (llama.cpp semantics), presence_penalty and frequency_penalty (OpenAI semantics). The Anthropic route accepts the first four.
• CORS headers, so browser-based tools can call it directly.

Known limitations:

• One request at a time. There is a single ANE engine and a single KV cache, so a second client waits for the first to finish rather than corrupting it. A concurrent request does not error, it just queues behind the socket.
• No tokeniser-level logprobs, n>1, tool/function calling, embeddings or /v1/images. Unsupported fields are ignored rather than rejected.
• The chat template is ChatML when the vocabulary has <|im_start|> and a plain User:/Assistant: transcript otherwise; a model's own Jinja template from GGUF metadata is not evaluated.

Long context

Two things dominate decode as the context grows, and both were measured rather than guessed (anedvd attnbench and the per-node ANE timing run prints):

• CPU attention. At 1024 tokens it was 48% of decode time, and 66% of that was the QK dot products: a plain dot += a*b reduction serialises on the add dependency chain because Zig/LLVM will not reassociate floats. The kernel now uses explicit 8-lane @Vector accumulators for the dots and keeps the value accumulation in registers. 967 -> 234 us per layer at ctx 1024 (4.1x).
• KV cache traffic. The cache is fp16 (llama.cpp's default), which halves both the resident memory and the bytes attention reads.

Decode on Qwen2.5-0.5B, fused FFN, ReleaseFast:

context	tok/s
16	30.2
128	28.6
512	26.8
1024	19.4

Those are single numbers, and this machine is noisy. Re-measuring the 16-token row with --repeat 3 on the current build gives a mean of 30.4 — consistent — but a range of 23.0 to 35.1. Treat the table as the shape of the curve rather than as a target, and use --repeat for any comparison you intend to act on.

Multi-turn chat does not re-prefill what is already cached: a transcript only appends, so generate.Session keeps the KV cache and prefills just the new suffix (anedvd chat and the server log both report how many prompt tokens were reused). In a two-turn example, turn 2 reported "prompt 44 (22 reused)".

--max-seq N sets the context (default 2048) on run, chat, serve, verify and layers, and it is the flag every truncation message tells you to raise. A prompt longer than the context is cut from the front — the newest tokens are kept — and that is reported rather than silent: usage.prompt_tokens_dropped and a log line on the server, and a note from chat and run. The KV cache is sized to max_seq rows, so raising it costs memory in both the engine and the CPU reference.

Sampling

The sampler is shared by the CLI, the HTTP API and the WebUI. It applies the penalties first, then keeps the candidates within max_logit - 20 * temperature (anything below contributes less than e^-20 and cannot be drawn), keeps the k largest of those, and truncates by top_p before drawing.

Two passes over the vocabulary plus a bounded selection replace the naive O(vocabulary × k) scan, which matters at Qwen2.5's 151 936 tokens. The numbers, measured rather than estimated:

• the two full-vocabulary walks cost 0.13 ms/token — that is the floor
• at temperature 1.0 the cut leaves ~45 000 candidates, and processing them was 1.33 ms/token because the selection sorted all of them to keep 40
• keeping the k largest and leaving the rest unordered took that to 0.47 ms/token (2.8×), and the cost tracks the candidate count, not the vocabulary: at temperature 0.3 the cut leaves 33 and sampling is 0.13 ms.

--temp 0 is greedy. For small models a repetition penalty is the single biggest quality lever — the same prompt with --temp 0.7 --top-p 0.9 --repeat-penalty 1.3 produces structured prose where greedy decoding loops.

anedvd chat is a small REPL over the same session:

$ ./zig-out/bin/anedvd chat --model models/qwen2.5-0.5b-q8_0.gguf
ready. /help for commands, /exit to quit.

you> What is the capital of France?

The capital of France is Paris.
[7 tokens, 23.4 tok/s, stop]

you> And what is it famous for? Answer in one sentence.

Paris, the capital city of France, holds significant cultural and historical
importance due to its iconic landmarks such as Notre-Dame Cathedral and Louvre
Museum among others...

How it works

            ┌─────────────────────── Zig ───────────────────────┐
GGUF ──► load_gguf ──► ModelWeights ──► engine.zig ──► tokens
            │  (dequant, fp16)             │
            │                              ├── mil.zig      MIL text
            │                              ├── weights.zig  ANE weight blobs
            │                              └── ane/runtime.zig  kernels + planar I/O
            └──────────────────────────────┴──► ane/shim.m (ObjC) ──► _ANEInMemoryModel

The private API path (no CoreML)

dlopen("/System/Library/PrivateFrameworks/AppleNeuralEngine.framework/AppleNeuralEngine")
NSClassFromString  →  _ANEInMemoryModelDescriptor, _ANEInMemoryModel,
                      _ANERequest, _ANEIOSurfaceObject
desc  = [Desc modelWithMILText:milData weights:wdict optionsPlist:nil]
model = [IMM inMemoryModelWithDescriptor:desc]
hex   = [model hexStringIdentifier]          // compiler reads $TMPDIR/<hex>/
write $TMPDIR/<hex>/model.mil and weights/wN.bin
[model compileWithQoS:21 options:@{} error:&e]     // weights are baked in here
[model loadWithQoS:21 options:@{} error:&e]
surfaces = IOSurfaceCreate(...) per MIL input/output
req = [_ANERequest requestWithInputs:inputIndices:outputs:outputIndices:
                             weightsBuffer:nil perfStats:nil procedureIndex:@0]
[model evaluateWithQoS:21 options:@{} request:req error:&e]
[model unloadWithQoS:21 error:&e]

Everything in that list is implemented in src/ane/shim.m; the rest of the project only sees a small C ABI.

Work split: ANE for projections, CPU for the rest

The ANE executes every linear projection as a 1×1 convolution with the weights baked into the compiled program. Normalisation, RoPE, attention (with the KV cache), the SwiGLU activation, residual adds and sampling run on the CPU.

That split is not laziness — the ANE's MIL dialect rejects rsqrt, reduce_mean, concat and gelu, and SDPA causal masks are silently ignored, so RMSNorm/attention cannot be expressed portably. The projections are >95% of the FLOPs, so the ANE still does the heavy lifting (82% of wall-clock in the run above).

Per layer: qkv (fused Wq|Wk|Wv), o, and one fused FFN kernel (gate→conv, up→conv, sigmoid, mul, mul, down→conv), plus one lm_head kernel — 3 kernels per layer, which measured ~45% faster decode than keeping the SiLU on the CPU and running gate_up/down as two kernels. --split restores the two-kernel path. Kernel launch overhead is ~90–110 µs, which is why the projections are fused as much as the data dependencies allow, and why prompt tokens are batched into one pass instead of one pass per token.

Build with zig build for a ReleaseFast binary (the default). The Zig side does real work — attention, batching, GGUF dequantisation — and Debug is roughly 8× slower there; zig build -Doptimize=debug is still available while developing.

ANE tensor layout (verified, not guessed)

Tensors are planar: channel c lives at base + c * PlaneStride, with

RowStride   = max(64, Width * elem_size)      // 64-byte minimum
PlaneStride = RowStride * Height
BatchStride = PlaneStride * Channels

For decode (Width == 1) every channel still occupies its own 64-byte plane, so a [1, 896, 1, 1] activation uses 57 KB instead of 1.8 KB. The authoritative numbers are read back from the loaded model (modelAttributes → NetworkStatusList[0] → LiveInputList/LiveOutputList) and src/ane/runtime.zig scatters/gathers against them — never against an assumed layout. IOSurfaces are allocated at max(declared, 49152) bytes; oversizing is harmless because the ANE keeps using its declared strides.

ANE weight blobs

[0,64)      file header        0x01 at 0, 0x02 at 4
per tensor:
  [H,H+64)  chunk header       0xEF 0xBE 0xAD 0xDE at H, version 1 at H+4,
                               u32 payload size at H+8, u32 data offset at H+16
  [H+64,..) fp16 payload       row-major [out_features][in_features]

The MIL BLOBFILE(offset = …) value is the payload's absolute file offset minus 64. Checked against the published fused_ffn.mil fixture (offsets 64 and 176 for a 48-byte first payload) and in src/ane/weights.zig's tests.

GGUF stores a linear layer as ne = [in, out] with the payload already in [out][in] C order — exactly this layout, so no transpose is needed.

Unified memory: what is shared, what is copied

CPU, GPU and ANE share one physical DRAM pool on Apple Silicon, and this project uses that where the ANE lets it:

• Activations are true zero-copy. Kernels exchange tensors through IOSurfaces, which are unified memory: the CPU writes a vector into the surface the ANE reads, with no copy in between (src/ane/runtime.zig).
• The token embedding can be zero-copy. For an fp16 GGUF the embedding is used straight out of the memory-mapped file — no dequantisation, no second copy (embed_owned = false in src/model.zig). A Q8_0 file has to be converted, so that one is allocated.
• Weights are streamed, not staged. Each layer's matrices are loaded, written into the ANE weight file, baked into the compiled program, and then freed before the next layer starts. The full model is never resident as fp16.
• The weight files are written straight from the caller's buffers in 64-byte headers plus payloads (write_weight_file in src/ane/shim.m), so no blob is materialised in Zig, and the compiler's weight dictionary carries only a SHA-256 digest rather than another copy of every tensor.
• Those files are deleted as soon as the model is loaded. They are inputs to ANECCompile(); the compiled program lives in the ANE daemon's own cache. Keeping them cost ~600 MB of disk per model.

Measured peak RSS, same machine, Qwen2.5-0.5B-Instruct Q8_0:

	peak RSS
eager load (whole model as fp16, all blobs in RAM)	1.39 GB
+ blob/NSData copies removed	1.78 GB → 965 MB
streaming + zero-copy + digest (current)	~0.72 GB

SmolLM2-135M for scale: 299 MB (Q8_0), 361 MB (F16, where the larger mmap accounts for the difference — its embedding is not allocated at all).

Two things are not shared, and they are the reason a bigger model still needs quantisation rather than unified memory:

1. The ANE bakes weights into the compiled program. A 0.5B model at fp16 is ~1 GB inside the ANE's own memory, on top of whatever the CPU side holds. int8 would halve that and halve the per-token bandwidth (the current limiting factor: ~20-40 GB/s measured).
2. The ANE program pool is shared machine-wide, not per-process. A running anedvd serve holds its kernels loaded; a second process that tries to load a full model gets no ANE resources (transient; retry) (status 0x5). The shim retries with exponential backoff (100 ms … 3.2 s), but the real answer is one engine per machine at a time.

Findings worth knowing (all reproduced locally)

1. The ANE is weight-bandwidth bound, not compute bound. A 2048×2048 1x1 conv takes ~290 µs whether it processes 1 column or 128, so the same kernel serves decode and 128-token prefill. Measured per-kernel bandwidth on real models ranges 5–40 GB/s of fp16 weights, with the big kernels at ~30-40 GB/s. Every design decision here follows from that (fused projections, prompt batching, and the fact that int8 would be the only way to go much faster).
2. No entitlement, no SIP changes, no signing. The ANE is reached over XPC through aned; the kernel-level com.apple.ane.iokit-user-access entitlement only gates the in-process H11ANEIn user client, which this project does not use. Codesigning with ANE entitlements while SIP is enabled gets the process SIGKILLed at launch (exit 137) — don't.
3. matmul is not usable. An ANE program containing MIL matmul compiles but fails at evaluation with kIOReturnUnsupported (0x1d). Every matrix product is a 1×1 conv.
4. sigmoid and mul do compile (used by the experimental fused FFN), but see the known issue below.
5. int8 weights are not reachable. constexpr_affine_dequantize is rejected with InvalidMILProgram for every layout and axis tried — including with an fp16 input, so the op itself is unavailable in this MIL dialect, not just the int8 dtype. That matches Orion ("quantization not yet supported") and Espresso ("INT8/quantized weights: unsupported"). See probe/ane_int8_probe.m to reproduce. Weight bandwidth therefore stays at fp16: ~20-40 GB/s on the larger kernels.
6. Compile budget: 121 ANECCompile() calls in one process worked here. Published reports put the daemon's limit near 119 on M4/macOS 15, so the engine keeps the per-layer kernel count low and reports ane.compileCount().
7. The RoPE convention is architecture-dependent — and getting it wrong does not produce garbage, it produces fluent repetition. convert_hf_to_gguf.py permutes Q/K rows of Llama-family models into llama.cpp's adjacent-pair layout (LLAMA_ROPE_TYPE_NORM), while Qwen2 (and other rotate_half architectures) keep the HF half-split layout (NEOX). Verified here: SmolLM2-135M needs adjacent, Qwen2.5-0.5B needs half-split — with the wrong one Qwen still emits grammatical English, just stuck in a loop. The architecture table lives in src/load_gguf.zig (ropeIsAdjacent), and --rope-hf / --rope-adjacent override it.
8. Qwen2 adds biases to Q/K/V (attn_q.bias …); Llama does not. Missing them yields multilingual garbage rather than an error. They are applied after the ANE projection (a vector add is not worth a kernel).
9. The ANE program pool is global. Two processes cannot each hold a full model's kernels; the second gets a transient "no ANE resources" error. See the unified-memory section above.
10. Weight bandwidth is the decode ceiling: Qwen2.5-0.5B reads ~1 GB of fp16 weights per token, which is ~50 ms at the ~20-40 GB/s the ANE delivers. Quantised weights would be the only large win left, and int8 is not reachable (finding 5), so decode speed is now bounded by the hardware.
11. The ANE is deterministic: the CPU reference in anedvd cpu reproduces the ANE's generated text token-for-token on both models.

Known issues

• A prompt longer than the context window is truncated from the front, keeping the newest tokens. This is now reported (a log warning, prompt_tokens_dropped in the OpenAI usage object, and a status line in the WebUI) rather than answered silently, but the tokens are still gone.

• MoE models work, but large ones are slow. Qwen2MoE runs correctly end to end: Qwen1.5-MoE-A2.7B (9.5 GB, 60 experts, top-4) answers "The capital of France is" -> "Paris" with attention and the shared expert on the ANE and the routed experts on the CPU. It measures ~35 s/token here, 89% of that the expert matmuls, because each token reads its experts out of the mapping and multiplies them in scalar code. research/moe-design.md has the arithmetic, the measured ceilings and the levers that would move it.

• Not all architectures are supported, and the two lists below are different. Run on a real model here (whole-model ANE-vs-CPU comparison passing, and coherent text): Llama (SmolLM2 in both GGUF and HF form, TinyLlama), Qwen2 (0.5B, 1.5B), Qwen3-0.6B, Qwen2-MoE (Qwen1.5-MoE-A2.7B and a tiny MoE) and Gemma 2 (which needs its (1 + w) RMSNorm, sqrt(hidden) embedding scale, tanh-GELU FFN, both logit soft-caps, alternating sliding-window attention and the "sandwich" norms — all of them implemented).

  Gemma 4 is in a category of its own, so it is stated separately. Its text stack works: anedvd cpu --model gemma-4-E2B-q4_0.gguf --prompt "The capital of France is" answers Paris., and "The capital of Japan is" answers Tokyo. It needs five things no other architecture here does — a per-layer head dimension (256 sliding, 512 global), per-layer FFN widths (6144 or 12288, use_double_wide_mlp), an explicit per-layer sliding list, K/V shared by the last 20 of its 35 layers, and per-layer input embeddings (PLE) — plus a nested text_config.

  But only the CPU path is verified. check, verify and run build ANE kernels and were not run against it, so whether the ANE kernels agree with the reference is untested. And E4B has not been run at all: the CPU reference loads every layer eagerly and E4B's layers are 10240 or 20480 wide, which is 8.6 GB against this machine's 8 GB.

  Recognised but not run here: Mistral, SmolLM3 and Gemma 3. Their configuration is parsed and unit-tested — Mistral's window applies to every layer, Gemma 2's alternates 1:1, and Gemma 3 repeats over six, all three now read from the config rather than assumed — but no checkpoint for any of them was available on this machine to run end to end, so treat their status as "implemented, unproven". (This list said "verified end to end" until a review: Mistral's window had in fact been ignored entirely, and the config tests that now cover it are the fix.)

  Anything with a different attention layout (MLA, other sliding-window variants) needs work. An unrecognised general.architecture is accepted with a warning, since the GGUF tensor names are the same — but check the output.

• A model's norm convention can depend on the file, not the architecture. Gemma's RMSNorm is x * (1 + w). An HF checkpoint stores the raw parameter and needs the offset added; the GGUF converter has already applied it (measured: blk.0.attn_norm.weight has mean 1.1927, the effective factor, where the raw parameter would be ~0.19). Applying it twice makes the model emit a run of dots instead of an answer.

• Sampling with top_p on a 151936-token vocabulary costs ~1.2 ms/token at temperature 1.0, about 10% of decode. Measured, not assumed: docs/RESULTS.md.

• Decode is at the hardware's weight-bandwidth ceiling (~20–40 GB/s of fp16 weights measured on this ANE). int8 would be the only large win and the ANE rejects the MIL op for it, so decode speed is bounded, not untuned.

• The HTTP server serialises requests. One ANE engine, one KV cache. Cheap GETs (/health, /v1/models) are answered mid-generation, but a second completion is refused with 503 Service Unavailable and Retry-After: 1 rather than queued, so a client can tell "busy" from "broken". This holds during a long prefill too, not just between decode tokens.

• No tokeniser-level logprobs, n>1, tool calling, embeddings or images.

• The chat template is selected by probing for ChatML markers; a model's own Jinja template from GGUF metadata is not evaluated.

• probe/*.m establishes the private-API facts but is not part of the build.

Layout

Path	Role
src/ane/shim.m	the only Objective-C / private-API code: compile, load, evaluate, IOSurface I/O
src/ane/runtime.zig	Zig kernel wrapper, planar fp16 scatter/gather
src/ane/mil.zig	MIL program generator (conv, add, sigmoid, mul)
src/ane/weights.zig	ANE weight-blob packer
src/engine.zig	transformer decode loop, kernel construction
src/cpu.zig	RMSNorm, RoPE (both conventions), attention, SwiGLU, sampling
src/gguf.zig	GGUF v2/v3 reader + dequantisation (15 ggml types, bit-exact vs ggml)
src/load_gguf.zig	GGUF → runtime weights + per-layer matrices
src/load_hf.zig, src/model_open.zig	HF safetensors → the same, plus format detection for --model
src/tokenizer.zig	byte-level BPE from GGUF vocab or HF tokenizer.json
src/safetensors.zig, src/hf.zig	HF checkpoint + config.json readers (not yet wired)
docs/RESULTS.md	measurements, accuracy tables, and the bugs found
probe/	the standalone Objective-C probes used to establish the API facts
research/	extracted constraints from the Orion paper, Espresso, and the entitlement analysis

Prior art and credits

ANEDVD is an independent Zig implementation, but it stands on published reverse-engineering work. These sources established the API path, the weight format and the MIL constraints that this project then reproduced and extended on A18 Pro / macOS 27:

• maderix/ANE — the _ANEInMemoryModel in-memory compile/load/evaluate recipe, the ANE weight-blob header layout and the IOSurface I/O pattern. src/ane/shim.m follows that sequence.
• Orion (Kumaresan, 2026; mechramc/Orion) — the catalogue of ANE restrictions on MIL programs, memory layout and compilation limits, including the BLOBFILE-slot budget and the ~119-compiles-per-process ceiling. The extracted notes are in research/orion-constraints.md.
• Espresso — a public reference for the private API call sequence and for real MIL fixtures (conv.mil, fused_ffn.mil); see research/espresso-ane-api.md.
• hollance/neural-engine — background on what the ANE exposes and why CoreML was the only sanctioned path.
• llama.cpp / ggml — the GGUF container, the block-quantisation formats (this project's dequantisers are checked bit-for-bit against ggml-quants.c) and the RoPE conventions that explain the per-architecture permutation.
• Hugging Face — tokenizer.json, config.json and safetensors formats, and the published models used for testing.

Quoted material in research/ keeps its original attribution; the Orion paper is CC BY 4.0. Apple's private frameworks are undocumented and unsupported: this project is for research and cannot be shipped on the App Store.

License

MIT — see LICENSE. Third-party material quoted in research/ keeps its original terms (the Orion paper is CC BY 4.0).

Disclaimer

This project uses AI-generated code (LLM). I do my best to review and test it, but I can't guarantee it's perfect. Please use it at your own risk.