review: the complete v0.2.10 line — agent tools, local chat, Flash default

[rejojer]

PageIndex SDK 0.2.10 adds two things:

1. Agent tools — plug PageIndex document retrieval into any agent framework with one call.
2. Local chat — ask questions about your documents: client.chat() for the answer, or three standard chat APIs for the full envelopes.

Both work in local mode (runs on your machine, with your model keys) and cloud mode (runs on api.pageindex.ai, with a PageIndex API key).

(Review note: this PR holds the complete 0.2.10 diff for review. The code is already on main via #396, #402, #404, #405, #406, #409, and #410. This PR is not meant to be merged; the tools layer's own review record is #393.)

Install

pip install pageindex==0.2.10.dev5                   # runs everything in this guide

pip install "pageindex[anthropic]==0.2.10.dev5"      # + Anthropic SDK (tool runner, messages())
pip install "pageindex[claude]==0.2.10.dev5"         # + Claude Agent SDK

0.2.10 is a pre-release — plain pip install pageindex still resolves 0.2.8, so pin the version. An extra only adds a vendor's own SDK surface; everything else ships with the base install.

Quick start

import os
from pageindex import PageIndexClient

os.environ["OPENAI_API_KEY"] = "your-openai-key"

client = PageIndexClient()                            # local by default; api_key="..." switches to cloud
doc = client.submit_document("report.pdf", wait=True)

answer = client.chat("What was Q3 revenue?", doc_id=doc["doc_id"])
print(answer)

One rule for keys: the key belongs to whatever model does the thinking — at indexing time the summary model, at chat time the agent's model. The navigation tools themselves make no LLM calls. A missing key fails fast, naming the variable to set; cloud mode needs only api_key from dash.pageindex.ai, no model keys. (PageIndexLocalClient / PageIndexCloudClient pin the mode explicitly instead of inferring it.)

Two model knobs: index_model (indexing — structure and summaries, default gpt-5.6-luna) and chat_model (every chat door, default gpt-5.6-sol), settable as constructor kwargs or in config.yaml. model sets both at once, and the released role names (summary_model, retrieve_model) stay accepted — new names win over old, specific over general. Chat-lane model names are LiteLLM's grammar, verbatim: bare names are OpenAI-compatible shorthand (OPENAI_BASE_URL picks the server), provider/model reaches that provider, and no prefix triggers a side lane.

Local indexing defaults to PageIndex Flash — the tree comes from the PDF's layout in seconds, the LLM only writes summaries. mode="standard" for the fully LLM-built tree. CLI: python run_pageindex.py --pdf_path doc.pdf. Documents persist under ./.pageindex — submit once, then reuse the doc_id (client.list_documents() shows what is stored).

Chat with your documents

chat() — question in, answer out

chat() returns just the answer; the agent loop — tree navigation, page reads — runs inside. It is stateless: you keep the history.

answer = client.chat("What was Q3 revenue?", doc_id=doc["doc_id"])

# Multi-turn — pass your own role/content history, keep doc_id the same
answer2 = client.chat(
    [{"role": "user", "content": "What was Q3 revenue?"},
     {"role": "assistant", "content": answer},
     {"role": "user", "content": "And how did it compare to last year?"}],
    doc_id=doc["doc_id"],
)

# Streaming — text chunks
for chunk in client.chat("Summarize the risk factors.", doc_id=doc["doc_id"], stream=True):
    print(chunk, end="")

# Any model — LiteLLM-style name, with that provider's key set
client.chat("What was Q3 revenue?", doc_id=doc["doc_id"], model="anthropic/claude-sonnet-4-6")

# How hard it thinks — unset leaves each model's own default behavior
client.chat("Reconcile the two revenue tables.", doc_id=doc["doc_id"], reasoning_effort="high")

chat()'s knobs stay business-level on purpose — who answers (model), about what (doc_id), how hard it thinks (reasoning_effort). Sampling and wire-level controls live on the protocol surfaces.

Three standard chat APIs

chat() is sugar over chat_completions(). When you need the envelope — usage accounting, streaming metadata, the tool-use process — call a protocol surface. Each speaks one standard wire format, so request and response look exactly like the API you already know; pass a plain string or full messages in that protocol's format.

# OpenAI Chat Completions — works on any OpenAI-compatible backend
r = client.chat_completions("What was Q3 revenue?", doc_id=doc["doc_id"])
print(r["choices"][0]["message"]["content"])

# OpenAI Responses — the full agentic transcript, built to round-trip
r = client.responses("Which section covers risk factors?", doc_id=doc["doc_id"])
print(r["output"][-1]["content"][0]["text"])

# Anthropic Messages — pip install "pageindex[anthropic]", needs ANTHROPIC_API_KEY
r = client.messages("What was Q3 revenue?", doc_id=doc["doc_id"], model="claude-sonnet-4-6")
print(r["content"][-1]["text"])

All three stream with stream=True and do multi-turn the protocol's own way: append the previous reply to your next call's messages. Provider prompt caching keeps working across turns — Claude models (Anthropic direct, Bedrock, Vertex) get the managed prefix cache-marked automatically. doc_id targeting is enforced by the tools, not just suggested to the model.

Tuning rides each protocol's own vocabulary, forwarded verbatim with no defaults of ours: reasoning via chat_completions(reasoning_effort=) / responses(reasoning={}) / messages(thinking={}); sampling and output caps via temperature, top_p, and max_tokens (max_output_tokens on responses()) — the caps bound each backend call in the agent loop, the way max_turns bounds the loop. Fields a method doesn't name go through extra_body, merged into the request last so caller keys win.

Connection config follows the same doctrine: index_backend / chat_backend on the constructor (plus per-call backend on the protocol doors, per-call keys winning) carry each lane's own connection vocabulary verbatim — LiteLLM params on the indexing lane and chat_completions(), openai SDK client params on responses(), anthropic SDK client params on messages() — so two clients can point at two providers without environment juggling. Credentials belong there, never in extra_body. extra_headers on all three doors sends raw HTTP headers (beta flags and the like); the one wire-probed exception is documented: LiteLLM's anthropic adapter owns anthropic-beta, so Anthropic beta flags belong on messages().

Bring your own agent framework

One call returns everything the framework needs — instructions plus tools. Local and cloud clients work identically. Agent frameworks are async-native, so the snippets assume an async context, with client and doc from the quick start.

OpenAI Agents SDK — ships with the SDK

from agents import Agent, Runner

agent = Agent(**client.openai_agent_config())
result = await Runner.run(agent, "What was total revenue this quarter, vs last year?")
print(result.final_output)

More configuration options

agent = Agent(
    name="PageIndex",
    instructions=client.agent_instructions(),   # doc_id targeting goes here
    tools=client.as_openai_tools(),             # include_management=True adds deletion
    model=client.chat_model,                    # local clients only — cloud omits it
)

Anthropic SDK tool runner — pip install "pageindex[anthropic]"

import anthropic

runner = anthropic.AsyncAnthropic().beta.messages.tool_runner(
    **client.anthropic_runner_config(model="claude-sonnet-4-6", asynchronous=True),
    messages=[{"role": "user", "content": "What was total revenue this quarter?"}],
)
final = await runner.until_done()
print(final.content[-1].text)

More configuration options

runner = anthropic.AsyncAnthropic().beta.messages.tool_runner(
    model="claude-sonnet-4-6",
    max_tokens=8192,                            # default resolved per model
    system=client.agent_instructions(),
    tools=client.as_anthropic_tools(asynchronous=True),
    max_iterations=10,
    messages=[{"role": "user", "content": "What was total revenue this quarter?"}],
)

Claude Agent SDK — pip install "pageindex[claude]"

from claude_agent_sdk import ClaudeAgentOptions, ResultMessage, query

options = ClaudeAgentOptions(**client.claude_agent_config())
async for message in query(prompt="What was total revenue this quarter?", options=options):
    if isinstance(message, ResultMessage):
        print(message.result)

More configuration options

options = ClaudeAgentOptions(
    system_prompt=client.agent_instructions(),
    mcp_servers={"pageindex": client.as_claude_mcp()},
    allowed_tools=["mcp__pageindex"],           # pre-approval; the server itself is gated
)

Any other framework — no extras needed

tools = client.agent_tools()   # plain functions returning JSON envelopes

Drop to the explicit calls to customize. All of these accept doc_id=... to point the agent at specific documents, and include_management=True to also expose document deletion (off by default).

What works where

Bring your own agent — the tools, on every major surface:

Surface	Local	Cloud
agent_tools() — plain functions, any framework	✅ in-process tools	✅ live cloud tool set over MCP
as_openai_tools() / openai_agent_config() — OpenAI Agents SDK	✅	✅ (hosted=True: execution on OpenAI's side, read-only endpoint by default)
as_anthropic_tools() / anthropic_runner_config() — Anthropic SDK tool runner	✅ sync & async	✅ sync & async
as_claude_mcp() / claude_agent_config() — Claude Agent SDK / Claude Code	✅ in-process MCP server	✅ remote MCP config — read-only endpoint by default
Standard MCP, no SDK involved	⬜ stdio entry point (follow-up)	✅ api.pageindex.ai/mcp (read-only: …/mcp?tools=read) — any MCP host, the Anthropic MCP connector, OpenAI hosted MCP

Managed chat — the SDK runs the loop:

Method	Wire format	Engine	Local	Cloud
chat()	answer string out — sugar over chat_completions()	openai-agents	✅	✅ hosted endpoint
chat_completions()	OpenAI chatcmpl	openai-agents	✅	✅ hosted endpoint
responses()	OpenAI Responses	openai-agents	✅	⬜ raises — cloud converges toward this later
messages()	Anthropic Messages	anthropic tool_runner	✅	⬜ raises

Local serves four read-only tools: browse_documents, get_document, get_document_structure, get_page_content (remove_document only with include_management=True). Cloud adds search_documents, folders, and get_document_image — discovered live from the server, never frozen into the SDK.

What a run looks like

An actual run (local mode, OpenAI Agents SDK, over examples/documents/q1-fy25-earnings.pdf):

Q: What was Disney's total revenue in Q1 FY2025, and how did it compare to the prior-year quarter? Cite the page you found it on.

[tool] get_document_structure({"doc_name": "q1-fy25-earnings.pdf", "part": 1})
[tool] get_page_content({"doc_name": "q1-fy25-earnings.pdf", "pages": "1,3"})

A: Disney's total revenue in Q1 FY2025 was $24.7 billion, up 5% from $23.5 billion in the prior-year quarter.
Citations — Page 1: "Revenues increased 5% for Q1 to $24.7 billion from $23.5 billion in Q1 fiscal 2024"; Page 3: table shows $24,690M vs $23,549M, +5%.

This is the intended loop: the agent reads the tree structure first, picks tight page ranges, and answers from tool output with page citations. No vector index, no chunking. The retrieval intelligence is your agent's own model — the navigation tools make no LLM calls.

---

Everything below is design rationale and the test record, written for reviewers. You don't need it to use the SDK.

Design — the tools layer

• The tool surface is the cloud MCP contract: browse_documents / get_document / get_document_structure / get_page_content, doc_name-addressed, same input schemas, descriptions, and JSON response envelopes as the hosted MCP server's tools/list — agent prompts port unchanged between the cloud MCP connection and these in-process tools. Adapters hand the contract/server schema to the framework verbatim (FunctionTool(params_json_schema=…), beta_tool(input_schema=…)) — no regeneration from Python signatures, so items/enum/pattern/bounds survive on every surface. tests/data/cloud_mcp_contract.json freezes the contract; a parity test guards drift.
• Local is an honest subset: tools that don't exist locally (folders, search_documents, get_document_image) are not registered, mirroring the server's gating semantics. Cloud-only parameters (folder_id, sort/query, recursive) are hidden from the local surface entirely — strict-schema frameworks then cannot express the dead-end calls, and the call_tool/MCP path still answers direct calls with a guided "works on PageIndex cloud" envelope as the backstop. Local descriptions and instructions teach only that surface: the exposed schema is the contract minus the documented hidden set (mechanically asserted), and a dead-reference test keeps local guidance from naming cloud-only tools. remove_document is off by default, behind include_management=True.
• The management gate is structural wherever possible. The config-handoff surfaces — as_claude_mcp(), as_openai_tools(hosted=True), the raw connector URL — point at the server's read-only endpoint (/mcp?tools=read) by default, so the URL itself is the gate and works identically in every MCP client. The in-process surfaces (agent_tools(), as_openai_tools(), as_anthropic_tools()) expose only tools the server marks readOnlyHint; local withholds remove_document at registration. include_management=True is the one switch that opens the complete list in either mode, on every surface. All four tool exports stay polymorphic: on a cloud client the live tool set — including new server-side tools — arrives without an SDK release.
• Tools never raise, and failures carry the protocol's own error marking — every failure returns the same {"error", "errorCode", "next_steps"} envelope the cloud emits, flagged through each channel that has one (MCP isError propagated, Anthropic tool runner is_error: true via ToolError), so the model can always tell a failed call from data. Destructive calls validate every argument before acting — a rejection envelope means nothing was deleted.
• agent_instructions(doc_id=None) supplies the retrieval playbook for the agent's system prompt. Cloud: the live instructions the MCP server serves for the key's tool set, captured from the initialize handshake over the same bridge session — server-side guidance updates arrive without an SDK release, and an empty server response raises instead of silently substituting. Local: the built-in playbook for the in-process tools, a trimmed subset with a consistency test that every tool it names exists locally. doc_id (str or list) appends the target documents — in the run above it is what let the agent skip discovery.
• One new base dependency — openai-agents, the chat engine (chat() is the front door; ~15 MB on a base tree already carrying litellm + openai; the >=0.18.1 floor is the live-probed minimum that works with current openai, and the latest release passes the full suite). claude-agent-sdk / anthropic stay call-time imports behind vendor extras with actionable errors; the [openai] extra remains declared but empty so existing install commands keep resolving. The litellm floor rises to 1.97.0 — the release whose bridge routes sol-class chatcmpl+tools calls onto /v1/responses automatically (A/B-verified against 1.96.2), so the default chat model works out of the box. That release also ships a Python 3.10 defect — Message/Delta annotations whose nested forward refs don't resolve, killing every completion() (upstream [Bug]: pydantic.errors.PydanticUserError: Message is not fully defined BerriAI/litellm#36384) — repaired by a version-gated, best-effort model rebuild at our completion gateways that no-ops on 3.11+ and on fixed releases.
• submit_document(wait=True) polls with growing intervals; returns on completed, raises on failed or after 30 minutes — the manual polling loop cloud callers write today spins forever on a failed document.
• Local indexing defaults to Flash with the full optimize pass (deterministic merge, then LLM expand; summaries and expand share summary_model). page_index_flash() takes optimize="full" (default) / "merge" / False; True is accepted as "full" for backward compatibility, unknown values raise instead of silently degrading. The CLI's --mode {flash,standard} replaces --flash (kept as a hidden compatibility alias); standard-only tuning flags now error in flash mode instead of being silently ignored; the missing-key pre-check (litellm.validate_environment, all providers) runs only when an LLM will actually be called. Both modes emit identical output schemas end to end.

Design — chat on the tools

• Basis is industry standards, not the cloud chat endpoint. The cloud /chat/completions quirks (history flattening, bespoke prompt, stateless re-reading, arbitrary caps) are not mirrored; local targets the standard formats and becomes the reference the cloud can later converge toward. responses()/messages() raise on cloud clients until then.
• Passthrough doctrine. Content is never rewritten — the caller's messages, the model's answers, tool outputs, native finish/stop reasons. The SDK owns exactly four things: gatekeeping (structural validation only — no message caps, sampling params pass through), table-setting (thin chat header + the local AGENT_INSTRUCTIONS; caller system content appended, not rejected; the doc_id targeting block leads the conversation and the tool layer enforces it), tool execution (read-only local set, scoped to doc_id), and billing (usage aggregation, envelope ids). Per-run tracing is disabled; prompt-cache routing keys are per-conversation, never pooled across users.
• Engines are the vendors' own loops, never hand-rolled: openai-agents for the two OpenAI protocols, the Anthropic SDK's tool_runner for messages() (floor 0.108.0 — the first release whose runner stops at a refusal carrying a tool_use block instead of executing the tool; verified by probing mock transports against 0.84.0 through 0.108.0). The chat lane routes every model through LiteLLM — bare names are OpenAI-compatible shorthand (env config unchanged), and no prefix triggers a direct lane, so a routing decision can never hide in a model-name spelling; responses() stays on the OpenAI SDK natively (LiteLLM cannot speak the Responses wire format). Rule of the layer: engines = each vendor's official thin loop; agent hosts (Claude Code et al.) only ever get tools.
• Envelopes report what actually happened. responses() carries the backend's real terminal status/incomplete_details (recorded at the transport layer — the engine discards them), a partial page read names every omitted page, and framework exceptions surface as PageIndexAPIError, never as raw engine types.
• Prompt-cache continuity is a tested contract. Round-tripped history reaches the backend as an item-for-item extension of the previous call's final model input — asserted on both engines against captured payloads. Anthropic's explicit cache_control breakpoints sit on the managed system blocks only. The same decision reaches the LiteLLM lane: Claude models routed through LiteLLM — Anthropic direct, Bedrock, and Vertex, resolved by LiteLLM's own get_llm_provider — pass LiteLLM's cache_control_injection_points (via the Agents SDK's extra_args, both documented parameters), so the managed prefix (tools + instructions + doc block) caches there too. Each channel live-verified with a write→read cycle (anthropic per-turn reads through the full stack; Bedrock 7264 and Vertex 4842 tokens read on the second call).
• chat() is output sugar, not a fourth protocol. It returns the answer string and hides the envelope; the wire underneath is chat_completions() unchanged, so it works on every backend in both modes. Its contract is the one surface not pinned to a wire format — a future engine= selector is a non-breaking add — while a merged multi-protocol method stays rejected: round-trip formats are protocol-specific, so a switch parameter abstracts nothing.
• Model roles resolve in one seam. ConfigLoader.load() fills index/summary/chat from whichever names were given — new names win over old, specific over general, model sets every role, and the built-in defaults close each chain. The packaged config.yaml ships no model keys (comments only), so key presence is what distinguishes a user's choice from a default; every name that ever shipped in a release stays accepted, without deprecation warnings.
• Tuning params are protocol-native and verbatim. Each door speaks its own protocol's vocabulary (reasoning_effort / reasoning / thinking; max_tokens / max_output_tokens), values forwarded untranslated with no defaults of ours — capability rules are the backend's (LiteLLM refuses unsupported combinations loudly, wrapped as PageIndexAPIError). The named sampling knobs ride ModelSettings fields — the one channel clean on every lane — while extra_body covers fields the methods don't name: OpenAI destinations get a true body merge, LiteLLM providers take the keys as LiteLLM's own params, messages() hands them to the Anthropic SDK's native extra_body. Merged last, so caller keys win.
• Connection config is per-lane and verbatim. index_backend / chat_backend (and per-call backend) hand each stack its own connection params untranslated: the indexing gateways take the dict as LiteLLM call kwargs scoped by a contextvar (the env-key pre-check yields to it; the openai-SDK fast path normalizes api_base), the chat lane lifts api_key/base_url into LitellmModel's two pinned constructor slots and rides the rest as call kwargs, responses()/messages() construct their SDK clients from the dict (unknown keys raise, wrapped uniformly). Per-call wins over per-client; chat() takes no per-call backend but honors the client's; config bundles deliberately carry no credentials — they run in your environment.
• enable_citations raises as cloud-only (citations need block-level OCR data local mode does not store).
• messages() resolves its max_tokens default per model (8192, or 4096 for the claude-3 generation), so the simple call needs only a question on any model.

Verification

• 295 tests green (plus 3 skipped without the claude extra and 3 key-gated live tests): tool behavior against a seeded store with no LLM calls; the real chat engines against scripted backends (a Model fake under openai-agents, a mock HTTP transport under the real anthropic SDK); contract parity vs the frozen snapshot; framework-missing/-installed behavior both ways; streaming on every chat surface; the chat() front door (answer extraction, streamed chunks, multi-turn history passthrough, cloud envelope unwrap); the round-trip prefix-extension assertions on both engines; doc_id scoping, error-marking, and envelope-honesty regressions; a model-resolution matrix with one row per released knob generation; per-door passthrough delivery (reasoning channels, sampling knobs on ModelSettings, extra_body asserted on the captured anthropic wire body); backend delivery per lane (contextvar scoping on both gateway paths, the env-precheck yield, constructor lift and kwargs remainder on the LiteLLM lane, SDK-client construction on responses/messages, merge precedence) and extra_headers on every door (anthropic-beta asserted on the anthropic wire).
• Live against the real cloud MCP server: agent_tools() and as_anthropic_tools() discovered this key's gated tool set (7 read-only tools; include_management=True adds remove_document); frozen-contract parity letter-for-letter; envelope field parity on the analogous calls; the server serves non-empty initialize.instructions.
• Live against real model backends: OpenAI — chat_completions answered with the structure-first loop; responses round-trip answered the follow-up with zero new tool calls. Anthropic — messages() history was accepted verbatim by the real API, follow-up answered with zero new tool turns, cache_control hit live (cache_read_input_tokens: 1826), native streaming; both the sync and AsyncAnthropic tool runners drove the live cloud tools end-to-end; the Messages API MCP connector reached api.pageindex.ai/mcp server-side (mcp_tool_use/mcp_tool_result in a single call).
• Review record: multiple independent multi-agent review rounds ran over each layer (adversarial runtime probes, claims-vs-code, refactor-equivalence audits, best-practice review against the frameworks' source); every finding was reproduced before being fixed, and deliberate non-changes are documented alongside. The tools layer's rounds are recorded in feat: agent tools — OpenAI Agents SDK, Claude Agent SDK, and any framework, local & cloud #393; the chat and tools-export rounds in this PR's commit messages. A maximum-effort whole-PR review round then ran over this diff — 26 verified findings, 20 fixed in the first pass (d87fa89, b135711, eb1a230), the remainder triaged with rationale. Nine further review rounds followed (commit messages 31c9150 through eebed64), covering argument coercion, protocol terminal states, envelope honesty, provider error containment, CodeQL findings, and SDK dependency floors — each finding reproduced before the fix landed. The feat: model knobs, LiteLLM-verbatim chat lane, per-door passthrough params #409 wave (model knobs, routing flip, passthrough params) went through three more audits with wire-level probes: LiteLLM's max_tokens translation verified on both of its paths (chatcmpl max_completion_tokens, bridge max_output_tokens), extra_body delivery captured per lane, and the sol-bridge litellm floor bisected to the exact release by source and behavioral A/B. The backend wave (feat: backend connection overrides; extra_headers on every local door #411) ran three further audits backed by ~20 mock-server wire probes that mapped LiteLLM's channel semantics per adapter (kwargs vs extra_body vs headers), re-proved custom-header forwarding after catching a case-sensitivity bug in the probe itself, and isolated the anthropic-beta drop to LiteLLM's completion() plumbing — its transformation layer merges user betas when called directly.
• Post-feat: agent tools and local chat for the PageIndex SDK (v0.2.10) #396 fixes carried here (merged to main via fix: post-merge review fixes for v0.2.10 #402/feat: Flash with full optimization becomes the default local indexing mode #404): conformant responses() envelope — official output (model items only) + items (full transcript for round-trip) + usage aggregated across turns, verified against the real OpenAI API; python floor declared >=3.10; stale anthropic>=0.84.0 hints updated to the real 0.108.0 floor; the bridge's binary-stub behavior disclosed on the two image-advertising tool surfaces; _run_sync moved off the except RuntimeError probe so user exceptions stop carrying a phantom "no running event loop" context.

Release gate — satisfied: the default cloud configs point at the read-only MCP endpoint, so VectifyAI/pageindex-chat#448 had to be deployed before 0.2.10 ships (an older server ignores the tools=read parameter and would silently serve the full set behind a URL that promises read-only). Verified live before publishing 0.2.10.dev1: …/mcp?tools=read serves 7 tools without remove_document, …/mcp serves 8 with it.

Follow-ups (not in this PR): an AsyncPageIndexClient twin per the industry dual-client pattern — every layer around the SDK is already async-native (FastAPI server, agent engines, agent frameworks); the async chat path is the engines' native form (drops the sync bridge, streams pass through as async for), and cloud transport gains an httpx track; a stdio pageindex-mcp entry point for non-Python MCP hosts; a public doc_id scope on the BYO tool exports (the chat surfaces already enforce it); the docs-site agent-integration page; cloud /responses·/messages convergence toward these surfaces.

[rejojer]

Code review

Found 1 issue:

1. The pypdfium2 5.x shim picks the wrong half of a split API, silently changing font_name semantics for every Flash layout heuristic (bug due to pageindex/flash/parser_pdfium_charlevel/geometry.py:9 — _get_font_name = getattr(pdfium_c, "FPDFFont_GetBaseFontName", None) or pdfium_c.FPDFFont_GetFontName)

57a77f6 describes this as an availability fix ("pypdfium2 5.x removed FPDFFont_GetFontName (now GetBaseFontName)"), but pdfium did not rename that call — it split it. FPDFFont_GetFontName returned CFX_Font::GetFamilyName(); the replacement with those semantics is FPDFFont_GetFamilyName, while FPDFFont_GetBaseFontName returns the /BaseFont value. So font_name drops from family granularity to per-face granularity at geometry.py:133.

That string is compared for equality by the block-join, heading-detection, body-text and dominant-font paths — flash/blocks/join_rules.py:132, flash/heading_detection/style_detectors.py:248, flash/classification/body_text.py:150, flash/stats/aggregates.py:127,175. On examples/documents/attention-residuals.pdf the name set goes from 5 family names (Computer Modern, Nimbus Roman No9 L, …) to 35 /BaseFont names (CMR10, CMBX10, CMMI10, …), so a CMBX10 heading and its CMR10 body stop comparing equal and the dominant-font tally fragments into ~26 buckets. The subset-prefix strip and alias map in flash/model/span_line.py:78-80 do not collapse them. _italic_font_re also starts matching CMMI5/7/10 and _bold_font_re matches *-BoldMT, neither of which the family names hit.

requirements.txt:7 pins pypdfium2==5.13.0, so this is what fresh installs get, while pyproject.toml keeps the floor at >=4.30.0 — both semantics are live at once. Preferring FPDFFont_GetFamilyName in the getattr chain restores the original behavior.

PageIndex/pageindex/flash/parser_pdfium_charlevel/geometry.py

Lines 8 to 10 in 28eaab8

	_get_font_name = getattr(pdfium_c, "FPDFFont_GetBaseFontName", None) or pdfium_c.FPDFFont_GetFontName

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[rejojer]

Correction to the review above

The finding's mechanism is confirmed, but its stated consequence is wrong — a full A/B run reverses it.

Confirmed: pdfium split FPDFFont_GetFontName rather than renaming it — FPDFFont_GetFamilyName (present in pypdfium2 5.13.0) returns byte-identical names to 4.30's GetFontName, while the shim's FPDFFont_GetBaseFontName returns per-face /BaseFont names. 57a77f6's "removed (now GetBaseFontName)" framing was inaccurate, and the granularity change is real (5 family names → 35 per-face names on attention-residuals.pdf).

Reversed: the predicted heuristic breakage did not materialize. Extracting all 9 example PDFs (1574 pages) under both versions:

• Tree skeletons: 6/9 byte-identical; the other 3 differ only in title text, every difference an improvement (Super visory → Supervisory, var[f (x)] → var[f(x)]). Node counts, nesting, and page ranges identical everywhere.
• Page text: 71% of pages byte-identical. 21% differ in whitespace only — 1036 spurious spaces removed (math notation: f (x) → f(x), N , → N,) vs 178 added (173 concentrated in four-lectures.pdf pseudocode: insert(i, → insert( i, — the one real regression, accepted). 8% reorder within the page with identical character multisets (table linearization; neither order is ground truth). Zero characters lost corpus-wide; one figure axis label 4.30 dropped is recovered.

What remains, now fixed in 228426d: the actual defect was the dual-semantics window — pyproject.toml allowed >=4.30.0 while requirements.txt pinned 5.13.0, so the same SDK version produced different extracted text depending on install path, with no signal. The floor is now >=5 and a sentinel test pins the 5.x extraction behavior:

PageIndex/pyproject.toml

Lines 34 to 36 in 228426d

	PyPDF2 = ">=3.0.0"
	pypdfium2 = ">=5"
	sortedcontainers = ">=2.4.0"

PageIndex/tests/test_flash_extraction.py

Lines 1 to 23 in 228426d

	"""Pins the pdfium 5.x text-extraction semantics the parser is calibrated to.
	pdfium split FPDFFont_GetFontName into GetBaseFontName (/BaseFont, per-face)
	and GetFamilyName (old family semantics); the parser uses the per-face names,
	which changes word joining and figure-label pickup. These sentinels come from
	a 4.30-vs-5.13 corpus A/B and fail if the semantics move again.
	"""
	from importlib.metadata import version
	from pathlib import Path
	import pytest
	PDF = Path(__file__).parent.parent / "examples" / "documents" / "earthmover.pdf"
	@pytest.mark.skipif(int(version("pypdfium2").split(".")[0]) < 5,
	reason="extraction is pinned to pdfium 5.x font-name semantics")
	def test_page_text_pins_pdfium5_semantics():
	from pageindex.flash.main import extract_toc
	page7 = extract_toc(str(PDF))["page_texts"][6]
	assert "p5\nEMD\n1.0" in page7 # figure axis label pdfium 4.x dropped
	assert "break loop\n5: if lbp" in page7 # pseudocode lines no longer glued

The getattr fallback in geometry.py stays so not-yet-upgraded environments keep running. Switching the shim to FPDFFont_GetFamilyName (restoring 4.30 output) was considered and rejected: it would revert the measured improvements to keep behavior nobody depends on.

🤖 Generated with Claude Code