Skip to content

feat(examples): add setup tooling so the examples can actually be run - #591

Closed
williedoran-neo4j wants to merge 15 commits into
neo4j:mainfrom
williedoran-neo4j:willie/examples-environment-setup
Closed

feat(examples): add setup tooling so the examples can actually be run#591
williedoran-neo4j wants to merge 15 commits into
neo4j:mainfrom
williedoran-neo4j:willie/examples-environment-setup

Conversation

@williedoran-neo4j

@williedoran-neo4j williedoran-neo4j commented Aug 10, 2026

Copy link
Copy Markdown

Linear: GENKGB-1857

Follow-up to #578, which fixed the gpt-5 breakage in the OpenAI examples. That work exposed a second problem: there is no way to run the other examples without reverse-engineering their requirements from the source.

examples/README.md is a pure link index — no mention of API keys, extras, Docker, or the fact that sixteen examples talk to a remote demo database. What setup knowledge exists is spread across the README's Tests section, three per-store READMEs, and tests/e2e/docker-compose.yml, which exists for the e2e suite rather than the examples. CI uses a fourth, divergent path: hand-rolled docker run, no compose.

What's here

scripts/setup_examples.py — a doctor and an interactive installer.

python scripts/setup_examples.py --check   # what is missing, changes nothing
python scripts/setup_examples.py           # walk through fixing it

--check resolves every example against a shared requirement model, probes packages, env vars, databases and containers, and reports the single first thing blocking each one, grouped so the biggest wins surface first. It writes nothing and exits 0 unless --strict. On a machine with keys set but no services running it reports 52 runnable, 34 blocked, 18 snippets — and 33 of those 34 are blocked only on a stopped Neo4j or Ollama, which is the point: bring the stack up and they run.

The installer works in tiers, so the free and local providers work before any cloud account is involved: extras and local Neo4j, then API keys, then local runtimes, then cloud. Every step is skippable and safe to re-run. Tier 3 asks whether you're using a Neo4j Aura dev environment or your own cloud accounts — the internal path is useless to an outside contributor, and the reverse is wrong for a Neo4j developer, and neither can be inferred safely.

scripts/example_requirements.py — one model of what each example needs, shared with check_examples.py. Package requirements come from an AST import scan mapping library symbols to extras, because an example imports OpenAILLM rather than openai, so the dependency is invisible from its imports alone. Env vars and datastores are read from os.getenv calls and connection URIs. Anything that leaves no trace in the source — APOC, a pre-existing index, outbound internet, or a component named only in a YAML/JSON config — is declared per path.

This also replaces providers_used() in check_examples.py, which matched provider names as substrings of the whole file: any file containing the word "google" counted as a Vertex example, and OpenAIEmbeddings couldn't be told from AzureOpenAIEmbeddings. --live now also skips snippets rather than running them and reporting a pass, and checks whether the services an example needs are actually up — a stopped Neo4j previously surfaced as FAIL on ~30 examples, which reads as "the examples are broken" when the truth is "nothing was running".

examples/docker-compose.yml — Neo4j with APOC by default; a vectordb profile adding Weaviate, Qdrant and Pinecone Local; an optional ollama profile. Services declare healthchecks so up --wait blocks until they actually answer; the e2e stack has none and CI polls readiness by hand, which leaves local users racing connection-refused. The one exception is pinecone-local, whose image ships no shell and no binary a healthcheck could run — documented in the file header. Pinecone Local is an in-memory emulator that ignores API keys, so that path no longer needs a hosted account. Same ports as the e2e stack, so the two are mutually exclusive — the file header says so.

examples/.env.example and examples/SETUP.md — a credential template, and per-example docs covering provider, extras, env vars and services, plus which providers are free without a card. Every non-OpenAI provider turns out to have a free or local path. examples/README.md now links it.

python-dotenv and requests declared in the examples extra — fourteen examples call load_dotenv() and tools_retriever_example.py calls a web API with requests, but neither was declared, so both resolved only by accident.

Credentials

Keys are read with getpass, validated against the provider with a free list-models call, and written only to a gitignored .env. A key that doesn't validate is never written — a bad value there turns a clear "unset" into a confusing 401. Nothing is printed in full, and cloud account identifiers are resolved at run time rather than persisted.

Two details worth calling out, because both are the kind of thing that is easy to get subtly wrong:

  • The key never travels in a URL. Gemini's validator sends it as an x-goog-api-key header, and no error message interpolates a URL, since a query string is logged by every proxy in the path and error strings are printed to the terminal.
  • .env is created at mode 0600 rather than created and then chmod-ed, so the secret never exists on disk world-readable, not even briefly. It is written through a temp file and os.replace, so an interrupted write cannot truncate the credentials already there.

Tests

tests/unit/scripts/ — 44 cases over the requirement model, the .env writer and the provider validators. No network (urlopen is replaced and a fixture fails the test if anything reaches out), no keys, no services.

They are there because they catch real things. Each of these was confirmed to fail against the code as it was before: the key-in-the-URL disclosure, the .env creation mode, and the three config-file examples being reported as needing nothing at all. Worth noting for anyone reviewing the permissions test: asserting the final mode of .env proves nothing, since the old and new code both end at 0600 — the test forbids the chmod fixup instead, so the mode has to be right at creation.

scripts/ is deliberately outside the coverage gate ([tool.coverage.run] source = ["src"]), so these are additive and cannot move the 90% number.

For review

  • The feat(scripts): add static checks for the examples directory commit is byte-identical to fix(examples): make OpenAI examples work with gpt-5 #578's copy of that file. It exists so this branch stands alone off main; it drops out cleanly on rebase once fix(examples): make OpenAI examples work with gpt-5 #578 merges. Nothing to re-review there. The same applies to the one-line chore(tests): put scripts/ on the pytest path commit, which both branches need independently.

    Both were verified rather than assumed — rebasing this branch onto a main containing fix(examples): make OpenAI examples work with gpt-5 #578 reports dropping … patch contents already upstream and skipped previously applied commit, 15 commits become 13, with zero conflicts.

  • With both branches together the checker reports 104 files, no problems — verified locally. On this branch alone it reports 22 problems, because this branch carries the checker but not fix(examples): make OpenAI examples work with gpt-5 #578's example fixes.

  • Commits are ordered to be reviewed one at a time: compose and env template first (no Python), then the requirement model, then the CLI on top, then docs.

  • Deliberately not fixed, documented in SETUP.md instead: two examples write message history to the read-only demo database and fail with Forbidden, and three need PYTHONPATH=examples/data.

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Documentation update
  • Project configuration change

Complexity

Complexity: Medium

How Has This Been Tested?

  • Unit tests
  • E2E tests
  • Manual tests

Exercised, not just reviewed. Brought the stack up from clean and watched the doctor's runnable count climb as services started. Ran tier 0 and confirmed it creates the four indexes the examples assume — including the moviePlotsEmbedding index on the local database that no script in this repo previously created. Confirmed all five key validators reject an invalid key and write nothing. Stopped Neo4j and confirmed the doctor reports blocked rather than passing. The t2v-transformers healthcheck was verified by running the image: it has no wget or curl but does ship python3, and the ready endpoint answers 204.

Green on ruff check, ruff format --check, mypy --strict, and the unit suite with coverage at the 90% gate.

Checklist

  • Documentation has been updated
  • Unit tests have been updated
  • E2E tests have been updated
  • Examples have been updated
  • New files have copyright header
  • CLA (https://neo4j.com/developer/cla/) has been signed
  • CHANGELOG.md updated if appropriate

🤖 Generated with Claude Code

williedoran-neo4j and others added 15 commits August 11, 2026 10:03
Fourteen examples call load_dotenv(), but python-dotenv was never
declared. It resolves today only transitively, through another extra,
so the examples work by accident and would break if that path changed.

Sits on the lock regeneration in the previous commit, which keeps the
lock diff here to the two lines this actually adds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The examples read five environment variables between them and there was
no template, no mention of them in examples/README.md, and no committed
guidance on where to put a key.

examples/.env.example is that template. Every credential line is
commented out on purpose: an empty FOO= would blank out an already
exported variable when the file is sourced, so a line is uncommented
only when it carries a real value.

.gitignore already covered .env; this widens it to .env.* with an
explicit !.env.example negation, so .env.local and similar variants
cannot be committed by accident while the template stays tracked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the examples locally needed Neo4j with APOC, and the vector
store examples needed Weaviate, Qdrant or Pinecone. That knowledge was
spread across the README's test section, three per-store READMEs and
tests/e2e/docker-compose.yml, which exists for the e2e suite rather
than for the examples.

Profiles keep the default cheap: bare 'up' starts Neo4j alone, and the
vector stores or a containerised Ollama are opt-in.

Two things this adds over the e2e stack:

- Healthchecks, so 'up --wait' blocks until the services actually
  answer. The e2e stack has none and CI polls readiness by hand, which
  leaves local users racing connection-refused.
- pinecone-local, an in-memory emulator that ignores API keys, so the
  Pinecone examples no longer require a hosted account.

It deliberately binds the same ports as tests/e2e/docker-compose.yml,
so the two stacks are mutually exclusive; the file header says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Identical to the file in PR neo4j#578 - reviewed there, reproduced here so
this branch stands alone off main. Nothing to re-review in this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Answers one question per example: what does it need in order to run.
Used by the setup tooling in the following commits, and by
check_examples.py once the commit after this one rewires it.

Requirements come from three places, in decreasing order of how well
they can be inferred:

- Packages, from an AST import scan. Examples import library symbols
  rather than provider SDKs - OpenAILLM, not openai - so the scan maps
  neo4j_graphrag symbols to extras, not just top-level modules.
- Env vars and datastores, read out of os.getenv calls and connection
  URIs appearing as string literals.
- Everything else - APOC, a pre-existing index, outbound internet -
  leaves no reliable trace in the source, so it is declared per path.

It also detects the three examples that import a vector from
examples/data as a bare module, which only resolves with that directory
on sys.path.

Stdlib only, so it runs before uv sync.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
providers_used() matched provider names as substrings of the whole
source, so any file containing the word 'google' counted as a Vertex
example and OpenAIEmbeddings could not be told from
AzureOpenAIEmbeddings. It now delegates to the shared requirement
model, which resolves providers from imported symbols.

--live gains two things from that model:

- It skips snippets, which have nothing to run, instead of executing
  them and reporting a pass.
- It checks whether the services an example needs are actually up. A
  stopped Neo4j previously surfaced as FAIL on ~30 examples, which
  reads as 'the examples are broken' when the truth is 'nothing was
  running'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Foundations for the setup tooling in the following commits, kept apart
because they carry the rules that matter most:

- envfile writes credentials to .env only, at mode 0600, and masks
  every value it echoes back. It also overlays .env onto the process,
  so a credential that exists only in that file is still visible to
  SDKs that read the real environment - without it a correctly
  configured AWS_PROFILE reports as 'no credentials'.
- probes answer 'is this present' for packages, commands, a Neo4j with
  APOC and its indexes, and pulled Ollama models. Everything degrades
  to a negative answer rather than raising, so the doctor still runs
  before uv sync.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One entry per service the examples talk to: which extra it needs, which
env vars, what its free tier gives you, and where to get a key.

Validators call the provider's list-models endpoint. That is free, and
it proves the key is live rather than merely well-formed, which matters
because the installer refuses to store a key that does not validate - a
bad value in .env turns a clear 'unset' into a confusing 401.

Bedrock and Vertex carry a credential probe instead of an env var:
their credentials come from a CLI login or a named profile, so testing
for a variable would report a working setup as broken.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The read-only half of the setup tooling. Resolves every example against
the shared requirement model, probes what is present, and reports the
single first thing standing in the way of each one, grouped so the
biggest wins surface first.

It writes nothing, reaches no further than a TCP connect, and exits 0
unless --strict, so it is safe to run casually.

Credentials are reported as set or unset and never printed, hashed, or
partially revealed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Walks the same ground as the doctor, in tiers, so the free and local
providers work before any cloud account is involved: extras and a local
Neo4j with the indexes examples assume, then API keys, then local
runtimes, then cloud. Every step is skippable and safe to re-run.

Keys are read with getpass, validated against the provider, and written
to .env only on success.

Tier 3 lives in its own module because it has to ask who is running it.
Reading an Aura dev environment through omni is useless to an outside
contributor, and telling a Neo4j developer to run 'aws configure' is
the wrong advice; neither can be inferred safely, so it prompts. Cloud
account identifiers are read at run time and never persisted, since
they change whenever an environment is rebuilt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
    python scripts/setup_examples.py --check   # what is missing
    python scripts/setup_examples.py           # walk through fixing it

Argument parsing and tier dispatch only; the work lives in the
examples_setup package.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ollama_llm.py and ollama_embeddings.py shipped a literal <model_name>
placeholder, so they could not run without being edited first.
ollama_tool_calls.py instead hardcoded mistral:latest, which 404s
unless you happen to have pulled that model.

There is no model name that is right for everyone - it depends on what
you have pulled - so it is now an argument, with a default that has
been run:

    ollama pull llama3.2
    python examples/customize/llms/ollama_llm.py llama3.2

Defaults are llama3.2 for the two LLM examples and nomic-embed-text for
embeddings. All three pass with no arguments once those are pulled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
examples/README.md indexes what the examples demonstrate but says
nothing about how to run one: no API keys, no extras, no Docker, no
mention that sixteen of them talk to a remote demo database.

SETUP.md covers that, including a per-example table of provider,
extras, env vars and services, and which providers are free without a
credit card - every non-OpenAI provider turns out to have a free or
local path.

It also records what running all of them turned up, so the next person
does not debug their own setup for hours over a defect that is already
known: three library bugs that stop Cohere and MistralAI working at
all, four dead model references, and the three examples that need
examples/data on PYTHONPATH. Each carries the replacement that was
verified working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/ is not a package and is not installed, so tests under tests/ cannot
import check_examples. Add it to pythonpath under the same top-level module
names mypy already resolves it under; importing it as scripts.check_examples
instead would give mypy the same file under two module names.

Coverage is unaffected: [tool.coverage.run] source is ["src"], so nothing under
scripts/ can enter the measured set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…detects

Review follow-ups on the setup tooling:

- The Gemini validator carried the key in the query string, and an unreachable
  host interpolated the whole URL into a string the installer prints. Any
  offline, DNS or proxy failure put the key in the user's scrollback. Sent as a
  header now, and only the host reaches the message.
- .env was created by write_text() and chmod-ed afterwards, so a fresh file held
  the key world-readable until the next statement. Created at 0600 through a
  temp file and os.replace, so an interrupted write also cannot truncate the
  credentials already there.
- The Ollama tier pulled qwen3:8b while the examples default to llama3.2, so a
  completed setup run left them failing, and it closed by telling the user to
  edit a <model_name> placeholder that argparse replaced earlier in this branch.
- Removed the placeholder machinery entirely: no example contains it any more,
  so the doctor blocker and the --live skip it fed were unreachable.
- Three config-file examples name their LLM in YAML/JSON, invisible to the
  import scan, and were reported as needing nothing. SERVICE_RULES entries can
  now declare extras, providers and env vars for cases like this.
- install_hints() could only ever return [], because analyse() narrowed modules
  to exactly the set install_hints() filters back out. It now surfaces the real
  case: tools_retriever_example.py imports requests, which nothing declared.
  Declared it, and python-dotenv, in the examples extra.

Also: linked SETUP.md from examples/README.md, gave t2v-transformers a
healthcheck (python3, since the image has no wget or curl) and documented why
pinecone-local cannot have one, honoured --non-interactive in the Aura tier, and
stopped the doctor parsing all 104 examples four times and re-resolving cloud
credentials once per example.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@williedoran-neo4j

Copy link
Copy Markdown
Author

Closing this in favour of five smaller PRs. At 15 commits and ~3,900 lines it was too much to ask of a reviewer, and going back over it turned up a fair amount that should not have shipped at all — a hand-maintained 86-row table with rows that were already wrong, a Provider.extra field with fifteen values and no readers, a --provider flag that silently did nothing for 7 of 15 providers, and a tier-0 step that pre-created the two indexes examples/database_operations/create_*_index.py exist to demonstrate, turning both examples into no-ops.

It also carried a Neo4j-internal omni/Aura path that prompted outside contributors with an option they cannot use. That is a sentence in examples/SETUP.md now, not a code path.

Replaced by:

Net effect: roughly 1,400 fewer lines than this PR, in pieces that can be reviewed and merged independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant