graphify export CLI hard-caps at 512 MB graph.json

[vahem-vmssoft]

I'm hitting this error on a large C codebase (~49k files):

ERROR: graph file graphify-out/graph.json is 624_595_520 bytes,
exceeds 536_870_912-byte cap

Final graph: 518,749 nodes / 1,283,722 edges / 22,565 communities built via /graphify (AST + Haiku 4.5 semantic pass on a scoped subset). The 624 MB graph.json written by to_json() is correct and complete — the cap is in the CLI loader, not in the data.

Workaround that worked: dropping to the Python API, calling graphify.export.to_html(G, communities, "graph.html", node_limit=5000, ...) directly produces a usable 6.5 MB aggregated community-meta-graph HTML without ever re-reading graph.json.

Probable solutions:

1. CLI flag (e.g. --max-graph-bytes or GRAPHIFY_MAX_GRAPH_BYTES env var) to override the cap for users who know what they're doing.
2. Or: when graph.json exceeds the cap, auto-fall-back to the same community-aggregation path to_html(node_limit=...) already implements, instead of failing.
3. Document the cap in the error message itself: cite the env var or flag that overrides it, so users don't have to dig into the source.

Versions: graphify installed today (uv tool install graphifyy), Python 3.13, macOS arm64.

I am happy to open a PR, but it would be good to hear from the community first, which solution(s) is/are better?

[a-marionette]

Also encountering the same.

[safishamsi]

Hey,feel free to send a PR

[safishamsi]

Fixed in HEAD — all three solutions you proposed are implemented, shipping in the next release.

Solution 1 — GRAPHIFY_MAX_GRAPH_BYTES env var

GRAPHIFY_MAX_GRAPH_BYTES=700MB graphify query "..."
GRAPHIFY_MAX_GRAPH_BYTES=2GB graphify html
GRAPHIFY_MAX_GRAPH_BYTES=671088640  # plain bytes also accepted

Accepts plain bytes, <N>MB, or <N>GB (1024-based). Read fresh on every size check, so it can be set in your shell profile, CI env, or .env.

Solution 2 — Better error message
The cap error now tells you exactly what to set:

error: graph file graphify-out/graph.json is 624_595_520 bytes, exceeds 536_870_912-byte cap
(set GRAPHIFY_MAX_GRAPH_BYTES=<bytes> or GRAPHIFY_MAX_GRAPH_BYTES=<N>GB to raise the limit)

Solution 3 — Auto-fallback for graphify html
When graph.json exceeds the cap and you run graphify export html, instead of a hard error it now automatically falls back to the community-aggregation view — the same to_html(..., node_limit=5000) path your Python-API workaround already found:

warning: graph.json exceeds cap (624595520 bytes); falling back to community-aggregation view (node_limit=5000)

Other export formats (obsidian, wiki, svg, graphml, neo4j) still error so you know to set the env var for those.

All three solutions pass smoke tests. Thank you for the detailed root-cause analysis and the clean solution proposals.