feat: Implement HTTP/2 optimized ArangoDB client with neural process isolation

[r3d91ll]

Overview

Implement a custom HTTP/2 Python client for ArangoDB that bypasses python-arango limitations and achieves 2-5x performance improvement over the current PHP subprocess bridge.

Background

As documented in PRD: ArangoDB Optimized Connection Architecture (/docs/prd/arango_optimized_connection_prd.md), we need to optimize database connections for our neural memory architecture where ArangoDB serves as the hippocampus for local AI models.

Current Performance

• PHP subprocess bridge: ~100ms per operation
• Target: 20-50ms (2-5x improvement)
• Ultimate goal: ~5ms with C++/CUDA (future)

Design Principles

Neural Process Isolation - No authentication, just like the hippocampus:

• Inference Mode (wake state): Read-only access via RO proxy
• Consolidation Mode (sleep state): Write access via RW proxy
• Unix sockets only: Never exposed to network
• Zero auth overhead: Process isolation via proxies

Architecture

┌─────────────┐      ┌──────────┐      ┌──────────┐
│  Inference  │─────▶│ RO Proxy │─────▶│          │
│  (reads)    │      │(allowlist)│      │ ArangoDB │
└─────────────┘      └──────────┘      │          │
                                        │ (single  │
┌─────────────┐      ┌──────────┐      │  socket) │
│Consolidation│─────▶│ RW Proxy │─────▶│          │
│  (writes)   │      │(allowlist)│      │          │
└─────────────┘      └──────────┘      └──────────┘

Implementation Tasks

Phase 1: Proof of Concept & Benchmark (Days 1-3)

• Create minimal HTTP/2 client with Unix socket support
• Implement basic operations (get, insert, query)
• Verify HTTP/2 negotiation via response.http_version
• Benchmark against PHP baseline with split metrics
• GO/NO-GO DECISION POINT (must achieve 2x improvement)

Phase 2: Full Implementation (Days 4-7) - ONLY IF Phase 1 Passes

• Implement RO proxy with allowlist enforcement
• Implement RW proxy with controlled write access
• Add connection pooling and keep-alive
• Implement streaming cursor support
• Add NDJSON bulk import with Content-Length header
• Optimize batch sizing based on benchmarks

Phase 3: Integration (Days 8-10) - ONLY IF Phase 2 Successful

• Update DatabaseFactory to use new client
• Modify workflows to use optimized batching
• Run end-to-end performance tests
• Document configuration and tuning

Critical Requirements

From expert review validation:

1. HTTPX Configuration: http2=True MUST be on Client
2. Content-Length header REQUIRED for NDJSON (ArangoDB requirement)
3. Proxy-based enforcement - Unix permissions alone don't restrict HTTP verbs
4. Socket modes: 0660 for proxy sockets (not 0644)
5. No authentication - Process isolation provides security

Success Metrics

Go/No-Go Criteria (Phase 1)

• Single document fetch: <50ms (2x improvement)
• Bulk insert (1000 docs): <400ms (2x improvement)
• Query execution: <200ms (2x improvement)
• Stretch goal: Any operation reaching ~20ms range

Files to Create

1. /core/database/arango/optimized_client.py - Custom HTTP/2 client
2. /core/database/arango/proxies/ro_proxy.go - Read-only proxy
3. /core/database/arango/proxies/rw_proxy.go - Read-write proxy
4. /tests/benchmarks/arango_connection_test.py - Benchmark suite

Fallback Plan

If benchmarks don't meet 2x improvement target, we continue using the existing PHP bridge which already works reliably.

References

• Full PRD: /docs/prd/arango_optimized_connection_prd.md
• Conveyance Framework: /CLAUDE.md
• Expert review validated approach as "sign-off ready"

---

Priority: High
Effort: 10 days (with go/no-go gates)

[r3d91ll]

Phase 1 & 2 update:\n\n- HTTP/2 client over Unix sockets is live () with GET/insert/query latency ~0.4ms / 6ms / 0.7ms.\n- Read-only and read-write proxies implemented (, ), enforce allowlists & 0660 sockets; proxy benchmarks still ~0.6ms GET, ~7ms insert, ~0.8ms query.\n- Benchmark helper () accepts credentials/socket paths.\n- PRD summary & README tables capture the gains.\n\nAll Phase 1 success criteria met; proxies confirm we maintain >2x improvement. Preparing for Phase 3 integration next.

[r3d91ll]

Phase 1 & 2 update:\n\n- HTTP/2 client over Unix sockets () achieves ~0.4 ms GET, ~6 ms bulk insert (1k docs), ~0.7 ms queries.\n- Read-only/read-write proxies (, ) enforce allowlists via Unix sockets; proxied benchmarks ~0.6 ms GET, ~7 ms insert, ~0.8 ms query.\n- Benchmark helper () updated with auth flags; doc/README refreshed with benchmark table.\n\nRequesting sign-off to proceed with Phase 3 integration (DatabaseFactory wiring, workflow updates).

[r3d91ll]

Updated proxy benchmarks: GET ≈0.53 ms, insert (1k docs) ≈6.3 ms, query (1k) ≈0.86 ms. Numbers remain >2× faster than PHP bridge (100/400/200 ms). Ready to move into Phase 3 integration when approved.

[r3d91ll]

Phase 3 Integration Complete

Deliverables

• Added DatabaseFactory.get_arango_memory_service() on top of the optimized HTTP/2 client.
• Replaced the PHP bridge in workflow_arxiv_initial_ingest_* with the new memory client (drop/create collections, bulk inserts, queries).
• Updated monitoring/analysis tools to talk to the RO/RW proxies via the memory client.
• Documentation refreshed: README, PRDs, and workflow guide now reference the proxy-based HTTP/2 path.

Benchmarks (RO proxy)

• GET by key (hot cache, end-to-end): median ~0.5 ms (p95 <0.8 ms).
• AQL cursor (LIMIT 1000, body consumed): median ~0.7 ms (p95 <1.1 ms).
• Bulk insert 1,000 docs (~1 KB/doc, waitForSync=false): median ~6–7 ms (~6–7 µs/doc).

These numbers are 20–200× faster than the old PHP subprocess path and collapse T in our conveyance equation, unlocking much higher effective throughput.

[r3d91ll]

Phase 4 – Validation & Hardening Update

Benchmarks (reports in benchmarks/reports/):

Scenario	Metric	Result
GET hot cache (get_hot.json)	TTFB p95 / p99	0.360 ms / 1.337 ms
	E2E p95 / p99	0.402 ms / 1.409 ms
GET cache-busting (get_hot_random.json)	TTFB p95 / p99	0.390 ms / 1.230 ms
	E2E p95 / p99	0.433 ms / 1.287 ms
GET concurrency 8 (get_hot_c8.json)	E2E p95 / p99	4.573 ms / 5.349 ms
GET concurrency 32 (get_hot_c32.json)	E2E p95 / p99	18.148 ms / 18.590 ms
GET concurrency 128 (get_hot_c128.json)	E2E p95 / p99	57.591 ms / 59.364 ms
Query 1k docs, hot (query_hot.json)	E2E p95 / p99	0.890 ms / 1.898 ms
Query 1k docs w/ varying offset (query_hot_var_offset.json)	E2E p95 / p99	0.918 ms / 1.688 ms
Insert 1k × 1 KB docs, async (insert_hot_async.json)	E2E p95 / p99	12.939 ms / 13.906 ms
Insert 1k × 1 KB docs, waitForSync (insert_hot_sync.json)	E2E p95 / p99	13.521 ms / 13.608 ms
Insert 1k × 4 KB docs (insert_hot_4k.json)	E2E p95 / p99	22.752 ms / 23.200 ms
Insert 1k × 16 KB docs (insert_hot_16k.json)	E2E p95 / p99	356.420 ms / 401.267 ms

Highlights

• Hot-cache GET and query latencies hit the Phase 4 SLOs (p95 well under 1 ms and 2 ms respectively).
• Bulk import meets the ≤ 15 ms p95 target for 1 KB documents; payload sweeps show the expected growth at 4 KB and 16 KB.
• Concurrency sweeps (1→128) document how latencies scale, providing a baseline for future tuning.
• All runs executed through the RO/RW proxies with auth; the JSON artefacts are ready for regression diffs.

Next Steps

1. Fold these numbers into the PRD (success metrics + verification checklist).
2. Capture cold-cache variants after restarts (for correctness suite).
3. Draft the systemd socket/service units (RO/RW proxies) so “neural process isolation” can be codified in-repo.
4. Re-run the waitForSync benchmark with more iterations to firm up p95/p99.