[WAF-09] Implement TX and persistent IP, SESSION, USER, and GLOBAL collections

[chrisbbreuer]

Objective

Implement bounded transaction-local TX mutation and cross-request IP, SESSION, USER, GLOBAL, and RESOURCE collections with ModSecurity 3.0.16 and Coraza 3.7.0 semantics. Persistence must be concurrency-safe, expiration-aware, and explicit about backend failure policy; ordinary requests that do not initialize a persistent collection must pay no persistence cost.

Compatibility contracts

• TX remains transaction-owned and is never written to a persistence backend.
• initcol binds a persistent namespace to a caller-supplied key, loads it at most once per transaction, lazily removes expired variables, and exposes the stored values through the normal collection API.
• setuid, setsid, and setrsc bind the USER, SESSION, and RESOURCE keys through the same initialization path.
• setvar supports bounded assignment, deletion, and signed numeric addition/subtraction with checked overflow and compatible missing/non-numeric handling.
• expirevar stores an absolute per-variable expiration derived from a bounded TTL. Expired values are invisible on load and removable by bounded maintenance cleanup.
• Only dirty persistent collections are committed. A commit atomically applies the transaction mutation log against an expected record revision.
• Revision conflicts use a bounded retry/rebase policy so concurrent increments do not silently lose updates.
• Backend unavailability, timeout, conflict exhaustion, corruption, and capacity errors remain distinct. The configured fail-open or fail-closed policy is observable and never silently changes ruleset behavior.

Deliverables

• A versioned backend callback contract with explicit ownership for load results, atomic commits, cleanup, feature discovery, and typed errors.
• A bounded transaction persistence session that tracks namespace keys, loaded revisions, dirty mutations, expiration metadata, and retry limits.
• A thread-safe in-memory backend for tests, embedded deployments, and concurrency qualification.
• A production LMDB backend linked from Pantry-managed openldap.org/liblmdb; no system-library fallback and no Git submodule.
• Builder/runtime configuration for backend ownership, failure policy, key/value/record limits, retry limits, deadlines, and cleanup budgets.
• Engine APIs required by the future SecLang action compiler for initcol, setuid, setsid, setrsc, setvar, and expirevar.
• Focused unit, threaded integration, restart/persistence, expiration, corruption, fault-injection, and performance coverage.
• Documentation and machine-readable compatibility evidence.

Interfaces

The backend receives a composite (namespace, collection_key) key. load returns an allocator-owned snapshot with a monotonic revision. commit receives the expected revision and an ordered mutation batch and must either apply all changes once or return a conflict without partial publication. cleanup removes no more than the configured record/variable budget.

The transaction layer owns loaded values and mutation data. Backend callbacks may block only when a persistent collection is explicitly initialized or flushed; they are never called by the default request lifecycle. Callers choose fail-open or fail-closed behavior at build time.

Resource limits

• Maximum initialized namespaces per transaction.
• Maximum collection-key, variable-name, and variable-value bytes.
• Maximum values and serialized bytes per persistent record.
• Maximum mutations, retry attempts, cleanup work, and backend wait duration.
• Checked timestamp and signed arithmetic with deterministic error reporting.

Acceptance criteria

• TX assignment, deletion, and checked arithmetic match the pinned compatibility fixtures.
• Every persistent namespace round-trips through the in-memory and LMDB backends.
• Expired variables are never exposed, and cleanup remains bounded.
• Concurrent increments preserve the exact total under real threads; conflict exhaustion follows the configured policy.
• LMDB restart tests prove committed records survive process/backend recreation and atomic failures leave the prior revision readable.
• Corrupt or oversized persisted data is rejected without unbounded allocation.
• Requests that never initialize persistence make zero backend calls.
• Exact Pantry-pinned Zig formatting, tests, compile checks, and the persistence benchmark pass locally and in hosted CI.
• Implementation lands on main in small conventional commits with issue evidence and no unexplained compatibility skip.

Test matrix

• Fresh/missing and existing collection initialization.
• Case-insensitive persistent variable keys and independent namespace/collection keys.
• Assignment, deletion, positive/negative delta, zero, invalid numeric text, and overflow.
• TTL boundary, clock rollback, lazy expiration, refresh, delete, and bounded cleanup.
• Concurrent distinct records and heavily contended single-record increments.
• Conflict retry success/exhaustion and backend timeout/unavailable/corrupt/capacity faults under both failure policies.
• LMDB open, close, restart, map-full, reader pressure, malformed bytes, and crash-safe atomic transaction behavior.
• No-persistence lifecycle and initialized-collection latency/allocation benchmarks.

Relationships

• Parent: WAF-00

Blocked by

• WAF-08 — complete

Blocks

• WAF-12 — persistent directives and collection configuration
• WAF-15 — ctl and action execution using collection mutation
• WAF-34 — SecRules Test Set persistence cases
• WAF-35 — ModSecurity differential fixtures
• WAF-36 — Coraza differential fixtures

Baselines and upstream references

• ModSecurity 3.0.16
• Coraza 3.7.0
• ModSecurity initcol
• ModSecurity expirevar
• LMDB
• Zig 0.17.0-dev.1441+d5181a9c9, resolved and locked through Pantry

[chrisbbreuer]

WAF-08 (#9) is complete and its collection contract is now available on main at 531cc68. Hosted qualification is green: https://github.com/zig-utils/zig-waf/actions/runs/29935997519

This removes the WAF-08 dependency for this issue. Preserve the published ownership, phase-gating, origin, bounded-storage, selector/error, and derived-family contracts when implementing this scope.

[chrisbbreuer]

First implementation slice landed in 011776f (feat(engine): add revisioned persistence backend).

It adds:

• versioned backend callback table with explicit snapshot ownership and typed failure modes
• bounded IP/SESSION/USER/GLOBAL/RESOURCE namespace and mutation contracts
• optimistic expected-revision atomic commits
• expiry-aware loads and bounded cleanup
• thread-safe in-memory backend that builds replacements before publication, so validation/allocation/conflict failures cannot expose partial mutation batches
• focused atomicity, conflict, expiry, cleanup, and capacity tests

Exact pinned Zig build test and build check passed before push. Engine transaction sessions, conflict-safe arithmetic, LMDB, concurrency qualification, docs, benchmarks, and hosted release qualification remain open in this issue.

[chrisbbreuer]

Conflict-safe transaction persistence landed in main as feat(engine): add persistent transaction sessions.

The session performs no backend I/O until explicit initialization, loads each namespace/key once, owns bounded mutation logs, flushes only dirty bindings, and retries optimistic conflicts by reloading the current revision. Ordered set, remove, add, and expire mutations are supported; numeric additions treat missing/non-numeric prior data as zero to match the pinned ModSecurity set_var.cc behavior while using checked i64 arithmetic instead of native integer overflow.

A focused two-session stale-revision test proves concurrent deltas compose to the exact stored total. Exact pinned Zig tests and compile checks passed before push.