docs: introduce Connect section, connect-string reference, QWP stubs#444
Open
bluestreak01 wants to merge 174 commits into
Open
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 174 commits into
bluestreak01 wants to merge 174 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 41b2a87
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
Closed
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
Reorder the Rust guide from quick start through connection, API choice, usage, and operational guidance before its reference sections. Reflect the default Rust query reader and Zstandard support, simplify the Cargo dependency, and remove the obsolete CMake reader toggle from the C/C++ guide.
The c-questdb-client now uses ws/wss exclusively; drop the qwpws/qwpwss alias mention from the C/C++ and Rust client pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The progress callback is now telemetry-only; without a reset callback a post-delivery failure returns FailoverWouldDuplicate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A flush of a non-empty buffer or chunk always yields an FSN; no value means there was nothing to publish, and the ack watermark has no value until the first frame on the borrow completes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the legacy ILP page with the pooled 5.0.0 surface: connect(), sender leases, DataFrame/Arrow ingestion, streaming queries, delivery and failover semantics. Structure mirrors the Rust client page; every example verified against questdb/questdb:nightly with the client built from source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cross-verified against the questdb 5.0 client source and a live server; every runnable code block now executes end-to-end. Corrections: document the bind-parameter API (the page claimed none existed and advised SQL interpolation); fix the pool configuration keys and defaults (sender_pool_min/max, query_pool_min/max, acquire_timeout_ms, idle_timeout_ms — the previous names fail conf parsing); document connection_listener and ConnectionEvent (the page claimed no callbacks exist); fix the FailoverRetry guidance, which told readers to re-send queued rows and would have caused duplicates; compression defaults to raw with opt-in zstd rather than automatic; close() waits for leases but not for in-flight one-shot results; INT/LONG map to plain numpy unless the column has nulls; timestamps are timezone-naive UTC. Additions: query leases via db.query() with no arguments; QueryResult release semantics including the ResourceWarning on abandonment; QWP version negotiation note; pool-exhaustion behavior; an actionable error-code table; naive-datetime rules for scalars, columns, and binds; cancel() granularity; a connection-listener example; the standalone Sender's QWP transports in the legacy section.
Recovery after a failed flush or ack wait turns on the error's in_doubt flag, not on the error code and not on whether the chunk still holds rows. The page previously claimed a chunk is "left untouched" on failure, which presented failure as a single deterministic state and invited callers to read chunk state as a safe-to-retry signal. Neither signal works. FailoverRetry is reported by both provably-undelivered and delivery-unknown failures, since a socket error is classified to that code on both paths. And an in-doubt chunk is cleared after a failed ack wait but populated when a split remainder fails, so chunk state distinguishes neither case. Buffers and chunks also differ. A buffer publishes as one indivisible frame and every publication failure is provably-not-delivered, so an in-doubt buffer failure always means the ack wait failed with every row queued, and waiting recovers it. A chunk over the frame cap is split across frames, and a failure partway through queues the earlier frames while never accepting the rest, so waiting alone delivers only part of the batch. Splitting is routine rather than exotic: the cap derives from sf_max_bytes, so a chunk over roughly 2 MiB splits at the default. Correct the backpressure note accordingly, which stated the buffer's reject-when-oversize behavior as if it were universal. Also document column_sender_chunk_clear for abandoning a built chunk; the flush path clears automatically once a frame is accepted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The page named err.in_doubt() as an accessor without saying what it means or which way to branch on it, leaving readers with no way to decide whether a failed flush is safe to retry. Give it a section, matching the C/C++ page against the same pool. Recovery turns on in_doubt(), not on code(): a delivery-unknown failure typically reports FailoverRetry, and a socket error is classified to that same code when the rows provably never left, so the code cannot make the decision on its own. Buffers and chunks then differ in how much waiting recovers. A Buffer is one indivisible publication whose every publication failure is provably-not-delivered, so an in-doubt buffer failure always means the ack wait failed with every row queued. A Chunk over the frame cap is split across frames, and a failure partway through queues the earlier frames while never accepting the rest, so waiting delivers only part of the batch. Document the split behavior in the chunk section, since it was absent and the recovery rule depends on it. State is not a substitute for the flag: an in-doubt chunk is cleared after a failed ack wait but populated when a split remainder fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"The 8-row floor" named an implementation detail the pages never define, leaving the reader no way to decode it. Say what happens instead: halving stops at 8 rows because validity bitmaps and boolean columns pack one row per bit, so a frame can only begin on a byte boundary. Also correct which limit binds. Two caps apply: a range that can still be halved targets about 2 MiB, while a block at the floor is held to the queue's real limit of about 4 MiB. "Still too large" read as "still over the 2 MiB just mentioned", naming the wrong threshold for rejection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The examples passed a bare 24 (or 35) as questdb_db_connect's conf_len without the page ever saying the argument is the config string's byte length. The only hint was strlen(conf) in the production example, 1440 lines after the first use. That number sits in the first C code a reader sees. Copying the quick start and changing the host silently passes a wrong length, truncating the config or reading past the literal — a correctness bug rather than a comprehension one. Use strlen(conf) throughout, matching the client's own C examples, and add the string.h include the five affected programs were missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several terms were lifted from the client's internals and left undefined, so a reader could not decode them without having read the source. "Latched" was the worst: four uses, no definition, and it carried the argument for why plain return is safe. Say what happens instead — the sender records the error permanently and every later call fails. The error table's "Latched terminal" cells become "Dead — every later call fails". Unify on "connection": "backend" and "conn" named the same thing in the two places that state the return-vs-drop rule, while the neighbouring sentence already said "connection". The rest, each replaced with what the reader can act on: "background runner" and "parked" hid the real payload, that the client owns a thread you did not create; "slot"/"dirty"/"recovery senders" opened the Durability section with terms defined 220 lines later, or never; "free lists", "publication boundary", and "final owner release" named data structures and lifecycles where a plain statement fits. The close-is-not-concurrency-safe caveat was encoded in a three-word noun phrase despite qualifying the page's claim that the pool is thread-safe. Pull definitions to first use: "ack barrier" was used 100 lines before its definition and FSN roughly 780 to 1000 lines before its own. Both now say what they mean where they first appear, and the contorted ACK-tracking paragraph becomes two. Fix two things the reader would otherwise trip on: the Concurrency section called chunks and buffers "borrowed" when they are caller-created, which sends them looking for borrow_buffer(); and "When in doubt after an ingestion error" collided with in_doubt, the page's own name for a specific flag, while "memory-safe" answered a question nobody asked. Also drop the inert SubmitTimedOut, an internal enum variant that appears in no documented API, and replace "API misuse" with the concrete causes the page documents elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bers "Earlier drafts of this API had one and it was removed deliberately" is history about an API the reader never saw, and "deliberately" argues with an imagined reviewer rather than informing anyone. The sentence that follows already states what is true and stands on its own. Keep the refutation itself: must_close() is a plausible call to go looking for. The split explanation also regressed while being cleaned up. It traded the concrete "more than roughly 2 MiB splits" for "any chunk of real size splits" — a hand-wave where a number belongs — and introduced "the queue's real limit", which implies some other limit is unreal. State both thresholds plainly: frames target 2 MiB, an 8-row frame is checked against the full 4 MiB. Same wording on both pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…API reference The page said tls_verify=unsafe_off "requires an insecure-skip-verify build", implying the reader must opt in, and named a switch spelled nowhere else — so they would guess -Dinsecure-skip-verify and fail. In fact QUESTDB_ENABLE_INSECURE_SKIP_VERIFY defaults to ON and CMake passes the cargo feature through, so a standard build already accepts the key. State the real switch and which direction it goes. "Scope and gaps" promised gaps and listed none. Its first bullet claimed parity with the Rust QuestDb surface, which speaks to someone auditing client consistency rather than to a C developer. Its second pointed at scripts/generate-c-api-reference.js — a script in the documentation repo that writes markdown to stdout and needs the client repo as an argument, so it is maintainer tooling a reader would never run. That script's own header names the real answer: the committed headers are the source of truth. Retitle to "Full API reference" and name them. Also finish the "backend" -> "connection" rename on the Rust page, which describes drop_on_return() against the same pool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The page told the reader three different things about which error
type the C pool uses: the quick start declared line_sender_error
with a comment claiming the pool uses it, the querying section
declared questdb_error for the same questdb_db_connect call, and
the two "throughout" claims contradicted each other outright. The
headers settle it -- questdb_error is a typedef of
line_sender_error, the codes are #define aliases, and the
accessors ship in both spellings. State that once, up front, in a
new subsection under The pool, and point the other three sites at
it. Before, only a reader who made it to the Conventions section
at the bottom could tell that the contradiction did not matter.
Drop the NumPy appender from the two places it was offered as a
general route to decimals, geohash, and arrays. It has no C++
binding at all -- the header marks it intentionally unwrapped as
"awkward to use from C++ without a NumPy host" -- so the page was
pointing C++ readers at an API they cannot call. It survives as a
note scoped to C and to programs embedding Python, carrying the
C-contiguity requirement, since the appender walks the buffer at
the dtype's native stride and reads out of bounds on a sliced or
transposed view.
Remove eight passages that argued with premises the reader never
had: a refutation of a must_close() call the page never mentions,
a denial of a reader feature toggle, a spool-corruption outcome
that cannot happen, and similar. The two negations that refute
plausible-but-nonexistent APIs, set_null and array binds, stay --
they stop a reader guessing at a call that does not exist.
Replace internal terms with what the caller observes: "drive
thread", "slot"/"mints" for the sf_dir directories, the reaper,
and the one-off coinages ("stacked caps", "escape hatch",
"portable receipts"). Retire the coined "ack barrier", which was
defined once and then used in sections a reader can land on cold.
Unpick the densest sentences, unify pool-matched/protocol-matched
and drop/force-drop, and collapse the in_doubt and durability
rules duplicated between the chunk notes and the durability
section down to a cross-reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client's pool configuration keys are split per pool, so the C/C++ and Rust pages describe them accordingly. Ingestion uses `sender_pool_min` / `sender_pool_max`, queries use `query_pool_min` / `query_pool_max`, and the idle lifetime key is `idle_timeout_ms`. The max default is 4 per pool, which is what the code examples and the sizing guidance now show. The at-cap borrow behavior also changes. A borrow that finds its pool at the cap waits up to `acquire_timeout_ms` (default 5000) for another thread to return a connection before failing with `invalid_api_call` / `ErrorCode::InvalidApiCall`, and `acquire_timeout_ms=0` restores fail-fast. Spelling out the wait keeps readers from building retry loops around a failure the pool already absorbs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Rust client page named several APIs without giving a way to reach them, so writing working code against the page alone required reading the crate source. Four gaps are closed. SYMBOL chunk columns had no example. The setter table described only "codes plus an i32-offset dictionary", leaving the three parallel arguments and the Arrow Utf8 dictionary layout to be inferred. A new section builds the dictionary and covers picking the code width. Validity was named but its constructor was not, so a bitmap could not be built from the page alone. Validity::from_bitmap now has an example, along with the constraint that bit_len equals the column data length. Terminal::ExecDone was described without naming Cursor::terminal(), the accessor that returns it. The DDL section now shows the match, and states that terminal() is None until the stream ends and that Terminal is non-exhaustive. The chunk at_* setters take raw epoch i64 values, which sent readers to hand-rolled SystemTime arithmetic. The page now points at TimestampNanos::as_i64(). Every snippet compiles, and the validity and symbol examples were run against a live server: bitmap 0b0000_0101 with bit_len 3 nulls exactly row 1, and codes [0, 1, 0] resolve as documented. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The page states that the crate has one questdb::Error and ErrorCode vocabulary, but never shows how to construct one, so Error::new was reachable only by reading the crate source. Adds the constructor with an example, and the guidance that goes with it: ErrorCode covers client, transport, and server failures, and no variant stands for an application rule such as a polling deadline, so those belong in the application's own error type. questdb::Error is Send + Sync and implements std::error::Error, so it propagates into a boxed error, anyhow, or thiserror with `?`, and one worker thread can report both kinds through a single type. Also records that ErrorCode is non-exhaustive, which a downstream match has to account for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move edge cases, error codes, and legacy comparisons out of the intro sections: lazy-connection details into the pool section, pool exhaustion into the settings table, compression and reset_symbol_dict next to their use sites, and the 4.x Sender refutation down to one sentence. All facts cross-checked against py-questdb-client (origin/jh_experiment_new_ilp). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.Add per-language file moves and HTTP-side URL redirects in a separate restructure PR.Redirects now handled in questdb/questdb.io#2917.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code