feat(connectors): add agent docs, per-batch observability, atomic state

[spetz]

Connector work lacked a concise agent-facing reference, and
runtime behavior around drops, shutdown, and state writes was
hard to reason about or debug. Per-batch latency visibility
was missing, some failure paths were under-accounted, hot
paths still paid avoidable metric lookup costs, and source
state persistence was not crash-atomic.

Add a repo-level AGENTS entry point plus .claude connector
skills anchored to concrete symbols so authoring rules survive
refactors. Extend the runtime with opt-in benchmark events,
always-on stage histograms, aligned sink/source stage labels,
JSON or text log selection, cached metric labels and counters
on hot paths, explicit filtered-message accounting, safer
source shutdown, and crash-atomic state saves with durable
parent-directory sync.

This keeps observability available without forcing benchmark
logs by default, improves failure-path correctness, and gives
future connector work one documented model to follow.

[codecov]

Codecov Report

❌ Patch coverage is 85.52361% with 141 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.95%. Comparing base (116103b) to head (4780631).

Files with missing lines	Patch %	Lines
core/connectors/runtime/src/state.rs	82.00%	29 Missing and 16 partials ⚠️
core/connectors/runtime/src/sink.rs	68.46%	32 Missing and 3 partials ⚠️
core/connectors/runtime/src/source.rs	73.28%	30 Missing and 5 partials ⚠️
core/connectors/runtime/src/benchmark.rs	94.32%	6 Missing and 2 partials ⚠️
core/connectors/runtime/src/stats.rs	50.00%	5 Missing ⚠️
core/connectors/runtime/src/log.rs	71.42%	4 Missing ⚠️
core/connectors/runtime/src/manager/source.rs	60.00%	4 Missing ⚠️
core/connectors/runtime/src/metrics.rs	98.90%	1 Missing and 2 partials ⚠️
core/connectors/runtime/src/configs/connectors.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3321       +/-   ##
=============================================
- Coverage     74.21%   54.95%   -19.26%     
  Complexity      943      943               
=============================================
  Files          1239     1240        +1     
  Lines        112620   102069    -10551     
  Branches      89106    78584    -10522     
=============================================
- Hits          83577    56093    -27484     
- Misses        26257    43334    +17077     
+ Partials       2786     2642      -144     

Components	Coverage Δ
Rust Core	50.23% <85.52%> (-25.13%)	⬇️
Java SDK	58.44% <ø> (ø)
C# SDK	70.12% <ø> (-0.53%)	⬇️
Python SDK	81.06% <ø> (ø)
Node SDK	91.44% <ø> (-0.10%)	⬇️
Go SDK	40.20% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/runtime/src/configs/runtime.rs	64.86% <100.00%> (+8.19%)	⬆️
core/connectors/runtime/src/error.rs	0.00% <ø> (ø)
core/connectors/runtime/src/main.rs	85.66% <100.00%> (ø)
core/connectors/runtime/src/manager/sink.rs	94.16% <100.00%> (+0.01%)	⬆️
core/connectors/sdk/src/api.rs	100.00% <ø> (ø)
core/connectors/runtime/src/configs/connectors.rs	27.67% <0.00%> (-0.51%)	⬇️
core/connectors/runtime/src/metrics.rs	99.10% <98.90%> (-0.25%)	⬇️
core/connectors/runtime/src/log.rs	29.91% <71.42%> (+4.02%)	⬆️
core/connectors/runtime/src/manager/source.rs	74.87% <60.00%> (-0.58%)	⬇️
core/connectors/runtime/src/stats.rs	72.22% <50.00%> (-2.78%)	⬇️
... and 4 more

... and 307 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hubcio]

a few non-blocking items that fall outside the changed hunks, recorded here so they don't get lost:

• core/connectors/runtime/src/configs/runtime.rs::Display for ConnectorsRuntimeConfig (around the write! near line 366) was not updated when logging: LoggingConfig was added at line 145 - operators dumping the config won't see the new format setting.
• core/connectors/runtime/src/configs/connectors.rs::Display for SinkConfig/SourceConfig (around line 313 / 332) similarly omits the new verbose and benchmark fields.
• core/connectors/runtime/src/sink.rs:42-44 imports only Instant, but Duration is reached via fully-qualified path at line 552. fold into use std::time::{Duration, Instant};.
• core/connectors/runtime/example_config/config.toml has no [logging] section and none of example_config/connectors/*.toml shows benchmark = true. operators learn the new knobs only from source. add one example each.
• the state save now does +1 dir-fsync per save vs the previous in-place code. pre-PR was 1 sync_all, post-PR is 1 sync_data + 1 parent-dir sync_all. on NVMe that's ~50-200 us extra per save - worth a one-liner in the PR body / CHANGELOG so operators don't blame the upgrade for state-save latency.

on a separate axis - there's a compound shutdown / restart bug between manager/source.rs::stop_connector and the silent-return added to source.rs::handle_produced_messages. the inline comment on source.rs covers it.

[hubcio]

this part is more focused on markdowns review that you're adding in this PR. will be a long one, so brace yourself 🦀

• CLAUDE.md / AGENTS.md direction is best left as discussion. AGENTS.md is the converging cross-tool standard (codex, cursor, aider, windsurf all read it); inverting would optimize for one ecosystem at the cost of cross-tool discoverability. cheap middle ground: add a CLAUDE.md symlink pointing to AGENTS.md so tools that hardcode the claude filename find the same content. zero churn.

• STOP-and-ask sections per skill: each SKILL.md has scattered "don't do X" / "gotchas" / "hard rules", but no consolidated 3-5 bullet list of triggers that should always surface to the user before action (e.g. bumping SDK version, changing FFI signature, renaming the default consumer group, touching state.rs save path, changing wire-format #[repr(C)] layout). a tight top-of-file STOP section would be high signal for autonomous agents.

• community pointer from skills (optional): AGENTS.md lists discord + github discussions in the support section at the bottom; individual skills don't. a one-liner at the end of each SKILL.md pointing at AGENTS.md instead of duplicating urls would be enough.

• universal-rule duplication: beyond AGENTS.md vs skills, per-area skills also restate rules (license header, verification order fmt->sort->clippy->test, tokio::sync::Mutex, zero-clone, forward-compat config). pick one owner per topic - repo-wide in AGENTS.md, connector-wide in connectors-overview, per-area skills link not restate. follow-up.

• cross-skill content duplication: benchmark + metrics block in connector-runtime (lines 75-114, 213-230) approximately equals connectors-overview (lines 165-196). SecretString pattern in sink (240-267), source (228-246), overview (155-163). verbose flag re-explained in 4 places. canonicalize and link. follow-up.

• no TOC in long files: all 7 SKILL.md plus AGENTS.md are >100 lines (largest: connector-sink 370, AGENTS.md 355, connector-source 346, connector-testing 336). short TOC at top helps partial-read previews show scope.

• no CI / pre-commit governance for .claude/skills/: this PR introduces the first AI-agent tree in the repo. nothing references .claude/ in .gitignore, .pre-commit-config.yaml, or .github/workflows/. follow-up worth scoping: validate YAML frontmatter on every SKILL.md (non-empty name: kebab-case <=64 chars, non-empty description: <=1024 chars), resolve markdown links inside .claude/skills/**/*.md, optionally grep-check that path::symbol references in skill files resolve to identifiers under core/. this catches both frontmatter drift and rename drift in one pass.

all above are follow-ups, not blockers.

BTW: in order to prepare this analysis, i used these resources:

• https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices — Anthropic skill authoring
  best practices
• https://code.claude.com/docs/en/skills — Claude Code Docs: Extend Claude with skills
• https://claude.com/blog/how-claude-code-works-in-large-codebases-best-practices-and-where-to-start — Anthropic
  blog: large codebases
• https://www.groff.dev/blog/implementing-claude-md-agent-skills — Groff: implementing CLAUDE.md + agent skills
• https://alexop.dev/posts/claude-code-customization-guide-claudemd-skills-subagents/ — alexop.dev:
  customization guide
• https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills — Anthropic
  engineering: equipping agents
• https://github.com/anthropics/skills — anthropics/skills reference repo

[spetz]

/ready

[spetz]

/ready