Streaming sideband feedback during push validation

[coopernetes]

Summary

Provide real-time feedback to the git client during push validation by streaming git sideband-2 (progress) messages as each action in the push chain executes. Today, the user sees nothing until the entire chain completes — either the push succeeds silently or a single error message appears. This makes debugging blocked pushes frustrating and opaque, especially in environments where multiple validation steps run sequentially.

Related: #1449 (Extensibility through Event Hooks) — this proposal focuses on the client-facing UX mechanism that lifecycle hooks would use to communicate progress.

Background

Git's smart HTTP protocol supports multiplexed sideband channels during git-receive-pack:

• 0x01 — pack/ack data
• 0x02 — progress (displayed as remote: ... on the client terminal)
• 0x03 — fatal error

git-proxy already uses sideband encoding in handleMessage() (src/proxy/routes/index.ts), but only for a single error message after the chain has finished. The 0x02 byte used there is actually the progress channel — the fatal error channel is 0x03.

Services like Heroku stream build output through sideband-2 during push, providing a rich interactive experience. git-proxy should do the same for its validation steps.

Current Architecture & Proposed Changes

Current flow (src/proxy/routes/index.ts + src/proxy/chain.ts)

Client POST git-receive-pack
  → extractRawBody (buffer full request)
  → executeChain(req, res) — runs ALL steps, returns Action
  → if action.error/blocked: sendErrorResponse(res, single message)
  → if allowed: return true (proxy to upstream)

The executeChain function iterates pushActionChain and returns a completed Action. The response (res) is passed but never written to during chain execution.

Proposed flow

Client POST git-receive-pack
  → extractRawBody (buffer full request)
  → parsePush (no streaming needed yet)
  → START HTTP 200 response with sideband stream
  → run remaining chain steps, writing sideband-2 progress per step
  → if blocked/error: write sideband-3 fatal error, close
  → if allowed: proxy to upstream, relay upstream sideband, close

Extending the Action class

The Action class should gain a sideband writer that chain steps can use:

// New: writable sideband stream attached to the Action or passed alongside it
interface SidebandWriter {
  progress(message: string): void;  // sideband-2 packet
  error(message: string): void;     // sideband-3 packet
  heartbeat(): void;                // keepalive for long operations
}

// Each step in pushActionChain could optionally use it:
// (req: Request, action: Action, sideband?: SidebandWriter) => Promise<Action>

The executeChain loop would write step-level progress:

for (const fn of actionFns) {
  sideband.progress(`[git-proxy] Running ${fn.name}...`);
  action = await fn(req, action, sideband);
  if (!action.continue() || action.allowPush) {
    break;
  }
  sideband.progress(`[git-proxy] ${fn.name}... ${action.error ? 'FAILED' : 'OK'}`);
}

Sideband packet encoding

The current handleMessage function:

const handleMessage = (message: string): string => {
  const body = `\t${message}`;
  const len = (6 + Buffer.byteLength(body)).toString(16).padStart(4, '0');
  return `${len}\x02${body}\n0000`;
};

This should be refactored into a proper SidebandWriter class that:

• Writes to the Express res stream incrementally (not buffered)
• Supports both sideband-2 (progress) and sideband-3 (fatal error) channels
• Includes a heartbeat() method for keeping connections alive during long operations
• Handles flush packets (0000) correctly

Response lifecycle change

The key change: proxyFilter currently returns true (proxy) or false (already responded). With streaming, the response starts before we know the outcome. Since git protocol always uses HTTP 200, this is safe:

// Start response immediately
res.writeHead(200, {
  'content-type': 'application/x-git-receive-pack-result',
  'transfer-encoding': 'chunked',
  // ... other headers
});

const sideband = new SidebandWriter(res);

// Run chain with streaming progress
const action = await executeChainStreaming(req, res, sideband);

if (action.allowPush) {
  // Proxy to upstream and pipe response through
  await proxyToUpstream(req, res, sideband);
} else {
  sideband.error(action.errorMessage || action.blockedMessage);
}

sideband.close();
return false; // we already handled the response

Example UX

What the user sees on git push:

$ git push origin main
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 1.02 KiB | 1.02 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: [git-proxy] Push received: main (3 commits)
remote: [git-proxy] Checking repository authorization... OK
remote: [git-proxy] Checking commit messages... OK
remote: [git-proxy] Checking author emails... OK
remote: [git-proxy] Scanning for secrets (gitleaks)... OK
remote: [git-proxy] Scanning diff... OK
remote: [git-proxy] All checks passed. Forwarding to origin.
To github.com:org/repo.git
   abc1234..def5678  main -> main

On failure:

remote: [git-proxy] Push received: main (3 commits)
remote: [git-proxy] Checking repository authorization... OK
remote: [git-proxy] Checking commit messages... FAILED
remote:
remote:   Commit abc1234 missing required prefix.
remote:   Expected format: JIRA-1234: description
remote:   Got: "fix stuff"
remote:
remote:   See: https://wiki.example.com/commit-standards
remote:
fatal: remote error: Push blocked by git-proxy policy

Considerations

• Backward compatibility: The existing handleMessage / sendErrorResponse path should remain as a fallback for non-streaming contexts (e.g., GET /info/refs errors)
• Plugin compatibility: Existing plugins with signature (req, action) => Promise<Action> should continue to work. The SidebandWriter can be an optional third parameter or attached to the Action.
• Timeout handling: For chain steps that call external systems, the heartbeat mechanism prevents intermediate proxies/load balancers from killing idle connections (most default to 60s idle timeout). Sending a sideband-2 dot packet every 10s keeps the connection alive.
• express-http-proxy interaction: The current proxy middleware expects to control the response. With streaming, we'd need to handle the upstream proxy call ourselves rather than returning true from proxyFilter.

Acceptance Criteria

• SidebandWriter class with progress(), error(), heartbeat(), close() methods
• Correct sideband channel usage: 0x02 for progress, 0x03 for fatal errors
• Chain execution streams per-step progress to client terminal
• Heartbeat mechanism for long-running steps
• Upstream response relay works after chain completion
• Existing plugins continue to work without modification
• Integration test: git push shows remote: progress lines during validation

[coopernetes]

One obvious caveat to using the sideband for fast feedback... Sideband in transparent proxy is fundamentally at odds with the protocol. Sideband messages are part of the git pack protocol response and multiplexed into the response stream from server to client using packet-line framing (band 2). In a transparent proxy:

1. Client sends POST /git-receive-pack with pack data
2. Proxy intercepts, parses the pack, runs checks
3. If checks pass, proxy forwards to upstream
4. Upstream's response goes back to the client

The proxy never owns the response stream. To inject sideband messages, we'd need to:

• NOT forward to upstream yet
• Construct our own git protocol response preamble with sideband messages
• Then somehow splice in the upstream response after
• Handle packet-line framing, chunked encoding, content-length mismatches

That's essentially rebuilding the git server protocol.

We already do a bit of "storing of local state" for diff generation. What's the appetite for a "store-and-forward" method instead where GitProxy does the fetch like it does today, runs through its validation after it receives the push, sends sideband feedback in real time, then handles the last leg itself to complete a push? There's always the credential handling... In this method, anything on the local file system would have to be unauthenticated (public/read only repos in the upstream only) and pushes would use the authentication in memory only (preventing leakage).

[coopernetes]

This is aspirational. Can be reopened but I don't see getting to this any time soon. Transparent proxy + double push is the accepted UX for the time being.

[jescalada]

Reopening this for #1637 - though my implementation differs somewhat from the acceptance criteria here!