Skip to content

Include the long command example for Windows Powershell in documentat… - #501

Merged
aaronmondal merged 1 commit into
TraceMachina:mainfrom
rockymaven:add-windows-long-command
Dec 18, 2023
Merged

Include the long command example for Windows Powershell in documentat…#501
aaronmondal merged 1 commit into
TraceMachina:mainfrom
rockymaven:add-windows-long-command

Conversation

@rockymaven

@rockymaven rockymaven commented Dec 18, 2023 •
edited by allada
Loading

Copy link
Copy Markdown
Contributor

Description

On Ubuntu and MacOS, they use "" character for line truncations of a long command.
But the problem is that on Windows, we use "^" character for line truncations on Command Prompt Tool and "`" character on Powershell Tool. Most users like to copy the whole lines of command without editing when they first approach this project.
More detailed description can be found in the issue below.

Fixes #500

Type of change

  • This change requires a documentation update

How Has This Been Tested?

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

allada
allada approved these changes Dec 18, 2023
View reviewed changes

@allada allada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r1, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @steed924)

aaronmondal
aaronmondal approved these changes Dec 18, 2023
View reviewed changes

@aaronmondal aaronmondal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

:lgtm:

Reviewed all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @steed924)

@aaronmondal
aaronmondal merged commit 0536d8e into TraceMachina:main Dec 18, 2023
MarcusSorealheis pushed a commit to MarcusSorealheis/nativelink that referenced this pull request Nov 3, 2025
@rockymaven
Include command example for Powershell in documentation files (TraceM…
a4db0ea 
…achina#501)
rejuvenile added a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026
@rejuvenile @claude
TraceMachina#501 (narrow scope): add 30s soft-warn observability laye…
ce0125d 
…r + folds TraceMachina#509 named-constant fix

Adds a per-site soft-warn observability layer at the three commit-watchdog
sites (v1 reaper, v2 sibling, BazelChunkedDispatcher AwaitCommit). The
soft-warn deadline is `PIN_TIMEOUT_SECS / 4 = 30s` — half of the existing
60s infra-integrity watchdog (now named `CHUNKED_COMMIT_WATCHDOG_SECS` at
the BazelChunkedDispatcher site per the TraceMachina#509 fold-in). Soft-warn is purely
observability: it emits a `warn!` log line + bumps a new
`commit_watchdog_soft_warn_total` counter via per-site `SoftWarnSet` dedup,
then continues awaiting infra-integrity. None of the 4 load-bearing
infra-integrity side effects (`discard_partial_best_effort`,
`failed_commit_sink`, in-flight removal, `JoinHandleDropGuard` abort)
change behavior or timing.

This is the narrow-scope TraceMachina#501 — post-mortem layer deferred. The previous
full Option A implementation (`6d7886c9`, separate worktree) was rejected
by all 5 reviewers (BLOCK / RECONSIDER-PREMISE); the critical finding was
that `force_dump_thread_stacks` called inside `tokio::spawn` blocks tokio
workers (CLAUDE.md HARD RULE violation). Dropping the post-mortem entirely
avoids that failure mode. Motivation for the scope reduction: 0 production
firings of the 60s infra-integrity watchdog in the 48h preceding the
review — operator demand for the early signal is weaker than feared.

Folds in TraceMachina#509 (`replace hardcoded 60 literal in BazelChunkedDispatcher
AwaitCommit watchdog`): the
`Duration::from_secs(60)` literal at the AwaitCommit site is replaced with
`Duration::from_secs(CHUNKED_COMMIT_WATCHDOG_SECS)` while we're touching
the same lines for the soft-warn select. Same numeric value (60s), now
grep-able + parametric.

Invariant being violated: operators have no early signal that a chunked
commit is approaching the 60s infra-integrity watchdog.

Mechanism that violates it: today's single 60s watchdog fires destructively
(synthesizes Err, removes from in-flight, fires failed_commit_sink) with
no advance observability.

Mechanism that re-establishes it: add a 30s soft-warn observability layer
at each of the 3 commit-watchdog sites. Soft-warn is one-shot per digest,
capped via `SoftWarnSet` (CAP=10_000 per site), fires `warn!` + bumps
`commit_watchdog_soft_warn_total`. Infra-integrity 60s timer is
byte-identical to today. Compile-time guard `_ASSERT_WATCHDOG_ORDERING`
pins `soft_warn(30) < infra_integrity(60) < pin_TTL(120)` against any
future regression.

Composite invariants this fix interacts with:
- WatchdogTimeoutSignal discriminator (`chunked_client.rs:618-628`) —
  UNCHANGED. v2 site preserves the TraceMachina#508 discriminator attachment on
  watchdog Err.
- PIN_TIMEOUT_SECS = 120s ceiling — soft-warn fires at
  `PIN_TIMEOUT_SECS / 4 = 30s`, well inside the pin window. Promoted
  `PIN_TIMEOUT_SECS` from module-private to `pub` so the derivation is
  grep-able.
- TraceMachina#509 BazelChunkedDispatcher AwaitCommit hardcoded-60 → named constant
  (folded in here).

Tests that prove the fix re-establishes the invariant: 6 tests in
`nativelink-service/tests/chunked_commit_soft_warn_test.rs`, 2 per site
(under-action: counter bumps at 30s with wedged-driver / no-publisher
fixture; over-action: counter stays at 0 when commit completes before
30s). All paused-time. All read the REAL counter
(`metrics.commit_watchdog_soft_warn_total.load(Relaxed)`) — not a
constant tautology (`assert!(CONST > 10)`), which the prior TraceMachina#501 reviewer
flagged. Each test mutation-verified per CLAUDE.md TDD step 5 with
bespoke `TraceMachina#501 narrow-scope` failure messages.

Sibling-audit note: a fourth commit-watchdog site exists at
`await_inflight_commit_with_watchdog` (`chunked_write_handler.rs:4286`),
used by the v1 worker `write_chunked` AwaitCommit path post-TraceMachina#447. The
TraceMachina#501 design audit named only 3 sites; this fourth site is NEW since the
audit. Per "narrow scope", it is NOT covered by this commit. Tracker
follow-up recommended to extend soft-warn coverage to the 4th site once
the 3-site rollout is validated in production.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rejuvenile added a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026
@rejuvenile @claude
TraceMachina#510: extend soft-warn to 4th watchdog site + consolidate…
998fa1d 
… v2 constant

## Summary

Two-part follow-up to TraceMachina#501 narrow-scope (`ce0125db`):

1. **4th-site soft-warn coverage.** TraceMachina#501 narrow-scope added the 30 s
   `commit_watchdog_soft_warn_total` observability arm to 3 of 4 commit-
   watchdog sites (v1 reaper, v2 sibling, BazelChunkedDispatcher
   AwaitCommit). The 4th site — `await_inflight_commit_with_watchdog`
   (introduced by TraceMachina#447 AFTER the TraceMachina#501 design was sketched, the v1 worker
   `WriteChunked` cross-version coordination path that parks on the
   per-digest Notify) — was left bare. This commit mirrors the same
   pattern at that site: dedicated per-site `SoftWarnSet` static
   (`AWAIT_INFLIGHT_SOFT_WARN_SEEN`), `tokio::select! { biased; ... }`
   with a soft-warn arm at `CHUNKED_COMMIT_SOFT_WARN_SECS` (30 s), one-
   shot per-digest dedup, counter always bumped (load-bearing
   operator-visible signal), `warn!` with the same shape as the other 3
   sites (digest + threshold_s + mode + cross-reference), and remove-on-
   completion (Ok / Err / watchdog-elapse) so the dedup set stays
   bounded.

2. **v2 constant consolidation.** `chunked_write_handler_v2.rs`'s
   `COMMIT_WAIT_WATCHDOG: Duration = Duration::from_secs(60)` was a
   local literal numerically identical to v1's
   `CHUNKED_COMMIT_WATCHDOG_SECS`. The local literal was a silent-drift
   risk: v1's `_ASSERT_WATCHDOG_ORDERING` compile-time guard pins the
   30 < 60 < 120 ordering on v1's constant only; a future commit
   editing v2's literal standalone would have escaped the assert. The
   v2 constant is now derived from `CHUNKED_COMMIT_WATCHDOG_SECS` and
   v2 carries a new `_ASSERT_V2_WATCHDOG_TRACKS_V1` compile-time guard
   that red-fails any future divergence with the bespoke
   "TraceMachina#510 invariant: v2's COMMIT_WAIT_WATCHDOG MUST equal v1's
   CHUNKED_COMMIT_WATCHDOG_SECS" message.

## Invariant walk

> **Invariant being violated:** all 4 commit-watchdog sites should fire
> soft-warn at 30 s (`PIN_TIMEOUT_SECS/4`) for operator early-warning;
> TraceMachina#501 narrow-scope covered 3 of 4 sites; the 4th
> (`await_inflight_commit_with_watchdog`, added by TraceMachina#447 post-TraceMachina#501-design)
> was deferred. Separately: v2's local literal `60` violates the
> single-source-of-truth invariant for the chunked-commit watchdog
> deadline.
>
> **Mechanism that violates it:**
> `nativelink-service/src/chunked_write_handler.rs:4302` (pre-fix) used
> bare `tokio::time::timeout(CHUNKED_COMMIT_WATCHDOG_SECS, notified)`
> with no soft-warn arm — operator gets no signal before infra-integrity
> fires at 60 s. Separately, `chunked_write_handler_v2.rs:115` (pre-fix)
> declared `const COMMIT_WAIT_WATCHDOG: Duration =
> Duration::from_secs(60)` — a local literal not constrained by any of
> v1's ordering asserts.
>
> **Mechanism that re-establishes it:**
> `nativelink-service/src/chunked_write_handler.rs:4329-4380`: new
> `tokio::select! { biased; r = &mut watchdog_fut => break r, () = &mut
> soft_warn_at, if !soft_warned => { ... bump + warn! + dedup-insert
> ... } }` plus drain-on-completion. The new
> `AWAIT_INFLIGHT_SOFT_WARN_SEEN` per-site dedup set lives at
> `chunked_write_handler.rs:412-418`. Caller at
> `chunked_write_handler.rs:1066-1071` threads `&self.metrics`.
> Separately, `chunked_write_handler_v2.rs:125` derives the v2 constant
> from v1; new `_ASSERT_V2_WATCHDOG_TRACKS_V1` compile-time guard at
> `chunked_write_handler_v2.rs:131-141` pins same-value invariant.
>
> **Composite invariants this fix interacts with:**
>   - `WatchdogTimeoutSignal` discriminator contract
>     (`chunked_client.rs:618-628`) UNCHANGED at all 4 sites; soft-warn
>     does NOT touch the watchdog Err arm's
>     `encode_watchdog_timeout_signal_any(...)` call so
>     `classify_retryable` still returns `Retry { WatchdogDeadline }`.
>   - `PIN_TIMEOUT_SECS = 120 s` ceiling — soft-warn fires at
>     `PIN_TIMEOUT_SECS/4 = 30 s`, well within ceiling at this 4th site
>     too. v1's `_ASSERT_WATCHDOG_ORDERING` pins 30 < 60 < 120;
>     v2's new `_ASSERT_V2_WATCHDOG_TRACKS_V1` pins v2's 60 == v1's 60,
>     transitively inheriting the v1 ordering.
>   - Per-site `SoftWarnSet` decision from TraceMachina#501 (cross-site sharing
>     rejected by red-team) — preserved; 4th site owns its own static.
>   - `commit_watchdog_soft_warn_total` metric is now aggregated across
>     4 producers (was 3); metric help-text updated to enumerate.
>
> **Tests that prove the fix re-establishes the invariant:** 2 new
> tests in `nativelink-service/tests/chunked_commit_soft_warn_test.rs`
> (`await_inflight_commit_soft_warn_fires_once_per_digest_at_30s` for
> under-action; `await_inflight_commit_soft_warn_does_not_fire_on_fast_resolve`
> for over-action). Both use REAL counter reads, paused-time virtual
> clock, and `tokio::time::timeout(Duration::from_secs(10))` real-time
> deadlock detectors with bespoke `.expect("TraceMachina#510: ...")` messages.
> Mutation-verified per CLAUDE.md TDD step 5: commenting out the
> counter bump red-fails the under-action test with "TraceMachina#510: await_inflight
> soft-warn did not fire at 30s"; moving the bump out of the select arm
> red-fails the over-action test with "TraceMachina#510 over-action: await_inflight
> soft-warn fired on fast-commit path"; mutating v2's literal to
> `CHUNKED_COMMIT_WATCHDOG_SECS + 1` red-fails the compile-time
> `_ASSERT_V2_WATCHDOG_TRACKS_V1` with the bespoke "TraceMachina#510 invariant"
> message. All 3 mutations verified at authorship time.

## Sibling audit

Workspace-wide grep for `tokio::time::timeout(...WATCHDOG...)`:
exactly 4 sites, all using `CHUNKED_COMMIT_WATCHDOG_SECS` (or
`COMMIT_WAIT_WATCHDOG` which now references it). No new sites have
appeared since TraceMachina#501 shipped. No stray `Duration::from_secs(60)`
literals near a chunked watchdog.

## Test plan

- `cargo check --workspace --features chunked_fast_slow,test-utils
  --tests`: clean (0 errors).
- `cargo test -p nativelink-service --features chunked_fast_slow,test-utils
  --test chunked_commit_soft_warn_test`: 8 passed (6 from TraceMachina#501 + 2 new).
- `cargo test -p nativelink-service --features chunked_fast_slow,test-utils
  --test chunked_write_handler_test`: 25 passed.
- `cargo test -p nativelink-service --features chunked_fast_slow,test-utils
  --test chunked_write_handler_v2_test`: 26 passed.
- `cargo test -p nativelink-service --features chunked_fast_slow,test-utils
  --test bazel_facing_internal_chunking_test`: 14 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rejuvenile added a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026
@rejuvenile @claude
specs: extend TLA+ for concurrent slow R/W + read-during-write + TLC …
8fdbbd6 
…runs

Brings the specs/ directory up to current origin/main + adds the new
StreamingBlobReadDuringWrite spec covering N concurrent slow readers x
slow writers with read-during-write over the streaming_blob primitive.

WHAT LANDED

1. ChunkRaceReadersV2.tla: appended "RECENT PROTOCOL EVOLUTION" block
   walking TraceMachina#447 / TraceMachina#508 / TraceMachina#502 / TraceMachina#500 / TraceMachina#487 / TraceMachina#503 / TraceMachina#504 and stating
   why each commit did or did not require a spec edit. No model changes.

2. Layer-1 verify_tla.sh gate fixes (specs-only, no tooling edits):
   - Added ChunkRaceReadersV2Fixed.cfg + ChunkRaceReadersV2Bugged.cfg
     so the gate's *Fixed.cfg / *Bugged.cfg classifier recognises a
     canonical pair for the ChunkRaceReadersV2 spec.
   - Removed legacy specs/ChunkRaceReadersV2.cfg (duplicate of Fixed;
     classifier read *V2.cfg as bugged and FAIL'd on the clean run).
   - Renamed ChunkRaceReadersV2_BuggedH3.tla ->
     ChunkRaceReadersBuggedH3.tla (and its three cfgs in tandem) to
     drop the shared `ChunkRaceReadersV2_` prefix; the gate's
     `${name}*.cfg` discovery glob no longer cross-runs H3-specific
     cfgs (referencing `H3HazardStateUnreachable`) against the
     canonical ChunkRaceReadersV2 spec that does not define that
     invariant.
   - Added ChunkRaceReadersBuggedH3Fixed.cfg + ChunkRaceReadersBuggedH3Bugged.cfg
     to satisfy the gate's per-spec Fixed+Bugged pair requirement on
     the renamed module.

3. NEW spec StreamingBlobReadDuringWrite.tla: models concurrent
   slow-writers x slow-readers x read-during-write over the
   streaming_blob primitive (nativelink-util/src/streaming_blob.rs).
   Captures the post-TraceMachina#500 + post-TraceMachina#502 wire-shape contract between
   StreamingBlobWriter::{send,send_eof,send_error,Drop} and
   StreamingBlobReader::next_chunk.

   CONSTANTS toggle the fix gates: ShortShieldOn (TraceMachina#502 boundary
   defense), SilentZeroPropOn (TraceMachina#500 producer-Err propagation),
   AllowShortEof/AllowErrEof/AllowDropWithoutEof for writer fault
   injection.

   Safety invariants added: NoSilentZeroToReader, NoSilentShortToReader,
   NoCorruptBytesToReader, PostFixReaderOutcomeIsTwoWay,
   ReaderOkImpliesWriterFullOk, PostFixErrPropagationReachesReader,
   PostFixFullOkReachesReaderAsOk, CursorAtMostChunkCountAtTerminal.

   Liveness: WriterEventuallyTerminates, ReadersEventuallyTerminate,
   EventualConsistencyOkFull, NoEternalParkAfterWriterDone.

   Configs:
   - StreamingBlobReadDuringWriteFixed.cfg (all gates ON, all writer
     fault modes ON) — clean, 795 states.
   - StreamingBlobReadDuringWriteBugged.cfg (ShortShieldOn=FALSE) —
     NoSilentShortToReader violated, 24 states.
   - StreamingBlobReadDuringWrite_BuggedSilentZero.cfg
     (SilentZeroPropOn=FALSE) — NoSilentZeroToReader violated, 32 states.
   - StreamingBlobReadDuringWrite_1w_3r_3c.cfg — RDW concurrency
     (17 740 states clean).
   - StreamingBlobReadDuringWrite_1w_4r_2c.cfg — more readers
     (70 743 states clean).

TLC RESULTS

verify_tla.sh full-run (Layer-1 gate): 19/19 specs PASS, 0 FAIL.

All NEW invariants are falsified by their Bugged.cfg variants with
SPECIFIC named-invariant violations (no generic is_err() catches).
See .claude/audits/tla-slow-rw-rdw-2026-05-16.md for the falsification
matrix and per-cfg trace shapes.

NO production source touched — specs/ + audit doc only.

VERIFIED AGAINST origin/main: 48be7c6 + TraceMachina#501 + TraceMachina#510 (post-rebase)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rejuvenile added a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026
@rejuvenile @claude
#chunked-v1-removal: complete v1 WriteChunked deletion (orphan fn + s…
ed904bc 
…tale docs)

Completes the e5e9ba28 v1 WriteChunked upload-path removal by deleting the
dead-code orphan the review cadre flagged (distsys MINOR-1, compiler-confirmed
`never used`) and every doc-comment left pointing at a deleted symbol.

Removed:
- `await_inflight_commit_with_watchdog` (module-private) + its
  `await_inflight_commit_with_watchdog_for_test` shim. Its ONLY caller was the
  deleted v1 `write_chunked_inner`; the Bazel path inlines its own notify-wait
  and v2 has its own awaiter.
- The now-orphaned `AWAIT_INFLIGHT_SOFT_WARN_SEEN` pub static (its sole
  consumer was the deleted function) + its doc-comment.
- The 2 SITE-4 `chunked_commit_soft_warn_test` cases that drove the orphan via
  the `_for_test` shim (test theatre for a deleted path):
  `await_inflight_commit_soft_warn_fires_once_per_digest_at_30s` and
  `await_inflight_commit_soft_warn_does_not_fire_on_fast_resolve`, plus the
  SITE-4 banner and the two now-unused test imports.

Doc-comment refs updated (backticked prose, not rustdoc links):
- 3 refs to the deleted `await_inflight_commit_with_watchdog` (soft-warn const
  doc, per-site SoftWarnSet doc, metric help text) reworded to the 3 remaining
  sites (v1 reaper, v2 sibling, BazelChunkedDispatcher AwaitCommit).
- `bytestream_server.rs` + `bytestream_server_test.rs` refs to the deleted
  `bounded_drain_grpc_stream` reworded (drop the dangling symbol / point at the
  live `drain_write_request_stream_to_eof`).
- 2 `chunked_write_handler.rs` refs to the deleted `chunked_client.rs:209`
  `client.write_chunked(..)` updated to `chunked_client.rs:227`
  `client.write_chunked_v2(..)`.

Invariant-walk:
- Invariant being violated: the tree must contain no dead/orphaned code or
  doc-comments referencing symbols removed with the v1 WriteChunked path
  (deletion-completeness of e5e9ba28).
- Mechanism that violates it: `chunked_write_handler.rs` `await_inflight_commit_with_watchdog`
  (+ shim + dedicated `AWAIT_INFLIGHT_SOFT_WARN_SEEN` static + 4 doc-comments)
  survived e5e9ba28 as dead code — sole caller was the deleted v1 handler, so
  the compiler emitted `function ... is never used`.
- Mechanism that re-establishes it: delete the function + shim + orphaned
  static + SITE-4 test-theatre cases; reword every doc-comment naming a deleted
  symbol.
- Composite invariants this interacts with: TraceMachina#501 per-site soft-warn
  observability on the 3 LIVE sites (v1 reaper, v2 sibling, BazelChunkedDispatcher
  AwaitCommit) stays intact and covered; `SoftWarnSet` /
  `commit_watchdog_soft_warn_total` remain live and exercised; the LIVE Bazel
  dispatch path (`BazelChunkedDispatcherImpl::dispatch`) and its tests are
  untouched.
- Test that proves the invariant: `chunked_commit_soft_warn_test` (6/6 pass;
  SITE 1/2/3) with `--features chunked_fast_slow,test-utils`; release
  `cargo check` no longer emits the `never used` warning; and
  `git grep -w await_inflight_commit_with_watchdog -- '*.rs'` is empty.

Behavior changes: none (dead-code + doc only). The deleted function had no
production caller (v1 path removed in e5e9ba28); the deleted static and test
cases served only that function; all doc edits are prose. No change to any LIVE
store/worker/service composition.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@allada allada allada approved these changes
@aaronmondal aaronmondal aaronmondal approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Include the long command example for Windows Powershell in documentation files

3 participants

@rockymaven @allada @aaronmondal