Skip to content

review(content-lane): fix unquoteYamlScalar's quote/comment ordering bug in duplicates.ts and source-evidence.ts #10290

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/review/content-lane/duplicates.ts and src/review/content-lane/source-evidence.ts each define
their own independent, near-identical copy of a minimal YAML frontmatter parser (parseSimpleFrontmatter),
including their own copy of a small unquoteYamlScalar helper. This duplication is intentional and
documented — both files are headed "SELF-CONTAINED NATIVE PORT (reviewbot→loopover convergence)" and
are deliberately kept self-contained rather than merged into one shared module — but the two copies of
unquoteYamlScalar are supposed to behave identically, and a comment in source-evidence.ts explicitly
says the block/sequence-parsing branch is "byte-equivalent to duplicates.ts's parser (#8016)".

Today, in BOTH files, unquoteYamlScalar has an operation-ordering bug:

function unquoteYamlScalar(value: string): string {
  const trimmed = value.trim();
  if ((trimmed.startsWith('"') && trimmed.endsWith('"')) || (trimmed.startsWith("'") && trimmed.endsWith("'"))) {
    return trimmed.slice(1, -1).trim();
  }
  return trimmed.replace(/\s+#.*$/, "").trim();
}

It checks whether the value is quote-wrapped BEFORE stripping a trailing inline YAML comment. For a
quoted frontmatter value with a trailing comment — e.g. a title: field written as
title: "My Skill" # published 2024trimmed is "My Skill" # published 2024. It starts with "
but does not end with " (it ends with the digit 4 from the comment), so the quote-stripping
branch is skipped entirely. Execution falls through to the final line, which strips the trailing
comment (.replace(/\s+#.*$/, "")) but returns "My Skill"the surrounding quote characters are
never removed.
The parsed field value is left as the 10-character literal string "My Skill"
(including both " characters) instead of the intended 8-character My Skill.

The same file (source-evidence.ts) already contains a SIBLING function, unquoteYamlValue, that gets
the order right:

function unquoteYamlValue(value: string): string {
  const trimmed = stripYamlComment(value);
  if ((trimmed.startsWith('"') && trimmed.endsWith('"')) || (trimmed.startsWith("'") && trimmed.endsWith("'"))) {
    return trimmed.slice(1, -1).trim();
  }
  return trimmed.trim();
}

unquoteYamlValue strips the comment first (via stripYamlComment, which both files already define
identically), THEN checks for surrounding quotes — so for the same input,
stripYamlComment('"My Skill" # published 2024') correctly yields "My Skill", which then correctly
matches the quote-wrapped check and un-quotes to My Skill.

This is a real, demonstrable parser defect, not a hypothetical: parseSimpleFrontmatter's own doc
comment states its job is to "capture each top-level field's value REGARDLESS of scalar style" for
duplicate-detection and signal extraction, and duplicates.ts's extractContentDuplicateSignals
stores the raw parsed value directly (title: fields.title || "") as part of the exported
ContentDuplicateSignals type used in the content-lane's duplicate-matching pipeline. A quoted title
(or any other scalar frontmatter field) with a trailing inline comment silently retains stray quote
characters that do not belong in the parsed value.

Requirements

  • In src/review/content-lane/duplicates.ts, fix unquoteYamlScalar so it strips a trailing YAML
    inline comment BEFORE evaluating whether the value is quote-wrapped, matching the correct order
    already used by source-evidence.ts's own unquoteYamlValue (reuse the file's own existing
    stripYamlComment helper for this — do not inline a second copy of the comment-stripping regex).
  • In src/review/content-lane/source-evidence.ts, apply the identical fix to its own separate copy of
    unquoteYamlScalar (this file already has stripYamlComment defined and already demonstrates the
    correct pattern via its sibling unquoteYamlValue — mirror that exact pattern for unquoteYamlScalar
    too).
  • The fix must not change behavior for any input that does not combine BOTH a quote-wrapped scalar AND
    a trailing inline comment — an unquoted value with a comment, a quoted value with no comment, and a
    plain unquoted/unadorned value must all continue to parse exactly as they do today in both files.
  • Do not merge the two files' parsers into one shared module and do not remove either file's
    "self-contained port" header comment — that architectural boundary is deliberate (see both files'
    module headers) and is out of scope for this fix.

Deliverables

  • src/review/content-lane/duplicates.ts's unquoteYamlScalar strips the trailing comment before
    checking for surrounding quotes, verified by a new test asserting that
    parseSimpleFrontmatter('---\ntitle: "My Skill" # published 2024\n---\n').title equals exactly
    "My Skill" the STRING VALUE My Skill (8 characters, no quote characters), not the literal
    10-character string including quotes.
  • src/review/content-lane/source-evidence.ts's unquoteYamlScalar gets the identical fix,
    verified by an equivalent new test in that file's own test suite covering the same
    quoted-value-with-trailing-comment case, asserting the parsed field value has no residual quote
    characters.
  • Both existing tests are added WITHOUT weakening or removing any existing test for
    parseSimpleFrontmatter, unquoteYamlScalar, unquoteYamlValue, or stripYamlComment in either
    file's test suite — the fix must not regress any already-passing case (an unquoted value with a
    comment, a quoted value without a comment, a block/folded scalar, a sequence value).

Both Deliverables (the fix in duplicates.ts AND the identical fix in source-evidence.ts) are
required in the same PR — this is the same bug, copy-pasted into two files, and fixing only one copy
does not resolve this issue.

Test Coverage Requirements

This repo's Codecov patch gate requires 99%+ patch coverage on every changed line and branch under
src/**. Both new regression tests above must exercise the exact failure mode described (a
quote-wrapped scalar with a trailing inline # comment) so the fix is locked in, not just the happy
path. duplicates.ts and source-evidence.ts are both under src/review/content-lane/**, inside
src/**, so this is fully gated by Codecov's patch coverage requirement — there is no
coverage.include exclusion applicable here.

Expected Outcome

A quoted YAML frontmatter scalar value with a trailing inline comment (e.g.
title: "My Skill" # published 2024, description: 'A short blurb.' # internal note) parses to its
clean, unquoted string value in both duplicates.ts's and source-evidence.ts's frontmatter parsers,
matching the already-correct behavior of source-evidence.ts's own unquoteYamlValue sibling
function. Duplicate-detection signals derived from a quoted-and-commented title/description field no
longer carry stray literal quote characters.

Links & Resources

  • src/review/content-lane/duplicates.tsunquoteYamlScalar (around line 57) and
    parseSimpleFrontmatter (around line 73).
  • src/review/content-lane/source-evidence.tsunquoteYamlScalar (around line 109) and its already-
    correct sibling unquoteYamlValue (around line 101), which is the exact pattern to mirror.
  • src/review/content-lane/duplicates.ts's extractContentDuplicateSignals (around line 274) — the
    real, live consumer of the raw (buggy) parsed title field.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions