Repository navigation
Redesign live LLM: monolingual transcript cleanup + Apple-only translation; LFM2.5 as experimental live model - #13
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, ja) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd88562ce9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| static let lfm2_5_230m = ModelOption( | ||
| id: "LiquidAI/LFM2.5-230M-MLX-4bit", |
There was a problem hiding this comment.
When the user has Download from set to ModelScope, a successful load ends by calling ModelScopeDownloader.removeSnapshots(notIn: Set(ModelCatalog.all.map(\.id))) in LLMService.performLoad(). This new live-only model is intentionally not part of ModelCatalog.all, so its just-downloaded ModelScope snapshot gets pruned immediately after load while the downloaded marker has already been set; Settings can then report it as downloaded, but the next live load after an unload has no local weights (or may try the network despite .requireDownloaded). Add the live-refine candidate to the cleanup keep set.
Useful? React with 👍 / 👎.
| // is on — accepted, AttachmentDescribeQueue already treats every | ||
| // describeImage failure as best-effort. Revisit if live photo | ||
| // description quality regressions get reported. | ||
| await llm.setModel(ModelCatalog.liveRefineModel()) |
There was a problem hiding this comment.
With the LFM flag on, this leaves the shared LLMService resident on a text-only model after a recording. attachImage(_:to:) for an already-saved session still goes through enqueueAttachmentDescription, which only enqueues when await llm.model.supportsVision; therefore adding a photo after an LFM-backed recording silently skips the VLM description even if the fixed 0.8B liveModel is downloaded. The accepted OCR-only degradation was for photos attached during recording, so the saved-session path should switch to the vision-capable live model (or otherwise avoid depending on the current live-refine resident model).
Useful? React with 👍 / 👎.
| requiredHeadroom: 300_000_000, // ~151 MB weights; tune on device | ||
| downloadBytes: 151_000_000, |
There was a problem hiding this comment.
For low-memory loads, LLMService.admittedCacheLimit treats requiredHeadroom as the amount needed for the weights plus the full 256 MB MLX cache, and only falls back to the 64 MB cache below that threshold. This new model sets the threshold to 300 MB while the comment says the weights alone are about 151 MB, so a device with roughly 301–407 MB free is admitted with the full cache even though weights + cache exceed available memory; use a headroom that includes the full cache so those cases take the tight-cache path instead of risking a failed load or jetsam during recording.
Useful? React with 👍 / 👎.
…afts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c14066cfb5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // length blowout, garbled decode) is distinguishable | ||
| // from one that never generated at all. | ||
| logger.warning( | ||
| "refinement rejected, draft kept: \(raw, privacy: .public)") |
There was a problem hiding this comment.
When a refinement is rejected, raw is the model output generated from the user's transcript/draft and can include private speech content. Marking it .public writes that text into OSLog/Xcode/sysdiagnose logs on every parse or acceptability failure, which is especially likely while evaluating this experimental model; keep the diagnostic but redact it, make it private, or log only metadata.
Useful? React with 👍 / 👎.
Supersedes the plan's live-refine-only constraint at the user's request; mirrors the Bonsai gating pattern (lineup membership, snap-back on disable, text-only vision back-fill already generic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca788b1d83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } else { | ||
| Button("Download live model") { | ||
| startDownload(ModelCatalog.liveModel) | ||
| startDownload(ModelCatalog.liveRefineModel()) |
There was a problem hiding this comment.
When the LFM flag is enabled, this button now downloads only ModelCatalog.liveRefineModel() (the text-only LFM), and refreshStats also marks the live row downloaded based on that model. I checked SummaryJobCenter.refreshAttachmentText, which still switches to ModelCatalog.liveModel before describeImage for post-recording photo back-fill; if a user skipped/removed the 0.8B download and then uses Settings to download the experimental live model, summaries with photos silently keep OCR-only because the required 0.8B VLM is not present and there is no longer a Settings row prompting for it. Keep the vision-capable live model tracked/downloadable separately from the swappable live-refine model.
Useful? React with 👍 / 👎.
| /// `model.liveRefineLFM2Enabled` (off by default); the same flag also | ||
| /// lists it in the summary lineup for A/B against the Qwen tiers. | ||
| static let lfm2_5_230m = ModelOption( | ||
| id: "LiquidAI/LFM2.5-230M-MLX-4bit", |
There was a problem hiding this comment.
When Settings has Download from set to ModelScope, ModelScopeDownloader derives the API path directly from this new repo id, so enabling the LFM toggle makes the live-model download request LiquidAI/LFM2.5-230M-MLX-4bit from ModelScope. I checked the public model listings: the exact MLX repo exists on Hugging Face, while ModelScope search only surfaced other LFM2.5 repos/precisions, not this id; in the ModelScope environment the new live model therefore fails at download time with modelNotFound. Hide this candidate for ModelScope or route it to a mirror that actually hosts the MLX snapshot.
Useful? React with 👍 / 👎.
… picker Bonsai promoted from experimental flag to standard (validated on device); 0.8B and LFM2.5 are live-role-only now. Existing 0.8B summary picks snap to the 2B default at launch via normalizeStoredSelection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live refinement role is now monolingual transcript cleanup: the LLM fixes recognition errors in the source sentence (fidelity-gated, raw ASR text kept in rawSourceText — the hotword-restore storage pattern, which also makes the post-hoc restore skip these entries), and Apple's Translation framework re-translates the cleaned sentence as the refined translation. Runs in transcribe-only sessions too. The old LLM-refines-the-translation path (HistoryTurn, userPrompt, systemPrompt(direction:), parseRefinement, isAcceptable(_:draft:), recentHistory) is deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lit (zh-Hans, ja) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DakePeng
changed the title
… 'experimental' A tier admitted below CaptionPipeline.memoryShedFloor (400 MB) can load into the shed zone and thrash (load -> critical-pressure shed -> silence-gap reload). LFM2.5's headroom rises 300->500 MB and a catalog test now guards the invariant for every tier. Bonsai's picker label loses the '— experimental' suffix to match its promotion (zh-Hans, ja localized). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 50de473a21
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| store.setRefined(nil, for: entryID) | ||
| return | ||
| } | ||
| store.applyCleanedSource(cleaned, for: entryID) |
There was a problem hiding this comment.
When live refinement accepts a cleanup, this only mutates CaptionStore; by this point processFinalizedEntry has already appended a value copy of the same CaptionEntry into liveChunker, and enqueueChunkNote later builds cached live notes from that copy's sourceText. In the normal case where the chunk closes after cleanup (silence gap or budget), the saved session has cleaned transcript entries but chunkNotes covering the prefix are distilled from the old ASR text, and SummaryEngine.uncoveredEntries reuses those notes during post-session summarization instead of remapping them. Propagate the cleaned text into the live chunker or append after refinement so accepted fixes are reflected in live notes.
Useful? React with 👍 / 👎.
| // is on — accepted, AttachmentDescribeQueue already treats every | ||
| // describeImage failure as best-effort. Revisit if live photo | ||
| // description quality regressions get reported. | ||
| await llm.setModel(ModelCatalog.liveRefineModel()) |
There was a problem hiding this comment.
This re-reads the experimental LFM flag on every silence-gap load, but live-note admission is frozen once in beginSession via liveMappingStopped. If a recording starts with LFM off and the user enables the toggle from the Settings tab during that recording, the next gap swaps the resident model to LFM while live notes remain enabled, so closed chunks are sent to the text-only model that this change explicitly says cannot produce reliable TSV records instead of deferring to post-session mapping. Capture the live-refine model / live-note-disabled decision once per session and use the same snapshot here.
Useful? React with 👍 / 👎.
| func parseRefinedSentence(_ raw: String) -> String? { | ||
| parseRestoredSentence(raw) |
There was a problem hiding this comment.
By routing the broad live cleanup parser through parseRestoredSentence, every accepted refinement goes through cleanResponse, whose stripModelTags removes any <tag>-shaped text. In recordings about HTML, XML, or generic type syntax (for example, “use the <title> tag”), the parsed cleanup can drop the literal tag, and the length/similarity gate can still accept that deletion, so the saved transcript loses spoken content. Use a source-transcript parser that avoids XML stripping for live sentence output or only removes known wrapper tags.
Useful? React with 👍 / 👎.
| store.setRefined(nil, for: entryID) // transcribe-only: done | ||
| return | ||
| } | ||
| Task { [translator, store] in |
There was a problem hiding this comment.
After an accepted cleanup, this unstructured task retranslates the cleaned source while the pipeline immediately returns to the user. If the user stops the recording before this draft finishes, archive.save persists displayTranslation as the old draft generated from the raw ASR text, and the task only updates the live store (or no-ops after the next session clears it), leaving the saved session with cleaned source but stale translation. Flush or await this retranslation before saving, or update the archived record when it lands.
Useful? React with 👍 / 👎.
| glossary: matcher.noteGlossaryLines( | ||
| language: direction.source, text: text))) |
There was a problem hiding this comment.
This sends the cleanup prompt noteGlossaryLines, which only emits the canonical rendering for a matching hotword. For a hotword with aliases (for example canonical “Robert” and alias “Bob”), a correctly recognized alias still scores as a match, but the prompt now only says to use “Robert”; the model can rewrite the spoken alias to the canonical name, and the similarity gate will accept that small name expansion. Include aliases in the live cleanup glossary or otherwise avoid forcing canonical spellings when the matched form is an alias.
Useful? React with 👍 / 👎.
| if entries[index].rawSourceText == nil { | ||
| entries[index].rawSourceText = entries[index].sourceText |
There was a problem hiding this comment.
This marks every accepted live cleanup with rawSourceText, even when the cleanup only changed punctuation or a non-vocabulary word. SummaryEngine.hygienePass treats rawSourceText != nil as proof that the LLM hotword restore already handled the entry and skips the targeted restore pass; if a near-miss remains below the deterministic replacement threshold, or the user adds the vocabulary before summarizing, the post-session summary/chat pipeline never gets its chance to repair the term. Use a separate marker for broad live cleanup or let hygiene run when the matcher still detects a forced refinement.
Useful? React with 👍 / 👎.
The 230M is now THE live model — monolingual transcript cleanup is within its capability and it runs far cooler beside ASR + diarization than the 0.8B did. Consequences: live notes always defer to post-session mapping, live photos always keep OCR text until the recording ends (0.8B shrinks to the vision-only fallback), and onboarding's live download drops from ~1.3 GB to ~151 MB (zh-Hans, ja localized). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dfc42307b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// Parse the cleaned sentence — same tolerance as the hotword restore | ||
| /// ("S:" tagged or whole output). | ||
| func parseRefinedSentence(_ raw: String) -> String? { | ||
| parseRestoredSentence(raw) |
There was a problem hiding this comment.
When the live model echoes the user prompt as Sentence (English): ..., this parser treats the whole echoed line as an untagged corrected sentence. For utterances longer than roughly a few dozen characters, the existing length/similarity gate can still accept the added label, so the saved transcript can be polluted with Sentence (...) instead of just the spoken text. Strip this known prompt label or reject labeled output before applying it to sourceText.
Useful? React with 👍 / 👎.
| // it cleans the transcript itself, and a translating session gets a | ||
| // second Apple pass over the cleaned sentence when it lands. | ||
| if direction.source != direction.target { | ||
| guard await produceDraft(for: entry) != nil else { return } |
There was a problem hiding this comment.
In translating sessions where Apple's TranslationCoordinator.draft throws, such as a missing/not-yet-mounted translation session or unavailable language pack, this early return skips the monolingual cleanup queue entirely for that entry. The new cleanup is supposed to improve the source transcript independently of Apple translation, so a draft failure should mark translation unavailable but still allow source cleanup/hotword refinement when the LLM is ready.
Useful? React with 👍 / 👎.
…ated LFM2.5 sentence cleanup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fixup again — before translating Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ared transcript polish The cleanup phase counts as LLM work for the memory-warning shed (usesLLM), and imports briefly load the 230M then release it — the auto-summary reloads what it needs itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1418155d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } catch { | ||
| logger.warning( | ||
| "offline cleanup generate failed, fixed sentence kept: \(error.localizedDescription, privacy: .public)") |
There was a problem hiding this comment.
When offline cleanup is enabled and the downloaded LFM model cannot load (for example LLMServiceError.insufficientMemory from llm.generate), this generic catch keeps the loop going, so a long import/re-transcribe repeats the full load/admission retry for every utterance before falling back to translation. Treat load-level failures like modelNotDownloaded and abort the cleanup pass after the first one, while still allowing true per-sentence generation failures to be best-effort.
Useful? React with 👍 / 👎.
| logger.warning( | ||
| "offline cleanup rejected, fixed sentence kept: \(raw, privacy: .public)") |
There was a problem hiding this comment.
When a cleanup is rejected, raw is model output generated from the user's imported/re-transcribed speech and can include private transcript content. Marking it .public writes that text into OSLog/Xcode/sysdiagnose logs; keep the diagnostic but log only metadata or mark the value private.
Useful? React with 👍 / 👎.
| await refinement?.enqueue(RefinementQueue.Job( | ||
| entryID: entry.id, | ||
| source: text, | ||
| draft: draft, | ||
| direction: direction, | ||
| history: history, | ||
| glossary: matcher.glossaryLines( | ||
| direction: direction, sourceText: text))) | ||
| language: direction.source, | ||
| context: store.recentSourceTexts( | ||
| limit: 3, language: direction.source, excluding: entry.id), | ||
| glossary: matcher.noteGlossaryLines( | ||
| language: direction.source, text: text))) |
There was a problem hiding this comment.
If the user taps Stop right after a final utterance, this enqueue returns immediately and endSession() archives archivableEntries without waiting for queued or in-flight RefinementQueue jobs. The cleanup can then land only in the live CaptionStore after the archive already owns the session (and sessionID is cleared), so the saved recording keeps the raw ASR text even though the live cleanup eventually succeeded; drain/cancel-and-apply these source-cleanup jobs before archive.save or update the archived record when they land.
Useful? React with 👍 / 👎.
| let polished = try await OfflineTranscriptPolisher(matcher: hotwords?.matcher).polish( | ||
| utterances.map(\.text), | ||
| language: direction.source, | ||
| runLLMCleanup: runCleanup, | ||
| generate: { [llm] in | ||
| guard let llm else { throw LLMServiceError.modelNotLoaded } | ||
| return try await llm.generate(system: $0, user: $1, maxTokens: $2) | ||
| }, | ||
| onProgress: { onPhase(.cleaningUpTranscript($0)) }) |
There was a problem hiding this comment.
If an import is cancelled, backgrounded, or yielded to a recording while this cleanup polish call is awaiting generation, the throw skips the unload below, leaving the 230M model resident even though the import job has ended or paused. Wrap the cleanup phase in a defer so imports still release the LLM on cancellation/failure, matching the comment that imports keep no resident LLM outside cleanup.
Useful? React with 👍 / 👎.
…abels Continuous Sortformer inference beside ASR + the LLM cost more heat than live labels were worth — and the offline Pyannote pass (post-process, re-transcribe, imports) relabels everything anyway. The speaker picker now only gates the post-process pass via recordingSpeakerCount; Settings and onboarding download the offline model instead of the streaming one; the mid-utterance speaker split and its store/pipeline machinery are deleted. Sessions recorded without saved audio no longer get speaker labels (they previously got live ones). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0ba7e3129f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Applied by the offline post-process pass after the recording ends. | ||
| // Default off so nothing ever downloads the speaker model silently. | ||
| @AppStorage("captions.speakerCount") private var speakerCount = 0 |
There was a problem hiding this comment.
With speakerCount now only recorded for offline post-processing, the normal Stop → choose summary style flow can no longer label speakers: I checked the post-stop onProceed in LiveCaptionsView, and it still calls pipeline.jobs.summarize(...) directly, while postProcessAndSummarizeNewSession is only reached from SessionDetailView.requestAutoSummarize. When a user enables 2+/Auto speakers and then taps a style card after stopping, the saved session is summarized without ever running the offline diarizer, and because speakerSeparationFailed is never set there is no retry affordance either.
Useful? React with 👍 / 👎.
…lScope VoiceprintService swaps FluidAudio's Pyannote CoreML bundle for sherpa-onnx offline diarization: pyannote segmentation-3.0 + 3D-Speaker's bilingual zh/en CAM++ embedding (~34 MB total vs ~80 MB), running on the already-vendored ONNX runtime. DiarizerModelStore follows the Qwen3-ASR dual-source pattern — Hugging Face plus a ModelScope mirror — so the DiarizerSource enum (HF/HF-Mirror only) retires in favor of the shared ASRModelSource; China-region onboarding now maps the diarizer to ModelScope like everything else. Explicit speaker picks force the cluster count; Auto discovers by distance threshold. The FluidAudio package dependency is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lines Diagnostic for the 'transcription not full' report: every cleaned entry keeps its pre-LFM2.5 text in rawSourceText. Settings -> Diagnostics -> 'Show original recognition' (default off) renders it in orange under each transcript line when it differs from the displayed text. A word present in the raw line but missing above means the AI cleanup dropped it (fidelity gate too loose); missing from both means the decoder/VAD never produced it. Read-only; uses data already stored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: afbb937af8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let segments = try await voiceprint.diarizeFile( | ||
| url: url, maxSpeakers: speakerCap, source: .current | ||
| url: url, speakerCount: speakerCount | ||
| ) { progress in |
There was a problem hiding this comment.
When speakerCount == -1 (Auto), this path now passes Auto straight into the diarizer and then applies the returned segments unconditionally. The retry path in SummaryJobCenter.retryDiarization explicitly rejects Auto results with more than 8 distinct speakers, but fresh post-stop processing and manual re-transcribe still reach this helper, so an over-split Auto result can archive dozens of phantom speaker labels and clear the failure state before the user has a retry affordance. Add the same plausibility guard before applying segments or surface the same error here.
Useful? React with 👍 / 👎.
| markPendingPostProcess( | ||
| sessionID: sessionID, style: style, length: length, | ||
| suggestVocabulary: suggestVocabulary) |
There was a problem hiding this comment.
When a user consents to download the summary model and starts the new-recording post-process, this persisted marker records the style/length but drops allowDownload. If the app is killed during the ASR/diarization part before markPendingSummary runs, resumePendingPostProcesses() restarts this with the default allowDownload: false, and runRetranscribe fails immediately with modelNotDownloaded even though the user already approved the download. Persist the consent on PendingPostProcess and pass it back into the resumed request.
Useful? React with 👍 / 👎.
| generate: Generate, | ||
| onProgress: (Double) -> Void = { _ in } | ||
| ) async throws -> Output { | ||
| let fixed = texts.map { fixup($0, language: language) } |
There was a problem hiding this comment.
For imports and re-transcribes where deterministic hotword fixup changes a decoded utterance, this immediately replaces the original ASR text with the fixed form and Output.originals is only populated later for accepted LLM cleanups. If LFM cleanup is skipped/unchanged, or if it runs after the fixup, rawSourceText is nil or the already-fixed sentence rather than the untouched ASR output, so the saved record loses the raw transcript needed for audit/recovery. Track the original per index before fixup and set originals whenever the final stored text differs.
Useful? React with 👍 / 👎.
| } | ||
| let attachment = SessionRecord.Attachment( | ||
| fileName: fileName, | ||
| timestamp: .now, |
There was a problem hiding this comment.
In live recordings where JPEG downscale/encode/write takes noticeable time, this async path still stamps the attachment after saveAttachmentJPEG returns. The live caption rows interleave photos by attachment.timestamp rather than anchorEntryID, so a photo tapped beside one utterance can render and archive next to later transcript entries if speech continues during the write. Capture the tap time before starting the task and use it for the attachment timestamp.
Useful? React with 👍 / 👎.
Re-transcribe required AI features on + the LLM downloaded, at both the UI gate and the job (guard llmEnabled else throw aiDisabled), because it always re-summarizes. But the ASR, deterministic hotword fixup, diarization, and Apple translation need no LLM — only the LFM2.5 cleanup and the re-summary do, and the polisher already skips cleanup when llmCleanupEnabled is false. So requiring AI was pure over-gating and blocked re-transcribing to compare backends or refresh a transcript with AI off. Now: AI on -> unchanged (cleanup + re-summarize, prompts for the model if absent). AI off -> re-transcribe replaces the transcript with a fresh ASR + fixup pass and skips cleanup and the re-summary; the sheet footer says so. Unblocks removing Dolphin to re-transcribe with SenseVoice, and running an AI-off pass to see raw recognition. Full suite passes; both targets build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Whole utterances went missing because Dolphin (CTC) emits an all-blank path for a whole segment when the VAD crops it tight at the speech boundaries or the clip is short, and the empty result was dropped. The existing offline rescue split the segment into halves and re-decoded — built for autoregressive token-budget blanks, but for CTC that makes the pieces SHORTER and blanks harder, exactly wrong. Replace it with the standard CTC rescue: on an empty decode, re-decode the segment padded with 0.3s of silence each side (settling frames + un-clipped edges), keeping the original VAD time range so diarization/seek/checkpoints are unaffected. Applied to both the offline pass (VADSegmentedTranscriber.decodeSegment) and the live finals (SenseVoiceEngine.drainFinalizedSegments, which previously had no rescue at all). Keeps Dolphin — no second model, no extra memory, runs only on the rare blank. Removes the now-dead retryHalves + suspiciousEmptySamples; swaps RetryHalvesTests for SilencePadRescueTests. 534 tests pass; both targets build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… .balanced) Re-transcribe called OfflineTranscriber.transcribe with no sensitivity, so it always ran the offline VAD at the default .balanced (silero threshold 0.3) — even for a recording made on Meeting room / far (threshold 0.2, far more sensitive). A quiet or distant meeting was re-transcribed LESS sensitively than it was recorded, dropping the faint utterances the far preset exists to catch. Recordings don't store their preset, so there was no way to recover it. Add a Mic pickup picker to the Re-transcribe sheet (defaults to the current preset) and thread the choice through retranscribeAndSummarize -> RetranscribeRequest -> SessionRetranscriber -> OfflineTranscriber.transcribe. The auto post-process path uses the current preset (the recording's best proxy). postProcessNewRecording threads it too. 534 tests pass; both targets build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto diarization can produce a slot whose segments never win any utterance's overlap during attribution, so it labels no entry and leaves a hole in the speaker numbers (the reported 1, 3, 4, 6, 7, 8 with 2 and 5 missing). Attribution numbered by diarization-segment order, not by which slots actually landed on the transcript. Add SpeakerAttribution.denselyRenumbered: after attribution, collapse to only the slots that reached an entry, relabeled 0-based in first-appearance order. Applied at all three sites — applyDiarizationSegments (retryDiarization + applySpeakerSeparation) and the import diarization path. Speaker 1 is still whoever talks first; the labels are now gap-free. (Separate from Auto over-splitting the count itself — for a known headcount, Identify speakers -> exact number stays the reliable path.) 535 tests pass; both targets build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Segmentation (live + offline):
- hybrid finals-only live VAD cap 12s->20s; run limiter scales to 2x
- live caption runs join by char budget, not punctuation (SenseVoice
over-punctuates cut fragments, Dolphin emits none); card gap 12->30s
- offline imports/re-transcribe attribute speakers on RAW utterances
first, then merge speaker-aware (heal pause-broken sentences; join a
speaker's consecutive sentences without fusing turn changes)
- SessionDetailView flows same-speaker entries into display paragraphs
- CJK joiner treats CJK punctuation/fullwidth as CJK (no stray spaces)
LFM2.5 cleanup (was rejecting 66/66 on device):
- pre-seed "S: " response prefix + shorter prompt so the 230M stops
echoing instructions / narrating; drop repetitionPenalty for the
copy-shaped cleanup task; rejection logs now name the failing check
Summary:
- fix source-id leak ("m004 m009 ...") into detail bullets:
strippedSourceIDTokens now collapses whitespace-separated id runs;
render/reduce drop records that scrub to empty
Background/foreground job resume:
- flip paused->queued synchronously on foreground; drain worker
outwaits cancelled predecessors (uncancellable sherpa decodes) so a
resumed session never races its own prior run's checkpoint writes
- summarize path got the same fix (was dropping summaries on 2nd hop)
- un-wedge the job center when a recording fails to start
Progress + task-management UX:
- .preparing / .coolingDown / .downloadingSpeakerModel activity states
replace lying "Transcribing 0%" and frozen thermal-hold bars
- setProgress clears stale ETA on every phase; monotonic display floor;
checkpoint-seeded resume (no 89->0 sawtooth); ASR caps at 99% until
decode drain; terminal translate ticks; "Finalizing summary"
- detail view shows progress for every job + a cancel button; wand
shows parked glyphs; search rows keep status; persisted import errors
Menu + re-transcribe options:
- group the session menu into intent sections; inline summary style/
length pickers (removed the SummaryOptionsSheet round-trip)
- make the re-transcribe pre-flight fully tunable in place (engine,
speakers, translate-to, mic pickup)
Task-conflict UI:
- every post-hoc action now shows a notice when a recording blocks it
instead of silently no-op'ing; guard Suggest hotwords (was swapping
the live model mid-recording); SelectorSheet locked-note banner
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- applySpeakerSeparation (re-transcribe path) now refuses an over-split Auto diarization result, matching retryDiarization; the bound is shared via SessionRetranscriber.isImplausibleAutoResult so the two can't drift (removed the duplicate constant in SummaryJobCenter) - PendingPostProcess persists allowDownload, so a kill during the ASR/diarization part of a new-recording post-process resumes with the user's download consent instead of failing "model not downloaded" - OfflineTranscriptPolisher preserves the true pre-fixup ASR as the entry original whenever deterministic fixup changed a line, even when LLM cleanup is skipped/aborted/unchanged (was lost before) Already addressed by earlier work on this branch (verified): rejected refinement/cleanup output is logged .private not .public; the offline attribute-then-merge reorder keeps VAD-fragment merges from fusing two speakers; direct retranslate/diarize jobs are cancelled-and-cleared on background/yield instead of stranding the session busy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- endSession now cancels the RefinementQueue (not just the downstream re-translations) before archive.save, so a queued/in-flight cleanup can't mutate the store AFTER the snapshot and leave the saved record holding raw text the screen had already upgraded — matching the comment's own promise of "a store no task will mutate afterwards" - cover the parked/started activity states in the background-suspend classification test (.preparing, .downloadingSpeakerModel, and the already-parked cases) Verified already-handled (from the review's Codex-P2 list): legacy hfMirror diarizer setting migrates to ModelScope (DiarizerModelStore); speaker controls are #if os(iOS) in ImportAudioSheet + SettingsView; post-recording photo descriptions are back-filled by the summarize pass (refreshAttachmentText), not lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e722d705f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| slots = SpeakerAttribution.denselyRenumbered( | ||
| SpeakerAttribution.attribute( | ||
| utterances: utterances.map { ($0.start, $0.end) }, | ||
| to: segments)) |
There was a problem hiding this comment.
For imports with speakerCount == -1, this path applies every segment returned by sherpa directly to the transcript. The standalone retry and re-transcribe paths now reject Auto results over SessionRetranscriber.maxPlausibleAutoSpeakers, but imported long/noisy audio can still archive dozens of phantom speakers here and leave speakerSeparationFailed unset, so the user gets no retry affordance; apply the same plausibility guard before attributing slots.
Useful? React with 👍 / 👎.
| if var last = runs.last, | ||
| let previous = last.entries.last, | ||
| previous.state != .volatile, | ||
| entry.state != .volatile, | ||
| previous.id != lastEntryID, |
There was a problem hiding this comment.
When the user changes spoken language or translation target mid-recording, adjacent finalized entries can have different directions, but this join condition still folds them into one CaptionRun. The folded row keeps the first entry's direction, so a target on/off change can make CaptionRow hide later translations or show only a partial translation under the combined source text; require the previous and next entries to have the same direction before joining.
Useful? React with 👍 / 👎.
| if wantsRefinement, llmEnabled, !isBackgrounded, | ||
| thermal.policy == .full, await llmIsReady() { |
There was a problem hiding this comment.
When a recording starts while the shared LLM is already ready with the summary/photo model, this readiness check lets RefinementQueue run sentence cleanup on that resident model because the queue never calls setModel(ModelCatalog.liveRefineModel). The first silence-load may swap later, but early finalized utterances can still use the heavy/wrong model during live capture; gate on the current model or set the live-refine model before accepting cleanup jobs.
Useful? React with 👍 / 👎.
…ng, live model Three comments Codex posted after the last push: - FileImportEngine: the Auto-diarization plausibility bound now also guards the IMPORT path (the earlier fix only covered re-transcribe and the standalone retry) — a long/noisy import can no longer archive dozens of phantom speakers with no retry affordance; over-split Auto results set speakerSeparationFailed and leave slots unlabeled. - CaptionRunGrouping: require the same `direction` before folding adjacent entries into a run. A mid-recording spoken/translation language change gave neighbours different directions; the folded row kept the first entry's direction and hid or truncated the later translation. - CaptionPipeline: live cleanup now gates on `liveRefineReady()` — the resident model must be the live-refine model, not a summary/photo model left loaded from before the recording (llmIsReady alone was true for any ready model, so early utterances could clean on the wrong, heavy model until the first silence-gap load swapped it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65350323af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Self.applyDiarizationSegments(segments, to: &updated) | ||
| updated.speakerSeparationFailed = nil |
There was a problem hiding this comment.
When the offline diarizer returns no usable segments (for example, short/noisy audio or VAD finding no speaker regions), this helper still applies the empty result and then clears speakerSeparationFailed, so fresh post-process or manual re-transcribe can finish with zero labels and no retry affordance while also dropping cached summary/notes. The standalone retry path already treats labeled == 0 as an error; add the same guard here before clearing the failure state/caches.
Useful? React with 👍 / 👎.
| if var last = paragraphs.last, let previous = last.last, | ||
| entry.timestamp.timeIntervalSince(previous.timestamp) <= maxStartGap, | ||
| last.reduce(entry.sourceText.count, { $0 + $1.sourceText.count }) | ||
| <= maxCharacters { |
There was a problem hiding this comment.
When a saved session contains adjacent entries with different directions, such as a spoken/target language change mid-recording or Auto detecting a code-switch, this new paragraph grouper still folds them solely by time and length. The detail transcript then renders one combined source with compacted translations, so a translation-off or target-language change can hide a line's translation or mix target languages under one row; mirror the live CaptionRunGrouping direction check before appending.
Useful? React with 👍 / 👎.
| persistChoices() | ||
| dismiss() | ||
| // Force the picked engine only when there was a choice. | ||
| start(sensitivity, availableBackends.count > 1 ? chosenBackend : nil) |
There was a problem hiding this comment.
When the summary model is not downloaded, start(...) opens the AI-model consent dialog and can return without running anything if the user cancels, but this sheet has already persisted the staged Translate to choice. That leaves translateToRaw changed while the entry directions/translations still describe the old target, and the Languages sheet now treats the new target as current so Apply is disabled; persist these choices only after consent/job admission or roll them back on cancel.
Useful? React with 👍 / 👎.
| Button("Remove Dolphin model", role: .destructive) { | ||
| dolphinStore.removeInstalled() | ||
| dolphinInstalled = DolphinModelStore.isInstalled |
There was a problem hiding this comment.
When a zh/ja/ko recording has prepared Hybrid Auto finals, this action can delete Dolphin's files before the lazy DolphinDecoder creates its sherpa recognizer on the first final; SherpaOnnxOfflineRecognizer fatalErrors if the model/tokens paths are gone, so switching to Settings and removing the model mid-recording can crash. Gate removal on no active recording/heavy job, or restart/cancel Dolphin users before deleting the directory.
Useful? React with 👍 / 👎.
The offline ASR loop used to fully stop at thermal .serious, which froze the progress bar on a merely-warm device even though the SoC was only throttled (grinding the full pool there buys little, but stopping entirely wasted the time). Now it degrades gracefully: - .serious: cap in-flight decodes to ONE (thermalConcurrencyCap) — fewer parallel ONNX sessions cut the heat while decode keeps advancing, so the bar moves instead of freezing - .critical: unchanged HARD hold (shouldHoldForThermals now gates on .critical only) — the level where iOS starts killing apps The "cooling down" UI state now shows only at the true .critical hold, not for the .serious slow-down. Live pipeline thermal policy untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 418a5c3869
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self.composer = HybridVolatileComposer( | ||
| separator: language.usesCJKScript ? "" : " ") |
There was a problem hiding this comment.
In Korean live sessions this separator becomes empty because AppLanguage.usesCJKScript is also true for Korean, and prepare() reinitializes the composer with the same expression below. HybridVolatileComposer uses this separator when joining Apple finalized fragments with the current volatile text, so Korean interim captions can render words stuck together (e.g. an Apple final plus the next volatile chunk) even though Korean uses spaces; use an empty separator only for zh/ja or a Hangul-aware join rule in both initializations.
Useful? React with 👍 / 👎.
| stagedSpoken != (session?.spokenLanguageRaw ?? "") | ||
| } | ||
| private var translateChanged: Bool { | ||
| stagedTranslate != Self.currentTranslateRaw(session) |
There was a problem hiding this comment.
After a mid-recording Translate-to change, a session can have mixed entry direction.target values but no explicit translateToRaw; currentTranslateRaw then shows the first entry's target in the picker. Because this compares the staged value to that inherited display value, choosing that displayed target leaves Apply disabled and never writes an override or re-translates later entries, so the transcript stays mixed when the user is trying to force one target; compare against the explicit stored value or otherwise distinguish inherited display from an applied override.
Useful? React with 👍 / 👎.
| folded.sourceText += UtteranceMerger.joiner( | ||
| between: folded.sourceText, and: entry.sourceText) + entry.sourceText | ||
| } | ||
| let translations = entries.compactMap(\.displayTranslation) |
There was a problem hiding this comment.
In translated live captions, adjacent finalized entries can be folded while one fragment's translation is still pending or has failed; compactMap drops that missing fragment and the folded row then shows a combined source line with only the available translation text and no spinner/warning. This makes the user read a partial translation as if it covered the whole folded utterance, so either wait to fold until every fragment has a translation state or carry per-fragment pending/failure through the folded row.
Useful? React with 👍 / 👎.
… task Starting a re-transcribe/summarize/identify on one session while another was still processing enqueued a second heavy ASR/LLM pass; even when the serial gate held, it strained memory and read as the app breaking apart. The per-session .disabled guards only covered the SAME session. Now every heavy action funnels through startHeavyAction: if another session has a running/queued job it presents "Another session is still processing — Stop it and continue / Not now" instead of silently queueing. Confirming cancels the other job(s) first, then proceeds. Covers Re-summarize, Re-transcribe, Identify speakers, the summary style/length pickers, and Suggest hotwords. (Batch re-transcribe from the Sessions list is a deliberate queue and is left alone.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Identifying speakers…" froze at 0% on import: sherpa runs a whole-file pyannote segmentation pass FIRST and reports no progress until the later embedding sweep, so the determinate bar sits at 0% for that window — minutes on a long file, worse under the thermal throttle that follows a transcription. The diarize call is one opaque C pass with no thermal pause, and nothing told the user it was still working. - render "Identifying speakers…" as an indeterminate spinner while the fraction is 0 (segmentation), switching to a real bar once the embedding sweep reports progress — both the detail row and the list - log the analysis's thermal state at start and wall duration at end, so a "stuck at 0%" report is diagnosable as throttled-slow vs truly hung Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A backgrounded import/re-transcribe used to only resume when the app was next foregrounded. ASR decode and sherpa diarization are CPU (provider "cpu"), NOT Metal — so unlike the LLM they don't hit the background process-abort wall; the only real blocker was the ~30s foreground-grace budget. Now those CPU stages continue in a BGProcessingTask. - Info.plist / project.yml: `processing` background mode + the permitted task identifier (com.kunzhipeng.loqi.background-asr) - BackgroundProcessingScheduler (hosted in SummaryJobCenter.swift to avoid an xcodegen pbxproj change): registers the handler at launch from the AppDelegate, schedules on background-enter when there's resumable CPU work, runs the window, re-arms if it ends early - SummaryJobCenter: `backgroundProcessingActive` relaxes the resume guards for the ASR-first jobs ONLY (import / re-transcribe / pending post-process); LLM cleanup/summary still park (isBackgrounded stays true) so no Metal runs in the background. runBackgroundASRWindow holds the OS window, then suspends (cancel + keep checkpoints) at expiration - requiresExternalPower=false: any idle window, safe off-charger thanks to the decode's thermal degradation + per-segment checkpointing Additive: the flag defaults off, so if iOS never schedules the task the behaviour is exactly today's resume-on-foreground. NOT device-tested — BGTaskScheduler only runs on hardware (dev-triggered via LLDB); the launch register + plist identifier match is the crash-risk to verify first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce6f4c8f15
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pipeline.archive.update(updated) | ||
| if translationAffected { | ||
| pipeline.jobs.retranslate(sessionID: sessionID) |
There was a problem hiding this comment.
When the user applies a spoken/translate override and then the app backgrounds or a recording starts before retranslate finishes, the final code cancels and clears direct retranslate jobs instead of resuming them. This write has already made the override current, while retranslate only calls archive.update(record) after all entries are redrafted, so the transcript remains on the old directions/translations and the Languages sheet sees no changes to apply; persist after the redraft is admitted/completes or roll back on cancellation.
Useful? React with 👍 / 👎.
| guard cleanupAndSummarize else { | ||
| logger.notice("retranscribe \(sessionID, privacy: .public): AI off — ASR-only, no cleanup or summary") | ||
| return |
There was a problem hiding this comment.
When this request is the new post-stop postProcessAndSummarizeNewSession path and AI features are disabled, cleanupAndSummarize is false, so the job updates the transcript/diarization then returns without creating the summary the user just picked and without surfacing aiDisabled. Manual re-transcribe's ASR-only behavior is fine, but the new-recording summarize path should still report or route the summary failure rather than silently consume the action.
Useful? React with 👍 / 👎.
| SpeakerAttribution.attribute( | ||
| utterances: utterances.map { ($0.start, $0.end) }, | ||
| to: segments)) | ||
| logger.info("import: \(Set(segments.map(\.slot)).count) speakers across \(segments.count) segments") |
There was a problem hiding this comment.
When an import requested speaker separation but sherpa returns no segments, such as for a short/noisy file or when VAD finds no speaker regions, this branch leaves speakerSeparationFailed false and proceeds with all-nil slots from SpeakerAttribution.attribute. The saved import therefore has no speaker labels and no retry affordance, unlike the standalone retry path that treats zero labeled entries as an error; mark the separation as failed before returning the record.
Useful? React with 👍 / 👎.
| } else { | ||
| parts.append((text: run.text, speaker: effective, start: run.start)) | ||
| } | ||
| store.applyCleanedSource(cleaned, for: entryID) |
There was a problem hiding this comment.
When a translated live entry accepts source cleanup here, the source text is swapped immediately while the existing draft translation still describes the pre-cleanup text until the async re-translation task completes. If the user stops or the app crashes before that task lands, the archive/journal persists entry.displayTranslation with the cleaned source, leaving saved transcripts and subtitles with mismatched translation; clear/withhold the old translation or commit the cleanup and re-translation atomically.
Useful? React with 👍 / 👎.
The launch handler crashed with a libdispatch queue assertion the moment
the simulated task fired: `register(using: nil)` delivers the handler on
a BACKGROUND queue, so MainActor.assumeIsolated trapped ("block not
expected to execute on queue"). Register with `using: .main` so the
handler runs on the main actor's executor and the assume is valid.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
"No task request has been scheduled" on _simulateLaunch despite a successful submit means the daemon isn't holding it. Log the pending count right after submit so the next device run shows whether the request actually queued (1) or was dropped (0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Two bugs from the device test of the BGProcessing window: 1. DATA LOSS (critical): the import's CancellationError handler decided keep-vs-delete by reading activities[id] at handler time. A late progress tick could reset that from .pausedForBackground back to .importing between the suspend's badge-set and the handler, so a SUSPENDED, still-checkpointed import was DELETED — the record vanished on foreground. Now suspension is tracked in a race-free suspendedImportIDs set (marked by suspend/yield, cleared when a fresh import task starts), and the handler keys on that. A user cancel still deletes; a suspend never does. 2. The window decoded nothing: runBackgroundASRWindow called resumeUnfinishedImports, which needs a NIL badge, but the suspend had left .pausedForBackground — so it skipped every job. Now the window un-pauses the ASR-first jobs first (as resumeBackgroundJobs does), exits its hold-loop on foreground too, and only re-suspends on cleanup if STILL backgrounded (so it can't cancel a foreground import). Still device-untested end to end — but the record can no longer be lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A BGProcessingTask window ran the full decoder pool (up to 3 parallel ONNX sessions × ~2 threads each) — fine on charger, but aggressive off-charger (requiresExternalPower=false): battery drain and a real risk of iOS's background energy monitor killing the task. Now the window caps in-flight decodes to ONE, folded into the existing thermalConcurrencyCap (background OR .serious → 1 decoder). Dynamic: a VADSegmentedTranscriber lock, set on the main actor by runBackgroundASRWindow and read off-main in the dispatch loop, so the pool lifts back to full the moment the app foregrounds and the window ends. Slower-but-steady; the per-segment checkpointing means the reduced parallelism loses nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Findings from the self-review of the BGProcessing window: - Re-arm loop (main): the window re-armed as long as the import was importing==true, but a background import parks at the LLM phase and never completes, so once ASR was fully decoded every window replayed the cache + redid CPU diarization for nothing. Now it re-arms ONLY when the decoded-segment count grew during the window (backgroundASRProgressMark); no new segments = ASR done = don't re-schedule. - Flag cleanup moved into a defer so backgroundProcessingActive and the backgroundWindowActive lock can never leak. - Documented why setTaskCompleted lives in the completion task, not the expiration handler (cleanup is prompt — the decode cancel is fire-and- forget). - Removed the getPendingTaskRequests device-diagnostic log (its job is done — device confirmed pending=1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 participant
Summary
This PR evolved across three phases:
LiquidAI/LFM2.5-230M-MLX-4bitbehind a Settings toggle for the live-refine role.rawSourceText), and Apple's Translation framework is the sole translator (instant draft from raw text, re-translate of the cleaned sentence). Thermal motivation: the LLM no longer fights SenseVoice + the diarizer for the GPU; Apple MT is ANE-cheap. Cleanup runs in transcribe-only sessions too.Model roles after this PR:
Also: LFM2.5's
requiredHeadroom(500 MB) clears the 400 MB memory-shed floor, guarded by a catalog test; rejected/failed cleanups log under therefinecategory.Test plan
rawSourceTextpresent in the archived sessionrefinelogs — this is also the LFM2.5 capability check for the monolingual task)🤖 Generated with Claude Code
Addendum: offline paths get the same polish
Re-transcribe and file import now run a shared
OfflineTranscriptPolisherafter ASR: deterministic hotword fixup (all backends), then — for SenseVoice/Apple only; Qwen3-ASR is already the accuracy pass with decoder hotword priming — a per-sentence LFM2.5 cleanup using the live prompt/fidelity-gate API (raw text kept inrawSourceText), then a second fixup pass to catch LLM drift, all before Apple translation drafts. Imports also unload the LLM unconditionally before decode and count the cleanup phase as LLM work for the memory-warning shed. New localized "Cleaning up transcript…" phase reports per-entry progress in the sessions list and detail view. Cleanup is best-effort: missing LFM2.5 weights or AI-off skip the phase without failing the job.Additional test plan:
OfflineTranscriptPolisherTests(fixup ordering, gating truth table, rawSourceText rules, best-effort/cancellation semantics); full suite 502 tests greenAddendum 2: live speaker diarization dropped
Continuous Sortformer inference beside ASR + the LLM was the biggest remaining live heat source, its labels were overwritten by the offline pass anyway, and the model currently fails to load on iOS 26.5. Live recording now runs ASR + LFM2.5 cleanup only; speaker labels come exclusively from the offline Pyannote pass (auto post-process / re-transcribe / imports), gated by the same speaker picker via
recordingSpeakerCount. Settings and onboarding download the offline diarizer instead of the streaming one. Behavior change: sessions recorded with audio-saving off get no speaker labels. Net −548 lines.Addendum 3: offline diarization on sherpa-onnx, ModelScope-mirrored
VoiceprintServiceswaps the FluidAudio Pyannote CoreML bundle (HF-only, ~80 MB) for sherpa-onnx offline diarization on the already-vendored ONNX runtime: pyannote segmentation-3.0 (~6 MB) + 3D-Speaker bilingual zh/en CAM++ embedding (~28 MB) — an embedding trained on exactly this app's two primary languages.DiarizerModelStorefollows the Qwen3-ASR dual-source pattern (Hugging Face + existing ModelScope community mirrors —pengzhendong/sherpa-onnx-pyannote-segmentation-3-0andfengge2024/3dspeaker_speech_campplus_sv_zh_en_16k-common_advanced.onnx, verified byte-identical), the HF-Mirror-onlyDiarizerSourceenum retires in favor of the sharedASRModelSource, and China-region onboarding maps the diarizer to ModelScope like everything else. Explicit speaker picks force the cluster count; Auto discovers by threshold. The FluidAudio package dependency is removed entirely. Pending: a device A/B of attribution quality vs. the old pipeline.Addendum 4: self-review fixes
An 8-angle review of this PR surfaced 6 correctness and 2 cleanup findings; all fixed in four follow-up commits: legacy
hfMirrordiarizer-source values migrate to ModelScope (pre-PR China onboarding was silently falling back to huggingface.co); orphaned FluidAudio caches are reclaimed at launch; explicit 5/6-speaker import picks are honored as exact cluster counts instead of silently becoming Auto (intent now travels throughdiarizeFileas the raw picker value); upgraders with the old 0.8B live model get a one-shot background download of LFM2.5 instead of a permanent "AI model not downloaded" banner; refinement re-translations are tracked, cancelled at session end, and journaled on delivery; the sherpa diarization call runs on a GCD thread instead of parking a cooperative-pool thread; and the sentence-cleanup pipeline and model-store download machinery each collapse to a single shared implementation.