Repository navigation
qwen27b-speculative-decoding-fix: fix long prompt crash and archive evidence files - #287
Open
Hinanaw-iTenshi wants to merge 33 commits into
Open
Hinanaw-iTenshi wants to merge 33 commits into
Hinanaw-iTenshi wants to merge 33 commits into
Conversation
…physical memory gate for draft_model llama_cpp.cpp: speculative n_ctx now prefers model_config_.get_context_size(), falls back to 32768 only when unset/0 (non-speculative path unchanged, still falls back to 8192). llama_speculative_config.h/.cpp: replace fixed 60GiB total-physical-memory threshold with EstimateSpeculativeMemoryRequirement() (weight file sizes + KV cache estimate using GGUF block_count as n_layer + fixed safety margin, calibrated against real llama-completion.exe PeakPagedMemorySize64 measurement) and GetAvailablePhysicalMemoryBytes() (compares against available, not total, physical memory). model_manager.cpp: GGUFVerify::CreateIfVerifiedImpl() calls the new dynamic estimator instead of the fixed threshold. model_manager.h: corrected now-inaccurate comment on kInsufficientMemory. Co-authored-by: Junie <junie@jetbrains.com>
…branch so fit can auto-size GPU offload within device memory Co-authored-by: Junie <junie@jetbrains.com>
…lazy-output detection Co-authored-by: Junie <junie@jetbrains.com>
…/keyword judging in benchmark_gguf_tuning.py Co-authored-by: Junie <junie@jetbrains.com>
- 新增 run_inception_precision_compare_task()：只在首轮发送用户原文 prompt， 不无条件追加续问；探测到 busy 时按既有 busy-poll 机制等待；探测到疑似 Blocking Condition（_looks_like_blocking_question 启发式）时才追加唯一一轮 最简中性继续指令，否则如实停止——用于验证'一次性自动跑完,无需中途确认' 这句话本身是否成立。 - 新增 _find_inception_artifacts()：三要素判定（FP16/W8A8模型目录存在、 mtime>=任务开始时间的新鲜推理证据、对比陈述关键词），避免复用 inception_v3/NOTES.md 里本来就有的旧数字蒙混过关。 - 新增 --task inception_precision_compare CLI 入口，与现有 model_conversion/model_build_probe/tetris 模式一致。 - 仓库范围核查确认 test_builder_agent_tasks.py 是唯一的真实接入脚本； git log --all --follow 对 test_builder_cli.py/test_qaimodelbuilder_remote.py 零命中，证实历史上不存在被遗漏的真实接入逻辑。 - 详细决策与核查证据记录在 test_builder_agent_tasks.py.notes.md（gitignored）。 Co-authored-by: Junie <junie@jetbrains.com>
…ng inception task Co-authored-by: Junie <junie@jetbrains.com>
Co-authored-by: Junie <junie@jetbrains.com>
…ard elimination Co-authored-by: Junie <junie@jetbrains.com>
…ist them in transcript/results.json Co-authored-by: Junie <junie@jetbrains.com>
…ro-score L0 fallback implementation Co-authored-by: Junie <junie@jetbrains.com>
…r per-chunk sub-inference RunSummarizationInference() (shared by LongTextSummarizer::MapChunk/ReduceSummaries and TaskMemoBuilder) had no independent timeout on context_->Query(). Real-machine repro (tmp/step4_run1) showed the Qwen3.8-27B driver model stuck in a non-converging <think> self-correction loop while trying to hit an exact summary length target, consuming the whole turn budget and leaving the main dialog to end with no output. Add a 300s steady_clock wall-clock budget to the per-token callback; on timeout the callback returns false (reusing the existing should_stop clean-exit path, same as client-disconnect detection), a new timed_out flag is checked before the existing success check (Query() returns true regardless of should_stop vs eog_hit, so success alone cannot distinguish a timed-out generation), and the function returns an empty string instead of the partial (likely mid-<think>) output. Empty result triggers the caller's existing summarization-failed/empty fallback (keep original chunk text), so no caller-side changes are needed. Co-authored-by: Junie <junie@jetbrains.com>
…usage field
ResponseTools::responseDataJson() previously hardcoded usage.{prompt_tokens,
completion_tokens,total_tokens} to 0 for every response, regardless of actual
generation. This caused downstream QAIModelBuilder to treat local-model
responses as an "empty usage block" and fall back to an unreliable tokenizer
estimate (confirmed via two independent real-machine runs of
inception_precision_compare producing an identical used_tokens=95 - a
deterministic-bug fingerprint, not model randomness).
Fix: responseDataJson() gains two optional trailing params
(prompt_tokens/completion_tokens, default 0, preserving existing behavior for
mid-stream content chunks per OpenAI streaming semantics). In SendResponse(),
reuse handle->HandleProfile() (already implemented by all 3 backends -
GGUF/MNN/QNN - for logging) to extract real num_prompt_tokens/
num_generated_tokens once per request, wired into the 3 finish/final response
call sites (connection_broken end frame, normal end frame, non-streaming final
response). Safe fallback to 0 on missing/non-numeric fields.
Co-authored-by: Junie <junie@jetbrains.com>
…long agentic chains Co-authored-by: Junie <junie@jetbrains.com>
…for local-only agentic tasks Co-authored-by: Junie <junie@jetbrains.com>
…o avoid premature history eviction of key facts Run9 real-machine test (Qwen3.8-27B+DFlash2, 16384 ctx) showed messages capped at 16 while token usage was only 29%-65%; this evicted the model's just-read qairt_env.json tool result (correct QAIRT SDK path), leaving only a bare numeric placeholder with no Task Memo fallback. The model then hallucinated a wrong tool path and fell into a PowerShell search loop until per_turn_timeout. Raising the count limit lets the 16384-token context budget become the binding constraint instead of message count. Co-authored-by: Junie <junie@jetbrains.com>
…-aware Step1 trimming - service_config.json: add prompt_optimization.task_memo (enabled=true, model_layer_enabled=false) - rule layer only, zero extra inference, 1.5x memory-frontier gain per docs/compression-reliability-assessment.md; task_memo loading code in model_manager.cpp already existed, unused until now. - message_pre_filter.cpp PreFilterMessages: Step 1 (max_messages_limit) now only trims when message count exceeds the limit AND token usage ratio reaches new token_pressure_trigger_ratio (default 0.75); previously trimmed purely by count regardless of token budget (Run9 repro: trimmed at 41% token usage, discarding a tool result the model needed, causing hallucination). Moved handle/system_tokens/available_tokens/calc_msg_tokens up before Step 1 to support this; removed the now-duplicate later calc_msg_tokens definition. Co-authored-by: Junie <junie@jetbrains.com>
…h on long prompts via per-chunk incremental common_speculative_process (replaces token-limit guard) Root cause: common_speculative_process() reads the target-side hidden-layer capture buffer sized to n_batch; every llama_decode overwrites it from row 0. The prefill loop already decodes the prompt in n_batch-sized chunks (correct), but the previous implementation called common_speculative_process() only once, after all chunks, with the entire prompt -- once the prompt exceeded one n_batch, that call read stale/out-of-bounds data left over from the last chunk's decode. Upstream common_speculative_begin()'s warning comment confirms that calling process() after every prefill chunk is the only correct way to drive this API; there is no "process the whole prompt once" alternative. Fix: call common_speculative_process() immediately after each chunk's llama_decode, using that chunk's own tokens with correctly offset pos values; removed the old one-shot post-prefill call. This supersedes the hard 2048-token guard added earlier in this same commit, which only masked the symptom by rejecting long requests outright instead of fixing the call pattern. Validated on remote ARM64 (full rebuild): GPU up to 8430 prompt tokens, CPU up to 6152 tokens, all HTTP 200 with coherent output and no crash; short-prompt decode speed unaffected (two repeats within noise of each other, no regression). There is no longer any speculative-decoding token limit independent of n_ctx. Deployment note (cost a full extra debugging detour before being found): after rebuilding, all dependent DLLs (llama-common.dll, llama-completion-impl.dll, GenieAPILibrary.dll) must be redeployed alongside the exe. Redeploying only GenieAPIService.exe silently leaves the process running stale logic and produces a false "fix doesn't work" result.
…parameters and repeat sampling to benchmark tool Support spec_draft_n_max, spec_draft_p_min, and repeated question sampling in benchmark_gguf_tuning.py to evaluate speculative decoding tuning configurations. discard config backup file after restoration in benchmark runner Call discard_backup after restore to ensure no bench_orig_backup files remain in model directory. Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
from
October 6, 2026 16:59
a5f8d36 to
eab5430
Compare
…ere before tmp/ cleanup) Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
changed the title
…iscarding context Discarding earlier context (sliding-window eviction) to let speculative generation continue past n_ctx was rejected: no previously-generated content should be dropped, and the limit the service honors should be the model's full configured n_ctx, not a value generation silently works around by forgetting its own output. Reused the existing stopped_by_output_limit_ flag (maps to finish_reason=length, the same mechanism already used when max_length is hit) so the speculative generation loop stops cleanly before decoding a batch that would exceed n_ctx, instead of evicting KV cache entries from either the target or draft context. Checked upstream first: 4453b535f and other recent fork commits touching speculative decoding only fix layer-input batch-order preservation, unrelated to context-limit handling, so there was no existing fix to adopt instead. Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
3 times, most recently
from
October 7, 2026 04:11
15cad6e to
7cbab2b
Compare
…e prompt, and lightweight page directory to Task Memo Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
from
October 7, 2026 05:19
7cbab2b to
390ef54
Compare
…ault tool_call_repair is a purely local tool-call retry/circuit-breaker mechanism with no cloud relevance, so it no longer belongs in service_config.json (which should stay scoped to edge-cloud routing). Removed the JSON section and its parsing block in model_manager.cpp; ToolCallRepairConfig now relies solely on its struct member defaults in model_config.h. Updated architecture.md and stale comments referencing the removed config field. Co-authored-by: Junie <junie@jetbrains.com>
service_config.json 收窄为端云结合最小配置 + 下游客户端私货文案（按客户端要求保留， 不迁出）；通用本地算法参数（消息裁剪/紧急截断/长文本摘要/Task Memo/技能目录格式等）、 本地调试开关（status_update_content_visible/log_inference_stream），以及此前硬编码的 tool_call_repair（恢复为可配置项，因与端云无关故不回归 service_config.json）， 全部迁至新建的独立文件 prompt_engineering.json，由单独加载流程无条件解析，不受 -n 门控。 log_rule_matches 调试开关归位到 routing.sensitivity_detection.debug_log_matches（它是 安全检测模块自身的子开关,此前错误地放在顶层 debug 节）。 同步更新 test_service.py 中依赖这些字段路径的测试基础设施（_read_long_text_trigger_ratio/ _ScArmConfigOverride/_LtmArmConfigOverride 及相关帮助文本），以及 CMakeLists.txt 的三处 构建产物拷贝命令，确保 prompt_engineering.json 随 service_config.json 一起部署。 已在远程 ARM64 机器上验证 GenieAPIService.exe 编译通过（exit_code=0）。 docs/architecture.md 对应章节已改写为准确反映当前归属（本机私有文档，不随此提交同步）。 Co-authored-by: Junie <junie@jetbrains.com>
…dating Qwen3.8-27B GGUF/GPU DFlash2 speculative decoding on QAIModelBuilder agent tasks
- model_input_builder.h/long_text_summarizer.cpp: Qwen3.5-generation models
(confirmed via official model card) do not honor the /think //no_think soft
switch; replace it with the official hard switch (prefilling an empty
<think></think> block) gated on is_thinking_model() &&
!getenableThinking() && is_speculative_draft_configured(), applied to both
the main prompt path and the LongTextSummarizer Map/Reduce sub-inference
prompts.
- llama_cpp.cpp: tighten the --reasoning-budget CLI arg to
is_thinking_model() only; pair the forced reasoning-budget closing branch's
llama_decode() call with common_speculative_process() to keep the
drafter's hidden-state capture buffer position watermark in sync, per the
invariant already required by the per-chunk prefill fix.
- chat_request_handler/tool_call_repetition_store.{h,cpp}(+selftest),
response_dispatcher.cpp, task_memo_builder.{h,cpp}: new session-scoped
LRU+TTL store that detects exact-signature repeated tool calls and injects
a one-shot corrective note via Task Memo (or a standalone fallback) once a
repeat threshold is hit.
- model_config.h/model_instance_config.h/model_manager.cpp/
prompt_engineering.json: add speculative_draft_configured_ flag (derived
from config.json's draft_model block) and a general, model-agnostic
Windows shell usage hint (cmd.exe vs PowerShell syntax) injected when a
request declares exec/shell-like tools on Windows builds.
- test/test_builder_agent_tasks.py: widen the SSE client read timeout
(10,25)->(15,60) to clear the server's 15s heartbeat interval under heavy
real workloads; make the four budget-related argparse defaults None so
they stop silently overriding each task function's own tuned defaults; add
a stream_timeout-triggered neutral continuation so a client-side SSE
deadline no longer prematurely ends a still-healthy turn.
4 complete real-machine reruns of inception_precision_compare with
qwen3.8-27b-q4_0 (step3_rerun/step3_final_rerun/step3_r3_final/
step3_final_v2) confirm zero crashes, zero protocol/circuit-breaker
anomalies, and zero context-overflow truncations across all rounds. The
three-factor acceptance criteria were not met in any round; the final
observed bottleneck is the model spending an entire turn re-reading skill/
guardrail documentation before reaching the real conversion commands, which
is a model-behavior/test-policy question (not a code defect), recorded as
a new open issue.
Co-authored-by: Junie <junie@jetbrains.com>
真机取证推翻了"模型只读文档不执行"这一归因：模型始终在发执行类调用，是服务端在它 看不见的地方逐级削弱了它的行动能力。四处根因与对应修复： - prompt_optimizer.cpp: BuildRelevanceKeywords() 把 messages 首条 user 消息作为任务 锚点无条件纳入关键词语料（anchor_first_user_message），多轮工具循环里滚动窗口只剩 assistant/tool 时关键词不再变空；AssignSkillDetailLevels() 补齐"无法判断相关性"这 第三态，keywords 为空时不再把技能目录整体拍扁到最低档；FilterToolsByRelevance() 新 增核心能力工具豁免（protect_core_capability_tools），文件读写/路径检索/命令执行类工 具不再因相关性零分被整条删除（仍无条件受 token 预算约束）。 - message_pre_filter.cpp: Phase 4 紧急截断改为在受保护区间内按 token 降序多目标选取 （target_largest_tool_messages），并把候选集从"仅 role=tool"扩展到带 tool_calls 的 assistant 消息正文——后者走结构感知截断，只收缩 arguments 里体量最大的那个字符串值， name 与 arguments 的键结构逐字节不动，畸形信封直接跳过而非输出半截 JSON；同时修正 Available 的 size_t 下溢与截断日志的引用别名缺陷。 - prompt_engineering.json: 新增 relevance_filter 节与 target_largest_tool_messages 开 关（均默认开启、可逐字节回退）；重写 windows_shell_hint.hint_text，改为给可直接照抄 的正反范式（每条正例都已在真实 PowerShell 上验证过）；摘要缓存 ttl_minutes 60->180 以覆盖单轮长任务全程。 - service_config.json: max_tool_call_retries 40->80。按工具调用次数硬截断只是失控的粗 粒度近似，真正的失控判据是已有的四套无进展信号；实测合法长链路任务需 55-70 次。 - test_service.py: 两处把 prompt_engineering.json 当成 <exe_dir> 根目录文件的路径 bug 修正为 <exe_dir>/config/，此前导致相关套件静默跑在默认配置上。 真机连续三轮复测依次解除三道成因各不相同的中断（紧急截断只看最后一条消息 -> 候选集漏 掉 assistant 正文 -> 次数上限），三轮均 crashed=0、零续问、零阻断命中，最终真实驱动完 整模型转换流水线并落盘首个上下文二进制产物。 Co-authored-by: Junie <junie@jetbrains.com>
…00 拦截与 tool_call_limit 回退分支），防失控交给按进展判定的机制 Co-authored-by: Junie <junie@jetbrains.com>
- context/llama_cpp.cpp|.h, context/context_base.h: 跨请求保留上次已 decode 的 token 序列，新请求按最长公共前缀对 target 与 DFlash draft 两侧 memory 调 seq_rm 裁剪，只 prefill 新增尾部；模型切换、Stop() 中断、上下文超限、前缀长度为 0 或任何异常时回退到 完全清空路径。开关为 prompt_engineering.json 的 llama_prefix_cache.enabled（默认开）。 同时补齐投机解码 proposed/accepted 计数与接受率，并按 query 导出追踪指标。 - chat_request_handler/request_trace_writer.h 及 chat_request_handler.cpp、prompt_ledger.h、 prompt_stats_helper.h: 每个请求写一行 JSONL（请求 id、模型、最终 prompt 全文、原始输出、 解析出的 tool_calls、prompt/生成 token 数、prefix_reused_tokens、prefix_reset_reason、 messages_dropped_prefix_len、TTFT、prefill/decode 速率、投机接受率、PromptLedger 摘要）， 由 prompt_engineering.json 的 debug.request_trace 开关与输出路径控制，默认关闭。 - chat_request_handler/message_pre_filter.cpp|.h: 驱逐状态按原始消息下标与原始内容核验 （不再对压缩后内容做哈希，否则每轮必然失配）；驱逐边界只能单调前进；Phase 4 紧急截断长度 量化到固定档位且跨轮只能变小；calc_msg_tokens 与批量丢弃都计入 tool_calls 结构占用；存活 非 system 消息条数设下限保护；重复工具调用结果以索引中的已有摘要做短路桩替换。 - chat_request_handler/task_memo_builder.cpp|.h, model_input_builder.h: Task Memo 记录已读 文件路径与截断摘要并渲染为独立段落，对存活消息中的重复读取追加提示。 - chat_history/*, model/*, response/*: 配合上述改动的接口透传与配置读取。 - tool/analyze_request_trace.py: 离线分析追踪 JSONL，统计重复工具调用、重复读取、失败重试、 每轮耗时与 token 占比，并按 prefix_reset_reason 分组给出 prefill 均值。 Co-authored-by: Junie <junie@jetbrains.com>
…ion_tokens=1、生成速率 0 的错误 Co-authored-by: Junie <junie@jetbrains.com>
….json to match other GGUF models Co-authored-by: Junie <junie@jetbrains.com>
Co-authored-by: Junie <junie@jetbrains.com>
Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
from
October 12, 2026 06:23
a94d70a to
47bd437
Compare
- message_pre_filter: 重复调用按首份存活与 40% 预算事前决策 stub/refill，决策跨轮固定；被重读结果驱逐时保留（常规 <=20%，含 must_keep 合计 <=40%），must_keep 首份不被 Drop/1A/1C/2A/2B/存活下限兜底压缩；SmartSelect 预判触发时全部 refill；新增 [RepeatRead] 日志 - message_pre_filter: prefix_hash 对称修复（msg_raw_for_hash 在占位改写前拷贝） - task_memo_builder: Render 超预算块 continue，read_files 前移，redundant_call_guard 文案如实反映是否仍在上下文 Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
2 times, most recently
from
October 12, 2026 23:53
c81499e to
8c5796c
Compare
…servable response_dispatcher: pass request_data into the Layer2 retry ModelInputBuilder so the retry path keeps skill catalog, relevance keywords, tool filtering and task memo. message_pre_filter: fold OpenAI tool_calls back into dropped-message content so TaskMemoBuilder can recognise read tool calls. llama_cpp: report prefix cache miss mode/n_cached/limit/common_prefix. task_memo_builder: report memo blocks skipped for budget. model_input_builder: report system prompt rebuild delta instead of mislabelled savings. test: FP16-only builder task variant, genie_service.log curve parser, per-turn timeout now follows the total deadline, existing_run run_id is recorded and the blocking-question heuristic no longer fires on a bare question mark. Co-authored-by: Junie <junie@jetbrains.com>
Hinanaw-iTenshi
force-pushed
the
main-fix
branch
from
October 13, 2026 03:49
8c5796c to
4e57002
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This PR provides a root-cause fix for long-prompt crashes in Qwen3.8-27B + DFlash2 speculative decoding, replacing the previous temporary 2048-token guard. The crash occurred because prompt hidden states were previously passed to common_speculative_process() once after full prefill completion, whereas the target hidden feature buffer is sized to n_batch and overwritten on each chunk decode. By processing common_speculative_process() incrementally per chunk after each chunk decode, prompts of arbitrary length are safely supported without memory corruption. The PR also updates reference architecture documentation with verified GPU offload measurements and relocates referenced verification artifacts from tmp/ into an untracked assert/ directory with .gitignore coverage.
Changes