Repository navigation
Highlighting mis-tokenization when a hunk starts inside an Elixir heredoc #664
Description
What happened?
If the hunk includes the end of an Elixir function's heredoc, with the opening @doc """ outside of the diff's range / behind a code fold, this happens...
Steps to reproduce
Given this block of code:
defmodule Repro do
@doc """
Line one.
Line two.
Line three.
Line four.
Line five.
"""
def hello do
:world
end
endSay you edit 'Line five.`
Viewing a hunk diff will cut off the initial @doc """ opener.
An easy way to reproduce this is to run the following into hunk patch:
hunk patch <<'EOF'
diff --git a/repro.ex b/repro.ex
--- a/repro.ex
+++ b/repro.ex
@@ -4,9 +4,9 @@
Line two.
Line three.
Line four.
- Line five.
+ Line five, edited.
"""
def hello do
:world
end
end
EOFWhich results in the comment block being tokenized as code, and the function code itself ending up all borked:
Expected behavior
I would expect that tokenization should be consistent, whether the opening @doc """ is initially visible or not:
Version
0.18.0-beta.0
Activity
benvinegar commentedon Aug 2, 2026
I just want to comment this is a 10/10 issue. Thank you!
jesseleite commentedon Aug 2, 2026
Ha! I spent years on open source for my day job, I get it ❤️
You're doing a killer job on Hunk, thank you! 🔥
benvinegar commentedon Aug 6, 2026
Fixed for source-backed reviews by #669. Hunk now highlights partial VCS diffs against validated old/new source, preserving grammar state when an opener such as
@doc """is hidden above the hunk; the reported Elixir case has unit and PTY regression coverage. Truly source-less patch input remains best-effort when the omitted lexical context is unavailable.This comment was generated by Pi using OpenAI GPT-5.6