fix(preview): decode UTF-8 runes in IsBufferPrintable text detection#1507
Open
KristianHolme wants to merge 3 commits into
Open
fix(preview): decode UTF-8 runes in IsBufferPrintable text detection#1507KristianHolme wants to merge 3 commits into
KristianHolme wants to merge 3 commits into
Conversation
IsTextFile() reads the first 1024 bytes and calls IsBufferPrintable(). The previous byte-wise check treated UTF-8 continuation bytes (0x80-0xBF) as C1 control characters, causing Slurm *_err logs with Julia stderr output (e.g. precompile checkmarks) to be rejected as non-text. Decode runes with utf8.DecodeRune and test printability per rune. Allow an incomplete UTF-8 sequence at the end of the buffer (fixed-size reads). Add unit tests for UTF-8 samples and a Slurm err log fixture. Co-authored-by: Kristian Holme <KristianHolme@users.noreply.github.com>
Co-authored-by: Kristian Holme <KristianHolme@users.noreply.github.com>
|
🎉 Thank you for your first contribution to superfile! We’re really excited to have you here 🙌 A maintainer might ask you to make a few changes before we can merge this PR. 👉 Please also take a moment to review our Contribution Guide If you have any questions, feel free to open a Discussion or just ask in the comments! |
github-actions
Bot
added
awaiting pr review
test related
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThe buffer printability check now scans UTF-8 runes instead of raw bytes, and the tests expand coverage for Unicode, invalid UTF-8, truncated sequences, and text-file detection. ChangesPrintability logic and tests
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/internal/common/string_function.go`:
- Around line 205-208: The UTF-8 detection branch in the function that calls
utf8.DecodeRune is treating any utf8.RuneError as invalid, which incorrectly
rejects valid U+FFFD text. Update this check to inspect the returned size from
utf8.DecodeRune and only return false when the result is utf8.RuneError with n
== 1, so the replacement character itself is accepted while malformed UTF-8 is
still rejected.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b2112447-4eaf-4f84-8907-6e4fe752735f
📒 Files selected for processing (2)
src/internal/common/string_function.gosrc/internal/common/string_function_test.go
utf8.RuneError equals U+FFFD, so decoding a valid replacement character returns r == utf8.RuneError with n == 3. Only reject malformed UTF-8 when n == 1. Co-authored-by: Kristian Holme <KristianHolme@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Had some issues with some text files with special characters not being able to preview. This fixes it. Hope it can be useful :)
🤖 Written with Cursor.
Description
Fix text preview failing on UTF-8 text files without a recognized extension.
IsTextFile()reads the first 1024 bytes and callsIsBufferPrintable(). The previous implementation checked each byte withunicode.IsPrint(rune(b)), which misclassifies UTF-8 continuation bytes (0x80–0xBF) as C1 control characters. Files containing multi-byte Unicode early in the buffer (e.g.✓, box-drawing characters) were incorrectly rejected as non-text and showed "unsupported format" in the preview panel.IsBufferPrintable()now decodes UTF-8 runes withutf8.DecodeRuneand checks printability per rune. Incomplete UTF-8 at the end of the buffer is allowed viautf8.FullRune(relevant for the fixed 1024-byte read inIsTextFile()). Binary content with NUL bytes is still rejected.Related Issues
✅ Pre-Submission Checklist
go fmt ./...to format the codegolangci-lint runand fixed any reported issuesSummary by CodeRabbit