perf(codegen): faster splitting comments into lines

[overlookmotel]

Follow-on after #13169.

Implement the first optimization mentioned in #13169 (comment). Iterate over string byte-by-byte rather than char-by-char.

It's amazing how bad Rust is at string operations. I tried it without unsafe code at first, but Rust inserts checks for whether a slice falls on a UTF-8 char boundary on every single operation, even though it's obvious from the context that these checks can never fail. It made the assembly x4 longer, which is no good as this is meant to be a tight loop.

[overlookmotel]

• refactor(codegen): reduce repeated code #13191
• perf(codegen): faster splitting comments into lines #13190 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #13190 will not alter performance

_{Comparing 08-18-perf_codegen_faster_splitting_comments_into_lines (e3bfff1) with main (ada4e84)¹}

Summary

✅ 34 untouched benchmarks

Footnotes

1. No successful run was found on main (e3bfff1) during the generation of this report, so ada4e84 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[Copilot]

Pull Request Overview

This PR optimizes the performance of splitting comments into lines by iterating over UTF-8 bytes instead of Unicode characters. The changes implement a byte-based approach to identify line terminators while properly handling CRLF sequences and Unicode line separators (LS and PS).

• Replaced character-by-character iteration with byte-by-byte processing for better performance
• Added support for Unicode line separators (LS and PS) in addition to CR/LF
• Removed the position field from the iterator struct in favor of modifying the text slice directly

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[graphite-app]

Merge activity

• Aug 19, 12:32 AM UTC: Boshen added this pull request to the Graphite merge queue.
• Aug 19, 12:38 AM UTC: Merged by the Graphite merge queue.

[hyrious]

@overlookmotel does https://doc.rust-lang.org/std/primitive.str.html#method.lines help?

[overlookmotel]

Unfortunately not. From the docs:

Note that any carriage return (\r) not immediately followed by a line feed (\n) does not split a line. These carriage returns are thereby included in the produced lines.

We need to split on \r, \n, \r\n, and also irregular Unicode line breaks <LS> and <PS>.