feat(infra): better tokenizer

[EYHN]

No description provided.

[graphite-app]

Your org has enabled the Graphite merge queue for merging into canary

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[EYHN]

• feat(infra): better tokenizer #8549 👈
• feat(infra): collect more data to indexer #8528
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @EYHN and the rest of your teammates on Graphite

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 72dc351. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 6 targets

• nx test:coverage @affine/monorepo
• nx build @affine/native -- --target x86_64-pc-windows-msvc --use-napi-cross
• nx build @affine/native -- --target x86_64-apple-darwin --use-napi-cross
• nx build @affine/server-native -- --target x86_64-unknown-linux-gnu --use-napi-cross
• nx build @affine/native -- --target x86_64-unknown-linux-gnu --use-napi-cross
• nx build @affine/native -- --target aarch64-apple-darwin --use-napi-cross

---

Sent with 💌 from NxCloud.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.34%. Comparing base (9ecbe30) to head (72dc351).

Additional details and impacted files

@@                           Coverage Diff                           @@
##           eyhn/add-performance-logger-indexer    #8549      +/-   ##
=======================================================================
+ Coverage                                70.31%   70.34%   +0.03%     
=======================================================================
  Files                                      534      535       +1     
  Lines                                    33585    33520      -65     
  Branches                                  3011     3013       +2     
=======================================================================
- Hits                                     23615    23581      -34     
+ Misses                                    9619     9584      -35     
- Partials                                   351      355       +4     

Flag	Coverage Δ
server-test	77.32% <ø> (+0.07%)	⬆️
unittest	45.35% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.