Remove tiktoken from the codebase

[aseembits93]

PR Type

Bug fix, Enhancement

---

Description

• Added encode_str stub for token calculation

• Removed tiktoken import and usage

• Updated token counts to use encode_str

• Ensures code runs without tiktoken dependency

---

Changes walkthrough 📝

	Relevant files
Enhancement	code_utils.py Add encode_str stub function                                                          codeflash/code_utils/code_utils.py Introduced encode_str stub returning half-length string Provided placeholder for future encoding logic +2/-0
Dependencies	code_context_extractor.py Replace tiktoken calls with encode_str                                      codeflash/context/code_context_extractor.py Removed tiktoken import and tokenizer setup Replaced tokenizer.encode calls with encode_str Added import for encode_str +6/-8

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit c5dbcbf)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Stub Implementation encode_str currently returns a substring instead of actual token encoding or count, which may lead to incorrect token calculations downstream. def encode_str(s: str) -> str: return s[:len(s)//2] Token Count Accuracy Using len(encode_str(...)) may undercount tokens due to the stub logic; verify that token limits are correctly enforced. final_read_writable_tokens = len(encode_str(final_read_writable_code)) if final_read_writable_tokens > optim_token_limit: raise ValueError("Read-writable code has exceeded token limit, cannot proceed") # Setup preexisting objects for code replacer preexisting_objects = set(

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Use ceiling for token estimate Use math.ceil instead of a raw multiplication to prevent underestimating the token count and ensure the limit check is safe. codeflash/context/code_context_extractor.py [75] -final_read_writable_tokens = len(final_read_writable_code)*0.75 +import math +final_read_writable_tokens = math.ceil(len(final_read_writable_code) * 0.75) Suggestion importance[1-10]: 7 __ Why: Using math.ceil prevents underestimating tokens and ensures the limit check errs on the safe side, which is beneficial for avoiding subtle bugs.	Medium
	Cast token estimate to integer Cast the result to an integer to avoid comparing a float against an integer limit and ensure consistent behavior. codeflash/context/code_context_extractor.py [75] -final_read_writable_tokens = len(final_read_writable_code)*0.75 +final_read_writable_tokens = int(len(final_read_writable_code) * 0.75) Suggestion importance[1-10]: 5 __ Why: Casting the token count to int avoids comparing a float to an integer limit and ensures consistent behavior with minimal impact.	Low

[github-actions]

Persistent review updated to latest commit c5dbcbf

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.

[misrasaurabh1]

the logic to encode_str into a half length str and then do the len operation is not a good way of doing it.

Write a function encoded_tokens_length and directly call it which results len(string)//2.
your code is very easy to misuse and does not clearly communicate what it is doing