Streaming diff - error on max tokens reached (+recursive framework)

[RomneyDa]

Description

Throw an error in edit/apply diff streaming when max token limit reached. Prevents truncation of files

I've placed this inside a recursive streaming function that would be great to implement soon but is commented out for now:

• Creates a recursive "Continue where you left off" generator that keeps track of tokens and recurses before hitting max tokens

• not infinite but makes the limiting factor the context length not the maxTokens, which is a 10x for many models

• with this implementation loses prediction abilities after first recursion
• in draft state because seems to be having whitespace issues that cause extra diff lines at recursion boundaries (see pictures with three recursions and a problematic diff at between the 2nd and 3rd group of ~400 tokens

[netlify]

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	2a91c4e
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/68154eb0e3e7940008dfeda9
😎 Deploy Preview	https://deploy-preview-5481--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[RomneyDa]

I have read the CLA Document and I hereby sign the CLA

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[sestinj]

recheck