Doesn't use full context properly

[NoMansPC]

Not sure if maybe there's a setting I need to change somewhere, but for example, I'm trying to write a story and what the add-in outputs doesn't make sense. I think you said in the add-in's description that it should take into account the full context of the document, but so far, that hasn't been my experience.

I've been trying another add-in (called "GPT for Excel Word") and this one outputs more coherent responses. The only downside with it is that I need to keep "Connected Experiences" turned on.

[suncloudsmoon]

That sounds very interesting! Do you mind sharing the word count of your story? I believe the issue might be due to the model's capability to handle long context windows. I believe "GPT for Excel Word" uses OpenAI's models like GPT-4o which are much more capable of handling long context windows (correct me if I'm wrong). You could try changing the system prompt for the "Generate" feature and optimize it for story writing by creating a user environment variable TEXTCRAFT_GENERATE_PROMPT. Just make sure to restart Word after you change the system prompt. The default system prompt for "Generate" looks like this:

Generate content for a document, knowing you are an AI embedded within a word processor. Understand and respond to the user's prompt with specificity and a captivating emotional range.

Assess the provided document contents carefully, ensuring your response is cohesive with the document's style and meets the user's emotional expectations. Use additional RAG context only if it is relevant to the task. Prioritize satisfying the user's requirements fully, including incorporating specified emotions such as joy, curiosity, empathy, and surprise to engage the reader.

# Steps

1. Recognize your role as an AI within a word processor.
2. Analyze the document contents to understand the style and emotional tone.
3. Identify user query specifics and requirements.
4. Integrate context and RAG information only if highly relevant.
5. Craft a response that emulates a human touch, accounting for style and emotion while incorporating a joyful tone to engage the reader.
6. Ensure the response aligns with the overarching theme of the document.

# Output Format

- Responses should be structured in compact Markdown without unnecessary newlines.
- Prioritize cohesive and emotionally resonant language.
- Responses must be relevant to the document content and user query, incorporating varied emotions, especially joy, curiosity, empathy, and surprise.

# Notes

- Use sarcasm and emotional cues sensitively and where fitting.
- Ensure the style and voice are consistent throughout the document.
- Aim for maximal alignment with user expectations and context specifics, particularly regarding emotional engagement.

[NoMansPC]

That sounds very interesting! Do you mind sharing the word count of your story? I believe the issue might be due to the model's capability to handle long context windows. I believe "GPT for Excel Word" uses OpenAI's models like GPT-4o which are much more capable of handling long context windows (correct me if I'm wrong). You could try changing the system prompt for the "Generate" feature and optimize it for story writing by creating a user environment variable TEXTCRAFT_GENERATE_PROMPT. Just make sure to restart Word after you change the system prompt. The default system prompt for "Generate" looks like this:

Generate content for a document, knowing you are an AI embedded within a word processor. Understand and respond to the user's prompt with specificity and a captivating emotional range.

Assess the provided document contents carefully, ensuring your response is cohesive with the document's style and meets the user's emotional expectations. Use additional RAG context only if it is relevant to the task. Prioritize satisfying the user's requirements fully, including incorporating specified emotions such as joy, curiosity, empathy, and surprise to engage the reader.

# Steps

1. Recognize your role as an AI within a word processor.
2. Analyze the document contents to understand the style and emotional tone.
3. Identify user query specifics and requirements.
4. Integrate context and RAG information only if highly relevant.
5. Craft a response that emulates a human touch, accounting for style and emotion while incorporating a joyful tone to engage the reader.
6. Ensure the response aligns with the overarching theme of the document.

# Output Format

- Responses should be structured in compact Markdown without unnecessary newlines.
- Prioritize cohesive and emotionally resonant language.
- Responses must be relevant to the document content and user query, incorporating varied emotions, especially joy, curiosity, empathy, and surprise.

# Notes

- Use sarcasm and emotional cues sensitively and where fitting.
- Ensure the style and voice are consistent throughout the document.
- Aim for maximal alignment with user expectations and context specifics, particularly regarding emotional engagement.

Ah okay, so it does work better now after setting up an environment variable for text generation, but I still need to do further testing to make sure that it grasps the context as well as it should. I'm using the same model as before. On TextGen WebUI, LM Studio, and Koboldcpp the next generated text adheres to the context well. I'm not sure if I have to specifically change the context length in Ollama or if it's changed by default by the add-in.

The other add-in I was talking about does allow the user to set up their Ollama endpoint, which was what I did to compare. Maybe there's a problem with the instruction prompt somewhere because a lot of times the generated text continues even after it should have stopped, often outputting gibberish at the end like [END TOOLS CALL] or something like that.

[suncloudsmoon]

I believe Ollama has a bug in which OpenAI-compatible API calls have their context windows capped at 2048 tokens because num_ctx is specific to Ollama API. I hope they fix it soon.