[Feature Request/Contribution] Add Ollama Support for Local LLM Inference

[lawjarp-rai]

Hi TWIX team,

I’d love to contribute a ollama integration for local LLM inference. This would let users run models locally, keeping their data private and avoiding external APIs.

I’ve started a branch with an Ollama-compatible interface, the code is ready as well (note, the normal functions are not working as expected, once you fix them I can test it out fully), I wanted to check:

• Is anyone else working on this?
• Would you be open to a PR adding Ollama as a local LLM backend?
• Are there any contribution guidelines or design preferences I should follow?

I think this could be a great privacy-focused option for users. Let me know if I should move ahead with a PR.

Thanks!

[yiminl18]

Thanks for your feedback — supporting Ollama would be a great addition to TWIX! We’ve updated the contributing guidelines in the README.

There are two possible reasons why open-source LLMs in Ollama might not work directly with TWIX:

1. Besides text-based only LLMs, TWIX also uses a vision-based LLM to learn certain visual information (e.g., headers) from the first two pages of the document. The model in Ollama might not support this vision input. However, this feature is optional; once we adapt text-only LLMs, we can remove this dependency and optimize accordingly.
2. It also depends on how the open-source LLMs handle instructions. We may need to make minor adjustments to the response processing to ensure compatibility.