Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Add support for structured extraction with Ollama models #68

Open
svilupp opened this issue Feb 5, 2024 · 4 comments
Open

[FR] Add support for structured extraction with Ollama models #68

svilupp opened this issue Feb 5, 2024 · 4 comments
Assignees

Comments

@svilupp
Copy link
Owner

svilupp commented Feb 5, 2024

It would be good to have aiextract enabled for Ollama models.

@svilupp
Copy link
Owner Author

svilupp commented Apr 10, 2024

There is no official support yet, but you can easily build it yourself with the following guide: https://svilupp.github.io/PromptingTools.jl/dev/how_it_works#Walkthrough-Example-for-aiextract

It works well with mixtral and similar models.

@cpfiffer
Copy link
Collaborator

I'm willing to handle this one, relatively straightforward to do. As a clarifying question, what's the difference between a regular schema and a managed one?

@cpfiffer cpfiffer self-assigned this Apr 16, 2024
@svilupp
Copy link
Owner Author

svilupp commented Apr 16, 2024

Great!

There are different API endpoints in Ollama:

  • generate - basically text completion-like; where you provide two fields: system, prompt; no multi-turn conversation (1 reply only)
  • api/chat - "message"-based with multiturn conversations. Similar to OpenAI-style
  • v1/chat/completions - fully OpenAI-compatible endpoint
  1. generate is the OllamaManagedSchema, because they manage everything. It's legacy and I didn't want to break things to much so I kept it
  2. api/chat is the OllamaSchema
  3. not implemented because there was no need / no advantage (at the time)

From that perspective, I'd assume you would add aiextract for OllamaSchema which is build around api/chat. You would just merge the api_kwargs to also include the format="json" and then try to convert to the return type in a try-catch block. Similar to the OpenAI implementation.

I'd suggest to avoid nested return_types (they are harder for OSS models). Good model to use for aiextract is mixtral if you can run it locally.

Does that answer your question?

@svilupp
Copy link
Owner Author

svilupp commented Apr 28, 2024

Just flagging that it might be easier to tackle this after we can provide "JSON type" representation to the open-source models (not JSON schema) - reference.

See #143

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants