Agent: Contextual Inferencer

[AtlantisPleb]

I want to build an agent that can replace my day-to-day usage of ChatGPT.

Most of what I do is paste code and documentation into ChatGPT along with a request.

To build this only with OpenAgents plugins, we need plugins for:

1. Given a text input, return a list of URLs
2. Given a URL, return its main content sanitized & converted to Markdown
3. Given an API key, host URL, model name, and system/user prompts, return a text inference from an OpenAI-compatible API

First we'll build those.

[AtlantisPleb]

Plugin #1 completed here: https://github.com/OpenAgentsInc/plugin-url-extractor/releases/tag/v0.0.1

And testable here: https://openagents.com/plugin/10

[AtlantisPleb]

Because WASM plugins don't natively enable HTTP fetching, we need to use Extism host functions.

Extism host functions are not yet implemented in the PHP SDK so we need to use a different SDK. [No longer true; was added 4 days ago]

Instead we wrote plugin #2 in Go (here, following this tutorial) and will deploy it separately to a new Go environment, which we may also put behind an L402 endpoint -- then consume that via our existing L402 integration.

[AtlantisPleb]

Instead we wrote plugin #2 in Go (here, following this tutorial) and will deploy it separately to a new Go environment, which we may also put behind an L402 endpoint -- then consume that via our existing L402 integration.

We have this working now via https://github.com/OpenAgentsInc/L402-plugin-endpoint deployed to https://l402-plugin-endpoint.fly.dev/fetch-url-content; L402 StepTest passes in main repo.

[AtlantisPleb]

Plugin #3 built in Rust works: https://github.com/OpenAgentsInc/plugin-llm-inferencer/releases/tag/v0.0.1

Tested: #106

[AtlantisPleb]

As of #142 this now works reliably. (Locally; requires OPENAI_API_KEY in .env)

Because Extism plugins seem to not currently support streaming, we had to reimplement #3 above with our own inference code to get streaming to work. #1 and #2 work as plugins.

Areas for improvement:

• Change the scraper plugin to be able to extract relevant content from a full HTML page; right now this only works reliably on raw GitHub files with no extra formatting
• Support scraping more than 1 URL per message
• Add inference providers other than OpenAI
• Unable to edit nodes; they are specified in ConnieSeeder
• To fully replace ChatGPT for my day-to-day workflow, it needs multimodal (image input); will work on that in a separate issue

[AtlantisPleb]

Addendum: image input added in #143