✨ feat: Add Fireworks AI Model Provider #3392

hezhijie0327 · 2024-08-03T23:35:11Z

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

基于 📑 [DEMO] This is a demo PR of how to add an OpenAI Compatible Provider #2804 实现支持 Fireworks AI
添加 firefunction-v2 及 firellava-13b 模型（FireworksAI model）
firefunction-v2 模型 function call 调用 ~~[TODO]~~ ~~禁用 Stream as Work Around~~ 已支持流式调用

📝 补充信息 | Additional Information

设置页面



对话页面


工具调用

vercel · 2024-08-03T23:35:16Z

@hezhijie0327 is attempting to deploy a commit to the LobeHub Pro Team on Vercel.

A member of the Team first needs to authorize it.

lobehubbot · 2024-08-03T23:35:26Z

👍 @hezhijie0327

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

codecov · 2024-08-03T23:38:06Z

Codecov Report

Attention: Patch coverage is 95.56650% with 9 lines in your changes missing coverage. Please review.

Project coverage is 91.85%. Comparing base (dd7c8df) to head (34506d8).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/app/api/chat/agentRuntime.ts	14.28%	6 Missing ⚠️
src/libs/agent-runtime/AgentRuntime.ts	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3392      +/-   ##
==========================================
+ Coverage   91.83%   91.85%   +0.02%     
==========================================
  Files         455      457       +2     
  Lines       30452    30654     +202     
  Branches     2116     2960     +844     
==========================================
+ Hits        27965    28158     +193     
- Misses       2487     2496       +9

Flag	Coverage Δ
app	`91.85% <95.56%> (+0.02%)`	⬆️
server	`97.36% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hezhijie0327 · 2024-08-04T02:13:42Z

Stream 关闭后能调用 Plug-in，但是调用后停止输出了

{"id":"414bef39-ab67-4058-983b-c8d473f4a200","object":"chat.completion","created":1722735109,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"message":{"role":"assistant","content":"I don't have real-time information. I need to fetch the current weather information for Shanghai.","tool_calls":[{"index":0,"id":"call_mwuk5cLkFm8vChkBrwm8vRn4","type":"function","function":{"name":"realtime-weather____fetchCurrentWeather","arguments":"{\"city\": \"Shanghai\"}"}}]},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":361,"total_tokens":407,"completion_tokens":46}}

Stream 启用后输出为空

[stream start] 2024-8-4 2:10:49.86
[chunk 0] 2024-8-4 2:10:49.206
{"id":"98d1cdb4-efb0-4a62-b4ec-6abaf128c596","object":"chat.completion.chunk","created":1722737448,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}
[chunk 1] 2024-8-4 2:10:49.246
{"id":"98d1cdb4-efb0-4a62-b4ec-6abaf128c596","object":"chat.completion.chunk","created":1722737448,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_jsvhZfOOhfQFNr6wstCPAZ7j","type":"function","function":{"name":"realtime-weather____fetchCurrentWeather"}}]},"finish_reason":null}],"usage":null}
[chunk 2] 2024-8-4 2:10:49.373
{"id":"98d1cdb4-efb0-4a62-b4ec-6abaf128c596","object":"chat.completion.chunk","created":1722737448,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":null,"type":"function","function":{"arguments":" {\"city\": \"上海"}}]},"finish_reason":null}],"usage":null}
[chunk 3] 2024-8-4 2:10:49.374
{"id":"98d1cdb4-efb0-4a62-b4ec-6abaf128c596","object":"chat.completion.chunk","created":1722737448,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":null,"type":"function","function":{"arguments":"\"}"}}]},"finish_reason":null}],"usage":null}
[chunk 4] 2024-8-4 2:10:49.375
{"id":"98d1cdb4-efb0-4a62-b4ec-6abaf128c596","object":"chat.completion.chunk","created":1722737448,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":425,"total_tokens":451,"completion_tokens":26}}
[stream finished] total chunks: 5

Stream 中存在一段 ECONNRESET

Error: aborted
    at abortIncoming (node:_http_server:797:17)
    at socketOnClose (node:_http_server:791:3)
    at Socket.emit (node:events:531:35)
    at TCP.<anonymous> (node:net:339:12)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 'ECONNRESET'
}

hezhijie0327 · 2024-08-04T02:45:49Z

神奇，现在 Stream 关闭后又可用了...

那现在就是 Stream 下有问题了...

lobehubbot · 2024-08-04T02:46:03Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Magic, now the Stream is available again after closing...

Then there is a problem under Stream...

hezhijie0327 · 2024-08-04T03:07:03Z

@arvinxx
遇到个很怪的问题，stream 中间会插一个 ECONNRESET，非 Stream 模式下就没事

Error: aborted
    at abortIncoming (node:_http_server:797:17)
    at socketOnClose (node:_http_server:791:3)
    at Socket.emit (node:events:531:35)
    at TCP.<anonymous> (node:net:339:12)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 'ECONNRESET'
}

This reverts commit c8422f4.

hezhijie0327 · 2024-08-04T06:21:25Z

对比了下其他 Provider，看上去是 Stream 第一个流中缺少 content?
{"id":"a3bf7f4f-bbcb-458b-8be4-5ff5e4c411bf","object":"chat.completion.chunk","created":1722751530,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}

Anyway, 暂时他们家也只有一个模型支持 function call，暂时禁用 stream 当 tools 存在时

arvinxx · 2024-08-05T07:43:10Z

@hezhijie0327 abort 大概率是 stream 的解析过程抛错导致。可能比较好的做法应该是等我搞完这个： #3361

然后针对 Firework 写一个自定义的 stream chunk 解析函数来规避 content 为空的情况

lobehubbot · 2024-08-05T07:43:23Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@hezhijie0327 abort is most likely caused by an error in the stream parsing process. Maybe a better approach would be to wait until I finish this: #3361

Then write a custom stream chunk parsing function for Firework to avoid the situation where content is empty.

arvinxx · 2024-08-05T07:44:18Z

OpenAI 的流默认解析规则在这里：https://github.com/lobehub/lobe-chat/blob/main/src/libs/agent-runtime/utils/streams/openai.ts#L15-L65

当然另外一种策略就是直接把 firework 的解析问题在 OpenAI 的流解析里也默认集成了，这样的好处就是能提升 OpenAI 流解析函数的鲁棒性，坏处就是存在耦合性

hezhijie0327 · 2024-08-05T07:52:32Z

那先等改造把，Spark 的 function call 我昨天也是试了下，他的 Stream 流会报一个 "e.message.tool_calls?.map is not a function 我过会儿补在 #3361

hezhijie0327 · 2024-09-08T00:23:40Z

@arvinxx 从昨天折腾 Bedrock Stream 时有了改造的思路

对以下两种情况进行优化：

tool_calls 存在，但 content 为空，引起 UI 对话界面显示空，但无报错

{"choices":[{"index":0,"message":{"role":"assistant","content":"","tool_calls":[{"id":"3NcHNntdRyaHu8zisKJAhQ","function":{"name":"realtime-weather____fetchCurrentWeather","arguments":"{\"city\": \"Singapore\"}"}}]},"stop_reason":"tool_calls"}]}

tool_calls.function 中，name 或 arguments 仅一项存在，引起 UI 界面报错，提示 "message": "[{"code": "invalid_type","expected": "string","received": "undefined","path": ["function","arguments"],"message": "Required"}]"

{"id":"c1c08c27-475e-4d8f-921b-557b9cb6ed68","object":"chat.completion.chunk","created":1725713803,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_VFFb6PEAADjxT0wk39AXJ5MQ","type":"function","function":{"name":"clock-time____getCurrentTime____standalone"}}]},"finish_reason":null}],"usage":null}`
{"id":"c1c08c27-475e-4d8f-921b-557b9cb6ed68","object":"chat.completion.chunk","created":1725713803,"model":"accounts/fireworks/models/firefunction-v2","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":null,"type":"function","function":{"arguments":" {"}}]},"finish_reason":null}],"usage":null}

arvinxx · 2024-09-10T09:20:19Z

@hezhijie0327 第一种状态看下能不能单独提出来拆一个 PR 看看？我刚有优化一个PR： #3872 ，有调整这部分，但是有人反馈说对一些模型有影响，看看你的方案是不是会更好一些

lobehubbot · 2024-09-10T09:20:33Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@hezhijie0327 In the first state, can you propose a separate PR and split it? I just optimized a PR: #3872. I have adjusted this part, but some people reported that it has an impact on some models. Let’s see if your solution is better.

arvinxx · 2024-09-10T09:30:53Z

@hezhijie0327 abort 大概率是 stream 的解析过程抛错导致。可能比较好的做法应该是等我搞完这个： #3361

另外我现在优化了一个实现，如果 stream 解析时候抛错了的话会直接传递 error chunk，所以这个应该可以先合了，后续有人反馈抛错可以再优化

lobehubbot · 2024-09-10T09:31:05Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@hezhijie0327 abort is most likely caused by an error in the stream parsing process. Maybe a better approach would be to wait until I finish this: #3361

In addition, I have now optimized an implementation. If an error is thrown during stream parsing, the error chunk will be passed directly, so this should be able to be combined first. If someone reports an error later, it can be optimized again.

hezhijie0327 · 2024-09-10T10:40:51Z

好的我过会儿拆一下

lobehubbot · 2024-09-10T10:41:03Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Okay, I'll take it apart later.

hezhijie0327 · 2024-09-10T11:57:17Z

@arvinxx 刚把 CI 修了下，没啥问题了

lobehubbot · 2024-09-10T11:57:28Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@arvinxx just fixed the CI and there is no problem anymore

vercel · 2024-09-10T15:59:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
lobe-chat-preview	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 10, 2024 4:08pm

lobehubbot · 2024-09-10T16:14:32Z

❤️ Great PR @hezhijie0327 ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。

## [Version 1.16.0](v1.15.35...v1.16.0) <sup>Released on **2024-09-10**</sup> #### ✨ Features - **misc**: Add Fireworks AI Model Provider, Add Spark model provider. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's improved * **misc**: Add Fireworks AI Model Provider, closes [#3392](#3392) [#48](#48) ([fa0d84d](fa0d84d)) * **misc**: Add Spark model provider, closes [#3098](#3098) [#25](#25) ([fc85c20](fc85c20)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

lobehubbot · 2024-09-10T16:21:50Z

🎉 This PR is included in version 1.16.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

## [Version 1.59.0](v1.58.16...v1.59.0) <sup>Released on **2024-09-11**</sup> #### ✨ Features - **misc**: Add Fireworks AI Model Provider, Add Spark model provider. #### 🐛 Bug Fixes - **misc**: Add `LLM_VISION_IMAGE_USE_BASE64` to support local s3 in vision model. #### 💄 Styles - **ui**: Improve UI layout and text. - **misc**: Reorder the provider list, Update CustomLogo, update spark check model to spark-lite & default disable useless model, update Upstage model list. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's improved * **misc**: Add Fireworks AI Model Provider, closes [lobehub#3392](https://github.com/bentwnghk/lobe-chat/issues/3392) [lobehub#48](https://github.com/bentwnghk/lobe-chat/issues/48) ([fa0d84d](fa0d84d)) * **misc**: Add Spark model provider, closes [lobehub#3098](https://github.com/bentwnghk/lobe-chat/issues/3098) [lobehub#25](https://github.com/bentwnghk/lobe-chat/issues/25) ([fc85c20](fc85c20)) #### What's fixed * **misc**: Add `LLM_VISION_IMAGE_USE_BASE64` to support local s3 in vision model, closes [lobehub#3887](https://github.com/bentwnghk/lobe-chat/issues/3887) ([16e57ed](16e57ed)) #### Styles * **ui**: Improve UI layout and text, closes [lobehub#3762](https://github.com/bentwnghk/lobe-chat/issues/3762) ([7c08f29](7c08f29)) * **misc**: Reorder the provider list, closes [lobehub#3886](https://github.com/bentwnghk/lobe-chat/issues/3886) ([4d641f5](4d641f5)) * **misc**: Update CustomLogo, closes [lobehub#3874](https://github.com/bentwnghk/lobe-chat/issues/3874) ([dd7c8df](dd7c8df)) * **misc**: Update spark check model to spark-lite & default disable useless model, closes [lobehub#3885](https://github.com/bentwnghk/lobe-chat/issues/3885) ([9d7e47c](9d7e47c)) * **misc**: Update Upstage model list, closes [lobehub#3890](https://github.com/bentwnghk/lobe-chat/issues/3890) ([82e2570](82e2570)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

✨ feat: Add Fireworks AI Model Provider

5ba4faf

hezhijie0327 marked this pull request as ready for review August 3, 2024 23:43

hezhijie0327 marked this pull request as draft August 4, 2024 01:04

💄 style: add FireworksAI self-models

04e3436

🐛 fix: support function call

c8422f4

hezhijie0327 marked this pull request as ready for review August 4, 2024 03:06

hezhijie0327 added 3 commits August 4, 2024 11:56

Revert "🐛 fix: support function call" (#48)

4f65076

This reverts commit c8422f4.

💄 style: update model info

d33a5a4

🐛 fix: disable stream if contains tools

4ff60a3

👷 build: add ENV for Fireworks AI provider

85952a0

hezhijie0327 added 8 commits August 5, 2024 20:40

Merge branch 'main' into fireworksai

a968889

Merge branch 'lobehub:main' into fireworksai

300991c

👷 build: update ENV

990d3c5

✨ feat: support proxy url

64932a3

🐛 fix: fix typo

4abe240

Merge branch 'main' into fireworksai

c624615

Merge branch 'main' into fireworksai

ff2c206

Merge branch 'main' into fireworksai

fba0c6a

hezhijie0327 added 3 commits September 10, 2024 19:02

🔨 chore: rollback changes

8c9eb74

Merge branch 'main' into fireworksai

0a88ec8

🐛 fix: fix CI error

d14d30b

hezhijie0327 mentioned this pull request Sep 10, 2024

♻️ refactor: change empty content stream behavior #3883

Merged

8 tasks

hezhijie0327 added 3 commits September 10, 2024 20:34

💄 style: update model list

a98e07b

Merge branch 'main' into fireworksai

f2ba16c

🔨 chore: fix rebase conflicts

e3291d4

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Sep 10, 2024

Merge branch 'main' into fireworksai

34506d8

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Sep 10, 2024

vercel bot deployed to Preview – lobe-chat-preview September 10, 2024 16:08 View deployment

arvinxx merged commit fa0d84d into lobehub:main Sep 10, 2024
7 of 8 checks passed

lobehubbot added the released label Sep 10, 2024

hezhijie0327 deleted the fireworksai branch September 11, 2024 01:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ feat: Add Fireworks AI Model Provider #3392

✨ feat: Add Fireworks AI Model Provider #3392

hezhijie0327 commented Aug 3, 2024 •

edited

Loading

vercel bot commented Aug 3, 2024

lobehubbot commented Aug 3, 2024

codecov bot commented Aug 3, 2024 •

edited

Loading

hezhijie0327 commented Aug 4, 2024 •

edited

Loading

hezhijie0327 commented Aug 4, 2024

lobehubbot commented Aug 4, 2024

hezhijie0327 commented Aug 4, 2024

hezhijie0327 commented Aug 4, 2024

arvinxx commented Aug 5, 2024

lobehubbot commented Aug 5, 2024

arvinxx commented Aug 5, 2024 •

edited

Loading

hezhijie0327 commented Aug 5, 2024

hezhijie0327 commented Sep 8, 2024

arvinxx commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

arvinxx commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

hezhijie0327 commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

hezhijie0327 commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

vercel bot commented Sep 10, 2024 •

edited

Loading

lobehubbot commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

✨ feat: Add Fireworks AI Model Provider #3392

✨ feat: Add Fireworks AI Model Provider #3392

Conversation

hezhijie0327 commented Aug 3, 2024 • edited Loading

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

vercel bot commented Aug 3, 2024

lobehubbot commented Aug 3, 2024

codecov bot commented Aug 3, 2024 • edited Loading

Codecov Report

hezhijie0327 commented Aug 4, 2024 • edited Loading

hezhijie0327 commented Aug 4, 2024

lobehubbot commented Aug 4, 2024

hezhijie0327 commented Aug 4, 2024

hezhijie0327 commented Aug 4, 2024

arvinxx commented Aug 5, 2024

lobehubbot commented Aug 5, 2024

arvinxx commented Aug 5, 2024 • edited Loading

hezhijie0327 commented Aug 5, 2024

hezhijie0327 commented Sep 8, 2024

arvinxx commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

arvinxx commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

hezhijie0327 commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

hezhijie0327 commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

vercel bot commented Sep 10, 2024 • edited Loading

lobehubbot commented Sep 10, 2024

lobehubbot commented Sep 10, 2024

hezhijie0327 commented Aug 3, 2024 •

edited

Loading

codecov bot commented Aug 3, 2024 •

edited

Loading

hezhijie0327 commented Aug 4, 2024 •

edited

Loading

arvinxx commented Aug 5, 2024 •

edited

Loading

vercel bot commented Sep 10, 2024 •

edited

Loading