feat(python backend): function call upgrade #368

EliwiiKeeya · 2024-08-01T12:33:23Z

This is a draft pull request to showcase the progress of my work on the function calling.

Changing few definations to fit OpenAI inference as new variables do not go well. Implementing the detection of funcion call type request by the existance of tool attribute, as function call request need to be specifically processed.

Modifying request classification logics, as mistake flags are leading to fault classification results.

- Add support for function calls in non-stream mode within the chatmode. - Update relevant modules and functions to handle non-stream responses. - Ensure compatibility with existing chatmode functionalities. Commit footprint: - Modify files: completion.py

Correct the data format to ensure compatibility with the OpenAI API. Update relevant modules and functions in completion.py. Details: Modify files: completion.py

This commit includes the implementation of a non-streaming function call process. It covers the entire workflow for invoking functions in a single call. The code has been simply passed the functional tests, but the content still need to be improved. Update relevant modules and functions in completion.py. Details: Modify files: completion.py Modify functions: eval_rwkv, chat_template, chat_with_tools,

I decided to move the directory structure from routes to backend-python because it aligns better with our project organization. This change won't impact existing functionality, but it will make the code clearer and easier to maintain. Details: Modify directory: tests Modify files: function_call.py

josStorer · 2024-08-02T04:15:56Z

#364

…cture Create postprocess_response function for text post-processing. Restore function rwkv_eval. Implement generation of tool_calls id. Details: Modify file(s): function_call.py

Add a file to test the ability to evaluate postprocess_response functions and post-process string functions. Details: Add file(s): tests/postprocss_response.py

Correct the data format to ensure compatibility with the OpenAI API, according to OpenAI API Reference docs. Update relevant modules and functions in completion.py. Details: Modify file(s): schema.py

Remove a data check field because it was already validated when it was passed in. Fix attribute name "logprobs". Details: Modify file(s): completion.py

backend-python/routes/completion.py

Modify the post-processing method of the string to match the inference results of the Mobius model. Synchronously modify the test file. Details: Modify File(s): completion.py, postprocess_response.py

Refactor the sinthesis method of append_message. Refactor the sinthesis method of tools_text. Details: Modify File(s): completion.py

josStorer · 2024-08-08T05:52:59Z

eb89837 做了一些调整，现在function_call测试脚本能够跑通

josStorer · 2024-08-08T06:09:02Z

考虑到现在 tool_choice 暂时仅开发 "auto" , 因此令服务端检测请求 body 中 tools 属性是否为 None, 将 tools 属性不为 None 视作开始输出 tool call 的标志, 此时产生一个 role 为 assistant, content 为 None 的响应
开始推理并持续检查推理结果, 直到推理结果中出现 <函数名>\n```<(缺省)编程语言>\ntool_call( 的字段, 中间不产生任何响应

参考 eb89837#diff-4e4a05876ad4fc0cfe0ed3cbdc219a3859cb0518161d9e959d52820520c87227R452, 只有当匹配到tool_call( 开始，确认AI进行了函数调用时，才返回null并开始tool_calls响应，同时stream发送函数名称。若ai回复出现了4个及以上的换行符，或累计字符串长度超过了40，或累计字符串长度超过30且没有出现过```三个点，则均视为非tool_calls响应，正常产生流式回复

先进行后处理, 例如出现的双引号(“) 替换为反斜杠+双引号 (")以及如果是最外层括号, 那么外面加上花括号等。

以及等于号替换为冒号

总体而言设想的stream流程没什么问题，只是不用立即返回一个content: null，而是确认找到函数名时再一同返回

Add a file for testing server-side completion stream transmission. Details: Add file(s): tests/function_call_stream.py

…ol calls. Create a draft of the stream_response_gen function, where I will implement generating a streaming function call later. Details: Modified File(s): completion.py

Adjust the print function. The current streaming test is still a draft because it is now testing choices[0].delta.content instead of choices[0].delta.tool_calls. Details: Modifiy File(s): tests/function_call_stream.py

Add an asynchronous generator that handles function call requests. The generator passes the test of distinguishing the non-function call response that is produced in the result of the inference. However, the functionality associated with function calls has not been tested. Details: Modify File(s): completion.py

EliwiiKeeya · 2024-08-09T12:18:42Z

我最新的 Commit c4b9c4e 尝试实现了 stream 模式下处理并响应包含 tools 的用户请求
其中简单测试通过了部分判断并产生正常流式回复的部分 (tests/function_call_stream.py)

AI 回复字数过少且未产生 markdown 代码块的情况
AI 回复累计字符串长度超过了 40 的情况

需要测试的情况:

其他 AI 回复不是 tool_calls 响应的情况
所有 AI 回复试图产生 tool_calls 响应的情况

我本地难以测试这些情况, 做调试及修 bug 的工作辛苦了。
或者, 如果有可能的远程资源提供的话, 可以由我可以进一步调试功能并修复潜在的漏洞。

josStorer · 2024-08-09T12:20:40Z

远程资源没有，我这边也是自己的电脑跑的😂

josStorer · 2024-08-09T12:21:58Z

我得后天再进行测试，明天周末休息一下吧😆

Use black formatter to change the code format, and these changes do not affect the operation of the code Remove unnecessary library. Details: Modify File(s): completion.py

josStorer · 2024-08-13T15:08:13Z

测试下来有一些问题, stream模式跑不通, 我在做调整

josStorer · 2024-08-14T10:02:18Z

做了一些调整: 现在stream模式的function call已经能够跑通: 3df4302

不过简单粗暴的等于号替换为冒号存在问题, 需要改为栈的方式处理

Modify the comments in the code to match the code content. This will not have an impact on functionality. Details: Modify File(s): completion.py

EliwiiKeeya · 2024-08-15T05:33:26Z

调整 stream 模式辛苦了。
现在 function call 还有需要调整的部分么?
如果没有, 我将转换为正式PR并提交 😄

josStorer · 2024-08-16T06:26:41Z

先转为正式PR提交吧，还有一些小细节我改一下

josStorer · 2024-08-16T06:26:56Z

非常感谢这么多天的工作！

* add function_call.py * feat(backend): add wip annotation * feat(backend): Synchronized Files * feat(Backend): detect function call request Changing few definations to fit OpenAI inference as new variables do not go well. Implementing the detection of funcion call type request by the existance of tool attribute, as function call request need to be specifically processed. * fix(Backend): detect function call request Modifying request classification logics, as mistake flags are leading to fault classification results. * feat(Backend): Implement function call in non-stream mode for chatmode - Add support for function calls in non-stream mode within the chatmode. - Update relevant modules and functions to handle non-stream responses. - Ensure compatibility with existing chatmode functionalities. Commit footprint: - Modify files: completion.py * fix(Backend): Fix OpenAI API data format compatibility Correct the data format to ensure compatibility with the OpenAI API. Update relevant modules and functions in completion.py. Details: Modify files: completion.py * feat(Backend): Implement non-streaming function call process This commit includes the implementation of a non-streaming function call process. It covers the entire workflow for invoking functions in a single call. The code has been simply passed the functional tests, but the content still need to be improved. Update relevant modules and functions in completion.py. Details: Modify files: completion.py Modify functions: eval_rwkv, chat_template, chat_with_tools, * fix(Backend): Move directory structure from routes to backend-python I decided to move the directory structure from routes to backend-python because it aligns better with our project organization. This change won't impact existing functionality, but it will make the code clearer and easier to maintain. Details: Modify directory: tests Modify files: function_call.py * fix(Backend): Update Prompts to Mobius example and refactor code structure Create postprocess_response function for text post-processing. Restore function rwkv_eval. Implement generation of tool_calls id. Details: Modify file(s): function_call.py * test(Backend): Add postprocss_response.py for tests Add a file to test the ability to evaluate postprocess_response functions and post-process string functions. Details: Add file(s): tests/postprocss_response.py * fix(Backend): Fix OpenAI API data format compatibility Correct the data format to ensure compatibility with the OpenAI API, according to OpenAI API Reference docs. Update relevant modules and functions in completion.py. Details: Modify file(s): schema.py * perf(Backend): Modify some attributes. Remove a data check field because it was already validated when it was passed in. Fix attribute name "logprobs". Details: Modify file(s): completion.py * fix(Backend): Fix postprocess_response funciton. Modify the post-processing method of the string to match the inference results of the Mobius model. Synchronously modify the test file. Details: Modify File(s): completion.py, postprocess_response.py * refactor(Backend): Refactor two places related to prompt word synthesis. Refactor the sinthesis method of append_message. Refactor the sinthesis method of tools_text. Details: Modify File(s): completion.py * 修正一些小细节使function_call工作正常 * test(Backend): Add function_call_stream.py Add a file for testing server-side completion stream transmission. Details: Add file(s): tests/function_call_stream.py * chore(Backend): Add a function to generate streaming responses for tool calls. Create a draft of the stream_response_gen function, where I will implement generating a streaming function call later. Details: Modified File(s): completion.py * fix(Backend): Adjust the output format of the streaming test file. Adjust the print function. The current streaming test is still a draft because it is now testing choices[0].delta.content instead of choices[0].delta.tool_calls. Details: Modifiy File(s): tests/function_call_stream.py * feat(Backend): Try to implement streaming function calls Add an asynchronous generator that handles function call requests. The generator passes the test of distinguishing the non-function call response that is produced in the result of the inference. However, the functionality associated with function calls has not been tested. Details: Modify File(s): completion.py * style(Backend): Change some format Use black formatter to change the code format, and these changes do not affect the operation of the code Remove unnecessary library. Details: Modify File(s): completion.py * stream function call细节调整 * docs(Backend): Modify comments in the code. Modify the comments in the code to match the code content. This will not have an impact on functionality. Details: Modify File(s): completion.py --------- Co-authored-by: josc146 <josStorer@outlook.com>

EliwiiKeeya added 9 commits July 28, 2024 18:55

add function_call.py

26077b8

feat(backend): add wip annotation

2e6bc03

feat(backend): Synchronized Files

f287e32

feat(Backend): detect function call request

542a9eb

Changing few definations to fit OpenAI inference as new variables do not go well. Implementing the detection of funcion call type request by the existance of tool attribute, as function call request need to be specifically processed.

fix(Backend): detect function call request

478151a

Modifying request classification logics, as mistake flags are leading to fault classification results.

fix(Backend): Fix OpenAI API data format compatibility

9cbef91

Correct the data format to ensure compatibility with the OpenAI API. Update relevant modules and functions in completion.py. Details: Modify files: completion.py

EliwiiKeeya added 3 commits August 2, 2024 16:56

fix(Backend): Update Prompts to Mobius example and refactor code stru…

432644d

…cture Create postprocess_response function for text post-processing. Restore function rwkv_eval. Implement generation of tool_calls id. Details: Modify file(s): function_call.py

test(Backend): Add postprocss_response.py for tests

b3994a3

Add a file to test the ability to evaluate postprocess_response functions and post-process string functions. Details: Add file(s): tests/postprocss_response.py

fix(Backend): Fix OpenAI API data format compatibility

0d06082

Correct the data format to ensure compatibility with the OpenAI API, according to OpenAI API Reference docs. Update relevant modules and functions in completion.py. Details: Modify file(s): schema.py

EliwiiKeeya force-pushed the feature/20240728/Eliwii_Keeya-function-calling branch from 035fca2 to 0d06082 Compare August 2, 2024 11:58

perf(Backend): Modify some attributes.

6dfc9ec

Remove a data check field because it was already validated when it was passed in. Fix attribute name "logprobs". Details: Modify file(s): completion.py

josStorer reviewed Aug 3, 2024

View reviewed changes

backend-python/routes/completion.py Outdated Show resolved Hide resolved

backend-python/routes/completion.py Outdated Show resolved Hide resolved

EliwiiKeeya and others added 3 commits August 3, 2024 16:37

fix(Backend): Fix postprocess_response funciton.

ff5c44f

Modify the post-processing method of the string to match the inference results of the Mobius model. Synchronously modify the test file. Details: Modify File(s): completion.py, postprocess_response.py

refactor(Backend): Refactor two places related to prompt word synthesis.

0345073

Refactor the sinthesis method of append_message. Refactor the sinthesis method of tools_text. Details: Modify File(s): completion.py

修正一些小细节使function_call工作正常

eb89837

josStorer mentioned this pull request Aug 8, 2024

function-calling-upgrade-sei:add wip annotation #364

Closed

EliwiiKeeya added 4 commits August 8, 2024 18:10

test(Backend): Add function_call_stream.py

8c93173

Add a file for testing server-side completion stream transmission. Details: Add file(s): tests/function_call_stream.py

chore(Backend): Add a function to generate streaming responses for to…

9747edf

…ol calls. Create a draft of the stream_response_gen function, where I will implement generating a streaming function call later. Details: Modified File(s): completion.py

fix(Backend): Adjust the output format of the streaming test file.

4917205

Adjust the print function. The current streaming test is still a draft because it is now testing choices[0].delta.content instead of choices[0].delta.tool_calls. Details: Modifiy File(s): tests/function_call_stream.py

style(Backend): Change some format

6b0f597

Use black formatter to change the code format, and these changes do not affect the operation of the code Remove unnecessary library. Details: Modify File(s): completion.py

stream function call细节调整

3df4302

docs(Backend): Modify comments in the code.

e88ee50

Modify the comments in the code to match the code content. This will not have an impact on functionality. Details: Modify File(s): completion.py

EliwiiKeeya marked this pull request as ready for review August 16, 2024 06:33

EliwiiKeeya changed the title ~~function-calling-upgrade-Eliwii_Keeya~~ feat(python backend): function call upgrade Aug 16, 2024

josStorer merged commit b6da6a8 into josStorer:dev Aug 17, 2024

josStorer added a commit that referenced this pull request Aug 27, 2024

improve #368 and python38 compatibility

120c98f

josStorer added a commit that referenced this pull request Aug 29, 2024

improve #368

c196638

HaloWang pushed a commit to HaloWang/RWKV-Runner that referenced this pull request Aug 30, 2024

improve josStorer#368 and python38 compatibility

d0324ff

HaloWang pushed a commit to HaloWang/RWKV-Runner that referenced this pull request Aug 30, 2024

improve josStorer#368

9080d83

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python backend): function call upgrade #368

feat(python backend): function call upgrade #368

EliwiiKeeya commented Aug 1, 2024

josStorer commented Aug 2, 2024

josStorer commented Aug 8, 2024

josStorer commented Aug 8, 2024 •

edited

Loading

EliwiiKeeya commented Aug 9, 2024 •

edited

Loading

josStorer commented Aug 9, 2024

josStorer commented Aug 9, 2024

josStorer commented Aug 13, 2024

josStorer commented Aug 14, 2024

EliwiiKeeya commented Aug 15, 2024

josStorer commented Aug 16, 2024

josStorer commented Aug 16, 2024

feat(python backend): function call upgrade #368

feat(python backend): function call upgrade #368

Conversation

EliwiiKeeya commented Aug 1, 2024

josStorer commented Aug 2, 2024

josStorer commented Aug 8, 2024

josStorer commented Aug 8, 2024 • edited Loading

EliwiiKeeya commented Aug 9, 2024 • edited Loading

需要测试的情况:

josStorer commented Aug 9, 2024

josStorer commented Aug 9, 2024

josStorer commented Aug 13, 2024

josStorer commented Aug 14, 2024

EliwiiKeeya commented Aug 15, 2024

josStorer commented Aug 16, 2024

josStorer commented Aug 16, 2024

josStorer commented Aug 8, 2024 •

edited

Loading

EliwiiKeeya commented Aug 9, 2024 •

edited

Loading