Skip to content

[feat][backend] code evaluator#220

Merged
lsy357 merged 41 commits intorelease/v1.4.0from
feat/open-code-eval
Oct 17, 2025
Merged

[feat][backend] code evaluator#220
lsy357 merged 41 commits intorelease/v1.4.0from
feat/open-code-eval

Conversation

@tpfz
Copy link
Collaborator

@tpfz tpfz commented Oct 10, 2025

What type of PR is this?

Check the PR title.

  • This PR title match the format: [<type>][<scope>]: <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese.

(Optional) More detailed description for this PR(en: English/zh: Chinese).

en:
zh(optional):

(Optional) Which issue(s) this PR fixes:

tpfz and others added 2 commits October 10, 2025 19:16
(LogID: 202510101913360100911101348473F04)

Co-Authored-By: Coda <coda@bytedance.com>
@codecov
Copy link

codecov bot commented Oct 10, 2025

Codecov Report

❌ Patch coverage is 81.67457% with 116 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ules/evaluation/infra/runtime/http_faas_runtime.go 68.20% 66 Missing and 3 partials ⚠️
...modules/evaluation/infra/runtime/python_runtime.go 85.22% 11 Missing and 2 partials ⚠️
...ackend/modules/evaluation/infra/runtime/factory.go 88.77% 8 Missing and 3 partials ⚠️
...les/evaluation/infra/runtime/javascript_runtime.go 88.88% 9 Missing and 1 partial ⚠️
...ackend/modules/evaluation/infra/runtime/manager.go 91.01% 6 Missing and 2 partials ⚠️
backend/modules/evaluation/pkg/conf/evaluator.go 90.00% 5 Missing ⚠️

Impacted file tree graph

@@                Coverage Diff                @@
##             release/v1.4.0     #220   +/-   ##
=================================================
  Coverage                  ?   64.27%           
=================================================
  Files                     ?      507           
  Lines                     ?    56715           
  Branches                  ?        0           
=================================================
  Hits                      ?    36451           
  Misses                    ?    17745           
  Partials                  ?     2519           
Flag Coverage Δ
unittests 64.27% <81.67%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...nd/api/handler/coze/loop/apis/evaluator_service.go 4.34% <100.00%> (ø)
backend/modules/evaluation/pkg/conf/evaluator.go 76.85% <90.00%> (ø)
...ackend/modules/evaluation/infra/runtime/manager.go 91.01% <91.01%> (ø)
...les/evaluation/infra/runtime/javascript_runtime.go 88.88% <88.88%> (ø)
...ackend/modules/evaluation/infra/runtime/factory.go 88.77% <88.77%> (ø)
...modules/evaluation/infra/runtime/python_runtime.go 85.22% <85.22%> (ø)
...ules/evaluation/infra/runtime/http_faas_runtime.go 68.20% <68.20%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cab8bde...6cab99d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

wec4dre96-w and others added 20 commits October 10, 2025 20:14
(LogID: 202510111803140100911101340639242)

Co-Authored-By: Coda <coda@bytedance.com>
…rectory

- Enhanced runtime package test coverage from 41.3% to 86.1%
- Added comprehensive tests for RuntimeFactory, RuntimeManager, HTTPFaaSRuntimeAdapter
- Extended PythonRuntime and JavaScriptRuntime testing with return_val functions
- Added concurrent access testing and error handling verification
- Implemented mock-based testing patterns following existing code style
- 移除测试文件中的多余空行
- 修复代码缩进不一致问题
- 使用go fmt格式化代码

这些修改仅涉及格式问题,未更改任何业务逻辑。
- 格式化运行时相关文件
- 修复代码缩进和空白字符问题
- 保持业务逻辑不变
- Fixed unchecked error returns in http_faas_runtime.go
- Added safe env var helper functions in runtime_test.go
- Removed unnecessary type conversions in evaluator.go
- All packages now pass golangci-lint checks
- Added missing Apache-2.0 license headers to 53 frontend TypeScript/TSX files
- Fixed 48 files automatically using license-eye tool
- Manually added headers to 5 .less files that weren't auto-fixable
- All files now pass license header validation
This reverts commit 2cee2c1.
- 统一docker和k8s版本的响应格式
- 实现Pyodide预加载机制,使用vendor离线依赖
- 优化进程池管理,支持预创建进程
- 改进代码执行逻辑,使用临时文件避免转义问题
- 添加完整的错误处理和日志记录
- 支持return_val函数和多种返回值格式提取
- 删除ModelConfig配置
@tpfz tpfz force-pushed the feat/open-code-eval branch from 777a25a to 4879e6e Compare October 15, 2025 06:52
Copy link
Collaborator

@CozeLoop CozeLoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary:

  • Must-Fix: JS return_val is not captured by the current JS FaaS wrapper because it logs a raw string. Please emit a single JSON line (e.g., console.log(JSON.stringify({ret_val}))) so the server can parse it, or add an explicit marker protocol.
  • Should-Fix: Provide fail-open defaults for COZE_LOOP_JS_FAAS_URL and COZE_LOOP_PYTHON_FAAS_URL to improve robustness across dev/test environments.
  • Suggestion: Use normalizeLanguage(language) in the HTTP adapter, and include TypeScript aliases (ts/typescript) in config key normalization.

These changes will restore functional correctness for JS evaluators, reduce configuration friction, and make the runtime adapter more resilient.

Copy link
Collaborator

@CozeLoop CozeLoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aime CR summary: Backend runtime integration (factory/manager + HTTP FaaS adapter) is correct and concurrency-safe. ValidateEvaluator handler correctly restores service invocation. Config decoding and language normalization look good. No Must-Fix or Should-Fix issues found on backend Go changes. A few non-blocking suggestions: centralize default FaaS BaseURL config, remove unused basicSyntaxValidation or wire it into ValidateCode, and consider unifying logs to English for consistency. Full report is available in Feishu: https://bytedance.larkoffice.com/docx/RbfJdgpZsoqiBhxJ6epcqYWXnQf

@lsy357 lsy357 changed the base branch from main to release/v1.4.0 October 17, 2025 05:51
@lsy357 lsy357 merged commit 0ff3e30 into release/v1.4.0 Oct 17, 2025
7 checks passed
@lsy357 lsy357 deleted the feat/open-code-eval branch October 17, 2025 09:46
lsy357 added a commit that referenced this pull request Oct 20, 2025
* fix(observability): chart cfg

* [feat][backend] code evaluator (#220)

---------

Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: wuwenqi <wuwenqi.depp@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>

* feat(all): release v1.4.0

---------

Co-authored-by: tpfz <155960978+tpfz@users.noreply.github.com>
Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: wuwenqi <wuwenqi.depp@bytedance.com>
wec4dre96-w added a commit that referenced this pull request Oct 21, 2025
* feat: code evaluator

---------

Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: wuwenqi <wuwenqi.depp@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>
wec4dre96-w added a commit that referenced this pull request Oct 21, 2025
* [feat][backend] code evaluator (#220)

* feat: code evaluator

---------

Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: wuwenqi <wuwenqi.depp@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>

* fix: 补充路由

* fix: 评估器创建参数模型初始化

* fix: 隐藏筛选入口

* fix: 实验创建问题修复

* fix: 评估器问题修复

* fix: 回退配置文件

* fix: 回退配置文件

---------

Co-authored-by: tpfz <155960978+tpfz@users.noreply.github.com>
Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>
wec4dre96-w added a commit that referenced this pull request Oct 21, 2025
* [feat][backend] code evaluator (#220)

* feat: code evaluator

---------

Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: wuwenqi <wuwenqi.depp@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>

* fix: 补充路由

* fix: 评估器创建参数模型初始化

* fix: 隐藏筛选入口

* fix: 实验创建问题修复

* fix: 评估器问题修复

* fix: 回退配置文件

* fix: 回退配置文件

---------

Co-authored-by: tpfz <155960978+tpfz@users.noreply.github.com>
Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>
wec4dre96-w added a commit that referenced this pull request Oct 21, 2025
* [feat][backend] code evaluator (#220)

* feat: code evaluator

---------





* fix: 补充路由

* fix: 评估器创建参数模型初始化

* fix: 隐藏筛选入口

* fix: 实验创建问题修复

* fix: 评估器问题修复

* fix: 回退配置文件

* fix: 回退配置文件

---------

Co-authored-by: tpfz <155960978+tpfz@users.noreply.github.com>
Co-authored-by: Coda <coda@bytedance.com>
Co-authored-by: liushengyang <liushengyang@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants