Release notes from DataMate

v0.3.0

2026-01-31T08:32:24Z

版本定位

DataMate 本版本聚焦数据治理能力增强与多租户隔离：补齐数据血缘、数据采集能力，完善标注工作流与算子生态，统一模型配置与数据权限，提升平台稳定性和可运维性。

核心功能更新

模块	更新内容
数据归集	• 新增：API Reader 采集插件，支持将API接口返回的数据以CSV文件的形式归集到DataMate。 • 新增：通用关系型数据库采集模板（RDBMS Reader），增加对于postgres、opengauss、sqlserver等关系型数据库的归集支持。
数据处理	• 变更：更新数据处理任务算子展示方式 • 新增：重试次数展示和日志根据重试次数查看
数据管理	• 新增：数据血缘页，实现数据归集、数据管理、数据清洗过程中数据血缘的追溯 • 新增：上传时支持按文件夹上传；前端预览文本和图片文件内容；前端查看已标注文件的标签 • 修复：未入库文件元数据无法查看，无法预览内容
数据标注	• 优化：整合并完善辅助标注和人工标注的同步逻辑，实现标注工具标注数据和数据集数据的双向同步 • 优化：保留一级操作（同步、编辑），并将低频操作（删除任务、编辑任务数据集、导出标注结果）收敛至二级菜单。 • 修复：分页参数错误导致的前端展示问题
算子市场	• 新增：算子按功能分类 • 新增：算子文档与示例补充 • 优化：前端展示（Card、OperatorServiceMonitor、Requirement 等组件）
知识生成	• 优化：知识图谱改用 2D 力导向图、渲染逻辑调整
模型和配置	• 重构：模型配置统一、LLM 客户端工厂标准化 • 优化：会话与模型创建时的用户上下文追踪
部署模块	• 新增：Docker 镜像按分支打 tag • 新增：MinerU 适配 310P、构建参数优化 • 新增：SECURITY.md 安全策略说明 • 修复：数据库日志目录权限、图片相似度重复重试误过滤 • 修复：编译阶段 three.js 依赖缺失
用户管理	• 新增：数据归集、数据集、清洗、合成、配比、评估、知识库、算子按创建者隔离，系统预置数据不隔离 • 新增：DataSetScope 数据权限、UserContext 传递

What's Changed

feat(data-management): add preview functionality for text and image items by @o0Shark0o in #259
fix: 添加编译阶段three.js依赖 by @Dallas98 in #261
fix: 增加three依赖 by @hhhhsc701 in #262
fix: 数据库修改日志目录权限/修复图片相似度重复重试任务时误过滤 by @hhhhsc701 in #264
Fix: Annotation template paginate can not show when template count > 12 by @q792602257 in #260
feat: refactor KnowledgeGraphView to use 2D force graph and improve rendering logic by @Dallas98 in #266
fix: 翻页参数问题 by @q792602257 in #265
feat(data-management): fix bulk upload issues and enhance UI & upload experience by @o0Shark0o in #269
feat(auto-annotation): sync tags and timestamps to datasets and optimize visibility by @o0Shark0o in #271
feat: 优化mineru构建部署参数，适配310P by @hhhhsc701 in #270
算子前端展示更新 by @hhhhsc701 in #273
realize that data sets, data cleaning, and knowledge bases are isolated according to the creator, and operators are not isolated. by @hefanli in #268
feat: 增加deepwiki链接 by @hhhhsc701 in #278
feat: 清洗任务增加重试次数记录和日志展示 by @hhhhsc701 in #280
feat: 数据库改为单例模式 by @hhhhsc701 in #281
feat: 补充算子文档和示例 by @hhhhsc701 in #282
feat(annotation): add bidirectional sync and flexible export for annotation tasks by @o0Shark0o in #284
feat: enhance dataset file download functionality to zip all files and improve path validation by @Dallas98 in #285
add data lineage page and data quality page by @hefanli in #287
Enhance README with new fields and unit updates by @hhhhsc701 in #288
feat: update Docker image tagging to use branch names for better identification by @Dallas98 in #289
feat: rename data cleansing references to data processing for consistency by @Dallas98 in #292
Create SECURITY.md for security policy by @yafengzhang2025 in #293
feat(annotation): simplify task creation and streamline sync workflow by @o0Shark0o in #294
增加数据库插入 by @hhhhsc701 in #295
增加空值处理 by @hhhhsc701 in #298
feat(annotation): refine sync behavior and add annotation export option by @o0Shark0o in #299
fixed the problem that the metadata of files that have not yet been stored in the database cannot be previewed. by @hefanli in #300
add apireader collection plug-in by @hefanli in #301
feature: added universal relational database collection template by @hefanli in #302
fixed issues where files inside folders could not be previewed and deleted by @hefanli in #303
refactor(models): unify model configuration and standardize LLM client factory by @Dallas98 in #283
feat(operator-market): reorganize operator categories by functionality by @o0Shark0o in #304
feat: enhance user tracking in session management and model creation by using effective user context by @Dallas98 in #305
fix: adapt to structural changes of COT data by @hefanli in #306

Full Changelog: v0.2.0...v0.3.0

v0.2.0

2026-01-16T08:54:38Z

1. 核心模块迭代

模块	更新内容	价值与影响
数据归集	将原来的OBS归集模板修改为S3归集模板，增强对于S3存储的支持能力增加MYSQL归集模板、StarRocks归集模板、GlusterFS归集模板	增加了数据归集支持的外部系统，丰富了数据归集的能力。
数据处理	支持清洗到已创建的数据集中；清洗任务创建筛选逻辑优化，支持多选；更新paddlepaddle图像文本方向识别模型；支持清洗表格标签；	提升数据处理的灵活性、效率。
数据管理	数据管理内增加新建文件夹功能，可在数据集中使用文件夹；数据管理中增加‘数据/文件夹’重命名功能；优化‘数据集’根据数据类型的图标和微动态互动效果。	增强了数据管理的组织与可维护能力，提升了复杂数据集在使用中的灵活性。提升了界面可读性与操作直观性。
数据标注	新增图像目标检测自动标注功能，支持80类可选分类；自动标注结果已可以与Label-Studio进行互联；自动/手动标注创建任务可以任选数据集和数据。	降低了人工标注成本，加快了大规模数据集的标注效率。提升了标注流程的灵活性与工程落地效率。
知识生成	知识生成新增生成知识图谱能力：支持.txt、.md、.doc、.docx、.pdf等泛文本文件生成知识图谱前端支持显示知识图谱关系，支持查询节点与边的详情	将非结构化文本转化为可视化、可检索的知识关联网络，可大幅提升 RAG 数据质量和效率
部署模块	重构docker-compose.yml文件，支持随版本发布docker-compose.yml支持配置证书以开启https，支持加密后的密钥统一数据库密码配置入口，密码存放到secret中支持ModelEngine对接，前端菜单能跳转到DataMate首页	简化docker安装方式，支持单行一键安装增加系统安全性
用户模块	增加用户登录和注册功能，默认部署不对用户进行要求用户功能开启后，仅可查看首页，其余页面点击后弹出登录弹窗	增加用户功能，增加系统安全性

2. 体验、兼容性优化与问题修复

开放数据集、数据清洗、算子市场相关接口mcp能力，支持通过Agent创建清洗任务
算子市场，对各个数据种类的算子进行了图标与颜色的全新设计，使前端更加美观，粗略可读性更强
数据库切换，由mysql切换为pgsql，去除GPL以便公司内部开源引入，且与LabelStudio的数据库合并，减少部署时资源消耗
修复数据清洗任务创建时切换选中算子导致已配置的参数丢失问题
修复算子市场分页状态在「全选 / 细分分类」切换时未重置，导致分页错乱及空结果问题
修复设置界面前端拉伸时会出现无限拉伸的情况。

What's Changed

fix: ray部署修改 by @hhhhsc701 in #221
feature: add mysql collection and starrocks collection by @hefanli in #222
feat(auto-annotation): integrate YOLO auto-labeling and enhance data management by @o0Shark0o in #223
fix the ratio task config by @hefanli in #224
优化合成任务数据集选择交互体验 by @o0Shark0o in #225
fix: 修复入库可能重复;筛选逻辑优化 by @hhhhsc701 in #226
feat: mcp支持创建清洗任务 by @hhhhsc701 in #227
feature: could create dataset while creating collection task by @hefanli in #228
feat(auto-annotation): integrate auto-label results with Label Studio and improve task creation by @o0Shark0o in #230
feat: enhance Docker configuration with additional services and profiles by @Dallas98 in #231
feature: support data intelligence orchestation front by @yafengzhang2025 in #232
feat: update knowledge base file detail view with pagination and metadata display by @Dallas98 in #233
feat(text-annotation): improve manual text annotation and enable visibility in Label Studio by @o0Shark0o in #235
refactor: replace the database from mysql to pgsql by @hefanli in #236
feat: enhance Makefile and README for improved image download process and user prompts by @Dallas98 in #237
fix: fix the LabelStudio jumps without login by @hefanli in #238
fix: 修复paddleocr模型报错 by @hhhhsc701 in #239
feat: enhance Makefile and update logging to mask database URL in main.py by @Dallas98 in #240
feat(operator): add YOLOv8 image object detection operator by @o0Shark0o in #241
feat: Add Glusterfs LocalFs S3-CompatibleFs Support by @q792602257 in #229
feat: nginx支持https/密码通过secret存放 by @hhhhsc701 in #242
fix: 修复参数传递问题 by @hhhhsc701 in #243
feature: add user login and signup functions by @hefanli in #244
fix(auto-annotation): resolve task bugs and add dataset and folder renaming support by @o0Shark0o in #245
fix: pg数据库环境变量 by @hhhhsc701 in #247
feat: 支持通过configmap挂载证书并解密 by @hhhhsc701 in #248
feat: 支持清洗到已有数据集中 by @hhhhsc701 in #249
add user-related interactions on the frontend by @hefanli in #250
fix: 修复https证书配置 by @hhhhsc701 in #252
feat: 新增知识图谱功能 by @Dallas98 in #251
no login required by default by @hefanli in #253
feat: 支持清洗表格标签 by @hhhhsc701 in #254
feat: add graph RAG service routing and storage configuration by @Dallas98 in #255
feat: add lightrag-hku dependency to pyproject.toml by @Dallas98 in #256
fix: 修复日志打印 by @hhhhsc701 in #257

New Contributors

@yafengzhang2025 made their first contribution in #232
@q792602257 made their first contribution in #229

Full Changelog: v0.1.0...v0.2.0

v0.1.0

2026-01-08T11:28:39Z

🔔 核心功能 / Key Enhancements

1️⃣ 数据归集：将外部系统的数据导入到Datamate / Data Collection: Importing Data from External Systems into Datamate

1）支持将外部存储系统（nas、s3存储）上的文件导入到Datamate上。
2）支持将关系型数据库（Mysql）、数仓数据存储（Starrocks）上的数据以csv文件的形式导入到Datamate上。
3）归集流程可追溯，能够查看归集任务的详细日志。

1)Supports importing files from external storage systems (NAS, S3 storage) into Datamate.
2)Supports importing data from relational databases (MySQL) and data warehouse storage (Starrocks) into Datamate in the form of CSV files.
3)The collection process is traceable, allowing users to view detailed logs of collection tasks.

2️⃣ 数据管理：提供文件粒度的数据集管理能力 / Data Management: Providing File-Granularity Dataset Management Capability

以文件粒度进行数据管理，支持文件标签管理和分布分析，是整个数据工作平台的核心。

Manages data at the file granularity level, supports file tagging management and distribution analysis, serving as the core of the entire data work platform.

3️⃣ 丰富的算子生态和灵活可监控的数据清洗流程 / Rich Operator Ecosystem and Flexible, Monitorable Data Cleaning Pipelines

1）集成了200+算子，拥有文本、图片、音频、视频多种模态的数据处理能力。
2）简单易上手的自定义算子开发指导，支持用户根据自身需要进行自定义算子开发。
3）基于算子进行数据清洗流程编排，清洗全程日志可视，对清洗过程可以进行监控和分析。

1)Integrates 200+ operators, offering data processing capabilities for text, images, audio, and video across multiple modalities.
2)Provides user-friendly, easy-to-follow custom operator development guidance, supporting users in developing custom operators according to their specific needs.
3)Enables data cleaning pipeline orchestration based on operators, with full-process logging visualization, allowing monitoring and analysis of the cleaning process.

4️⃣ 智能数据合成和配比 / Intelligent Data Synthesis and Proportioning

1）通过工具链与智能算法，从PDF、纯文本等泛文本文档中提取信息，经清洗、格式转换后，生成结构化QA对、思维链（COT）数据，形成适配模型训练的高质量数据集。
2）基于模型需求智能调整数据比例：支持以下标签配比、标签更新时间配比、数量配比等配比策略。

1)Utilizes toolchains and intelligent algorithms to extract information from general text documents such as PDFs and plain text. After cleaning and format conversion, it generates structured QA pairs and Chain-of-Thought (CoT) data, forming high-quality datasets suitable for model training.
2)Intelligently adjusts data proportions based on model requirements: supports proportioning strategies such as label-based proportioning, label update time proportioning, quantity proportioning, etc.

5️⃣ 面向RAG的知识库构建 / RAG-Oriented Knowledge Base Construction

1）支持按照章节、段落、固定长度、自定义规则进行知识切片，确保知识切片是完整可用的。
2）基于 Milvus 2.6 构建向量知识库

1)Supports knowledge slicing by chapters, paragraphs, fixed lengths, and custom rules, ensuring that knowledge slices are complete and usable.
2)Builds a vector knowledge base based on Milvus 2.6.

6️⃣ 深度集成 LabelStudio，提供多模态标注 / Deep Integration with LabelStudio, Providing Multi-Modal Annotation

提供图像分类、关键词抽取、目标检测等多个标注模板，支持文本、图片等多模态数据标注能力

Offers various annotation templates including image classification, keyword extraction, object detection, etc., supporting multi-modal data annotation capabilities for text, images, and more.

🧭 特性预览 / Feature Preview

1️⃣ 数据血缘，基于有向无环图实现全链路血缘追踪。/ Data Lineage: Implements full-link lineage tracking based on Directed Acyclic Graphs (DAGs).

2️⃣ 自动标注能力，基于模型提供分类、目标检测等自动标注能力，构建人机协助标注体系。/ Automatic Annotation Capability: Provides automatic annotation capabilities such as classification and object detection based on models, building a human-machine collaborative annotation system.

3️⃣ 用户接入，提供基于用户的数据隔离能力。/ User Access: Offers data isolation capabilities based on user accounts.

What's Changed

feat: add GitHub Actions workflow for releasing Docker images by @Dallas98 in #136
feat(SynthDataDetail): add chunk/synthesis data management with edit/delete & UI enhancements by @Dallas98 in #139
feature: 增加压缩包上传功能 by @hefanli in #137
feat(config): add proxy configuration for evaluation API endpoint by @Dallas98 in #141
fix: k8s部署场景下，backend-python服务挂载需要存储 by @hefanli in #144
feat(dependencies): update greenlet to version 3.3.0 and adjust Python version compatibility by @Dallas98 in #145
docs: update README and Makefile for clarity and new development instructions by @Dallas98 in #147
修复部分功能 by @hhhhsc701 in #138
feature: data management supports nested folders by @hefanli in #150
feat(chart): add Helm chart for deploying Label Studio with PostgreSQL by @Dallas98 in #152
feature: 增加水印去除/高级匿名化算子 by @hhhhsc701 in #151
feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings by @Dallas98 in #153
fix: fix the Data Evaluation Detail page by @hefanli in #154
feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings by @Dallas98 in #155
feat: enhance dataset file fetching with improved pagination and document loading support by @Dallas98 in #156
feature: 增加data-juicer算子 by @hhhhsc701 in #157
Revert "feature: 增加data-juicer算子" by @hhhhsc701 in #158
fix: update MILVUS_URI in .env.example for correct service endpoint by @hhhhsc701 in #159
feat: add .env and conf.yaml for deer-flow configuration by @hhhhsc701 in #160
docs: update README files to include instructions for accessing the f… by @hhhhsc701 in #161
Fix ratio by @hefanli in #162
feat(frontend): 增强Synthesis Data Detail页面UX体验 by @Dallas98 in #163
bugfix by @hhhhsc701 in #164
补充modal范围 by @hhhhsc701 in #165
feature: 商业版构建 by @hhhhsc701 in #166
feature: add the pipeline for pushing images to Huawei Cloud by @hefanli in #167
修改系统参数的预置数据 by @hefanli in #170
fix: the interface for querying data set files is compatible with ret… by @hefanli in #171
bugfix: update values.yaml to enhance ray-cluster configuration with security context, environment variables, and resource limits by @hhhhsc701 in #172
fix: update Dockerfile to improve pip installation process and remove unnecessary uninstalls by @hhhhsc701 in #173
支持mineru npu处理 by @hhhhsc701 in #174
fix: streamline Dockerfile by removing redundant mirror configuration… by @hhhhsc701 in #175
fix(docker-compose): update entrypoint and command for mineru-openai-server configuration by @Dallas98 in #176
fix: optimize PDF parsing by implementing concurrent processing with … by @hhhhsc701 in #177
feat：问题生成过程优化及COT数据生成优化 by @Dallas98 in #169
feat(model_chat): enhance JSON parsing by removing additional thought tags and improving fallback logic by @Dallas98 in #178
feat: add operator-packages-volume to docker-compose and update Docke… by @hhhhsc701 in #179
feat(generation_service): add document filtering to remove short documents based on chunk size by @Dallas98 in #180
add export type settings and enhance metadata structure by @hhhhsc701 in #181
feat(generation_service): add image URL extraction and random QA generation logic by @Dallas98 in #182
配置文件更改 by @Dallas98 in #186
feat: add download functionality for example operator and update Dock… by @hhhhsc701 in #188
add gateway by @hefanli in #187
fix: fix the routes definition by @hefanli in #189
Feat gateway by @hefanli in #191
fix: prevent deletion of predefined operators and improve error handling by @hhhhsc701 in #192
feat: deer-flow通过gateway转发 by @hhhhsc701 in #193
Fix the ratio task bug by @hefanli in #194
feat: add CodeQL analysis workflow configuration by @Dallas98 in #196
feat: 支持npu自动扩缩容 by @hhhhsc701 in #197
feat: label-studio构建脚本 by @hhhhsc701 in #198
feat: label-studio支持指定sc by @hhhhsc701 in #200
feature: LabelStudio jumps without login by @hefanli in #201
feat: fix the problem in the Operator Market frontend pages by @o0Shark0o in #203
Revert "feat: fix the problem in the Operator Market frontend pages" by @o0Shark0o in #204
fix: reset pagination when switching operator market category filters by @o0Shark0o in #205
feat: 增加label-studio的k8s部署卸载 by @Dallas98 in #206
feat: 修复清洗任务创建 by @hhhhsc701 in #207
refactor: modify data collection to python implementation by @hefanli in #214
feat: 支持运行data-juicer算子 by @hhhhsc701 in #215
feat: 算子上传适配 by @hhhhsc701 in #216
polish(operator cards): improve icon color distinction and subtle UI … by @o0Shark0o in #217
fix: fix the collection for nfs by @hefanli in #218
feat: update Docker configuration to include backend Nginx settings by @Dallas98 in #219

New Contributors

@o0Shark0o made their first contribution in #203

Full Changelog: v0.0.1-beta...v0.1.0

v0.0.1-beta

2025-12-05T09:56:43Z

Full Changelog: v0.0.1-beta...v0.0.1-beta

What's Changed

feat: Add download and load images functionality to Makefile by @JasonW404 in #115
Integrated Redux for state management with auth and settings slices. by @chenghh-9609 in #117
bugfix: 清洗/算子支持名称/描述搜索 by @hhhhsc701 in #116
bugfix: 支持使用runtime单实例 by @hhhhsc701 in #118
feat: new endpoint allowing only add file path to dataset record without any FS operations by @JasonW404 in #119
feat: Add original file ID to document metadata in RagEtlService by @Dallas98 in #121
feat: Implement data synthesis task management with database models and API endpoints by @Dallas98 in #122
feat: Update Python version requirements and refine dependency constraints in configuration by @Dallas98 in #123
bugfix: 修复deer-flow部署 by @hhhhsc701 in #124
fix: Correctly extract file values from selectedFilesMap in AddDataDialog by @Dallas98 in #125
优化部分问题 by @hhhhsc701 in #126
feature: 修改算子开发指南 by @hhhhsc701 in #127
feature: add data-evaluation by @hefanli in #128
feat: enhance backend deployment, frontend file selection and synthesis task management by @Dallas98 in #129
feat(synthesis): add chunk-level synthesis data detail page & refine APIs/routing by @Dallas98 in #130
feature: 增加对redis未部署时异常捕获 by @hhhhsc701 in #131
feat(synthesis): add functionality to archive synthesis tasks to existing datasets by @Dallas98 in #132
fix: 修复评估时模型输出json格式不对导致读取错误的问题 by @hefanli in #133
算子将抽取与落盘固定到流程中 by @hhhhsc701 in #134

New Contributors

@chenghh-9609 made their first contribution in #12
@JasonW404 made their first contribution in #13
@Startalker made their first contribution in #17
@szc0616 made their first contribution in #59

Full Changelog: https://github.com/ModelEngine-Group/DataMate/commits/v0.0.1-beta