Skip to content

feat(datasets): add DatasetMetadataClient for pure DB-backed metadata management#1171

Open
jake11-oho wants to merge 1 commit into
alibaba:masterfrom
jake11-oho:feat/datasets-sdk-v2
Open

feat(datasets): add DatasetMetadataClient for pure DB-backed metadata management#1171
jake11-oho wants to merge 1 commit into
alibaba:masterfrom
jake11-oho:feat/datasets-sdk-v2

Conversation

@jake11-oho

@jake11-oho jake11-oho commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

引入纯 DB 元数据客户端 DatasetMetadataClient,与 OSS 文件操作彻底解耦。

  • 新增 Database ORM 模型(Dataset, Instance, Image, Permission, AuditEvent)
  • 新增 DbDatasetRegistry:PostgreSQL CRUD + SQLite 方言支持
  • 新增 DatasetMetadataClient:面向用户的元数据 SDK
  • 新增数据模型(PageResult, DatasetInfo, TaskEntry, ImageInfo 等)
  • 41 个单元测试覆盖 dataset/instance/image/permission/audit CRUD

架构设计

DB 管元数据,OSS 管文件,两者独立使用:

# 元数据操作(纯 DB)
from rock.sdk.envhub.datasets import DatasetMetadataClient
meta = DatasetMetadataClient(db_url)
meta.register_dataset("org", "name")

# 文件操作(纯 OSS,不在本 PR 范围)
from rock.sdk.envhub.datasets.registry.oss import OssDatasetRegistry
oss = OssDatasetRegistry(registry_info)
oss.download_task(...)

fixes #1170

Test plan

  • 41 unit tests for DatasetMetadataClient (dataset/instance/image/permission/audit CRUD)
  • Uses SQLite in-memory DB for fast testing without PostgreSQL
  • CI full fast tests pass (1794 passed, 0 regressions)

🤖 Generated with Claude Code

@jake11-oho
jake11-oho force-pushed the feat/datasets-sdk-v2 branch from b57f4bc to 2958c3b Compare July 1, 2026 03:39
@jake11-oho jake11-oho changed the title feat(datasets): enhance SDK with pagination, file ops, query APIs, and format parsing feat(datasets): enhance SDK with pagination, file ops, query APIs, and DB-backed metadata Jul 1, 2026
@jake11-oho jake11-oho changed the title feat(datasets): enhance SDK with pagination, file ops, query APIs, and DB-backed metadata feat(datasets): enhance SDK with pagination, file ops, query APIs, and separated DB/OSS architecture Jul 1, 2026
@jake11-oho
jake11-oho force-pushed the feat/datasets-sdk-v2 branch 2 times, most recently from 8398ecc to 2c62b01 Compare July 1, 2026 05:25
@jake11-oho jake11-oho changed the title feat(datasets): enhance SDK with pagination, file ops, query APIs, and separated DB/OSS architecture feat(datasets): add DatasetMetadataClient for pure DB-backed metadata management Jul 1, 2026
@jake11-oho
jake11-oho force-pushed the feat/datasets-sdk-v2 branch 2 times, most recently from 230644f to d35e66f Compare July 8, 2026 08:41
… management

Introduce a pure DB-backed metadata client decoupled from the server
package. ORM models (Dataset, Instance, Image, Permission, AuditEvent)
are defined in rock/sdk/envhub/datasets/database.py with their own Base,
eliminating the import chain through rock.envhub that pulled in fastapi,
nacos, redis, and ray.

Key changes:
- DatasetMetadataClient as user-facing SDK entry point
- DbDatasetRegistry with connection pooling, engine dedup, batch upsert
- pool_pre_ping parameter, create_all dedup per db_url
- Split table with task_count, tags on Instance, timestamp fields
- Sorting support (default: org+name ASC) with SortField/SortOrder enums
- DatasetInfo.splits returns list[SplitInfo] with full metadata
- source field on Dataset for tracking data origin
- 63 unit tests covering all CRUD operations, sorting, and edge cases

fixes alibaba#1170

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jake11-oho
jake11-oho force-pushed the feat/datasets-sdk-v2 branch from 05e22fb to 41899d8 Compare July 10, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: enhance Datasets SDK with pagination, file operations, and DB-backed metadata

1 participant