Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(evaluation): support multi datasets #1502

Merged

Conversation

goldenxinxing
Copy link
Contributor

@goldenxinxing goldenxinxing commented Nov 15, 2022

Description

  1. sever: Support multi datasets, the model's relation are as below:
    session -> consumer -> dataset
    1 : n : m

  2. client: In order to ensure the uniqueness of data under multiple datasets, add uri as prefix for data id/index
    image

  3. client: add multi dataset parm for the cmd "swcli eval run" and "swcli model eval"

Modules

  • UI
  • Controller
  • Agent
  • Client
  • Python-SDK
  • Others

Checklist

  • run code format and lint check
  • add unit test
  • add necessary doc

@codecov
Copy link

codecov bot commented Nov 15, 2022

Codecov Report

Merging #1502 (d18b0fa) into main (7b8cd24) will increase coverage by 0.00%.
The diff coverage is 93.69%.

@@            Coverage Diff            @@
##               main    #1502   +/-   ##
=========================================
  Coverage     82.08%   82.08%           
- Complexity     1830     1831    +1     
=========================================
  Files           327      327           
  Lines         15978    15998   +20     
  Branches        858      859    +1     
=========================================
+ Hits          13115    13132   +17     
- Misses         2521     2523    +2     
- Partials        342      343    +1     
Flag Coverage Δ
controller 76.02% <97.67%> (+0.02%) ⬆️
standalone 88.13% <80.00%> (-0.01%) ⬇️
unittests 88.13% <80.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...n/dataset/dataloader/mapper/DataReadLogMapper.java 90.00% <ø> (ø)
client/starwhale/api/_impl/model.py 90.66% <79.16%> (+0.19%) ⬆️
.../domain/dataset/dataloader/dao/DataReadLogDao.java 61.90% <80.00%> (ø)
...lops/domain/dataset/dataloader/dao/SessionDao.java 81.81% <80.00%> (+6.81%) ⬆️
client/starwhale/core/eval/executor.py 91.66% <100.00%> (ø)
...le/mlops/domain/dataset/dataloader/DataLoader.java 100.00% <100.00%> (ø)
...ops/domain/dataset/dataloader/DataReadManager.java 100.00% <100.00%> (ø)
...dataset/dataloader/converter/SessionConverter.java 100.00% <100.00%> (ø)
...starwhale/mlops/schedule/k8s/K8sTaskScheduler.java 78.12% <100.00%> (+0.52%) ⬆️
client/starwhale/api/_impl/dataset/loader.py 98.01% <0.00%> (-1.00%) ⬇️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@goldenxinxing goldenxinxing force-pushed the feat/support_multi_dataset branch 3 times, most recently from dd788ea to 3443b0b Compare November 15, 2022 06:18
@goldenxinxing goldenxinxing self-assigned this Nov 15, 2022
@goldenxinxing goldenxinxing added the feature ✨ new feature label Nov 15, 2022
@goldenxinxing goldenxinxing force-pushed the feat/support_multi_dataset branch 6 times, most recently from cabf7af to 5e3cf17 Compare November 16, 2022 05:23
@goldenxinxing goldenxinxing marked this pull request as ready for review November 16, 2022 05:43
@tianweidut tianweidut added this to the v0.3.2 milestone Nov 17, 2022
jialeicui
jialeicui previously approved these changes Nov 17, 2022
Copy link
Contributor

@jialeicui jialeicui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tianweidut
Copy link
Member

Others LGTM

@tianweidut tianweidut changed the title feat(evaluation): support multi dataset feat(evaluation): support multi datasets Nov 17, 2022
@goldenxinxing goldenxinxing deleted the feat/support_multi_dataset branch April 27, 2023 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature ✨ new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants