Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(job): custom job scheduler #880

Merged
merged 119 commits into from
Aug 18, 2022
Merged

feat(job): custom job scheduler #880

merged 119 commits into from
Aug 18, 2022

Commits on Aug 2, 2022

  1. draft new job scheduler

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    c72ce28 View commit details
    Browse the repository at this point in the history

  2. tune example for mnist

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    adada06 View commit details
    Browse the repository at this point in the history

  3. update mnist model file

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    0832876 View commit details
    Browse the repository at this point in the history

  4. refactor(eval):job dag

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    5d3d41e View commit details
    Browse the repository at this point in the history

  5. fix yaml dump problem

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    c2f7653 View commit details
    Browse the repository at this point in the history

  6. run eval with existing version; single task execute

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    7522875 View commit details
    Browse the repository at this point in the history

  7. dataset uri sent by param

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    53fa451 View commit details
    Browse the repository at this point in the history

  8. optimise entrypoint

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    58c9402 View commit details
    Browse the repository at this point in the history

  9. code format

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    d8d9c3a View commit details
    Browse the repository at this point in the history

  10. feat:add default handler for evaluation

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    9849301 View commit details
    Browse the repository at this point in the history

  11. update ignore

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    e7c2408 View commit details
    Browse the repository at this point in the history

  12. add default handler for eval

    gaoxinxing committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    d1db3fd View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2022

  1. change job values to list

    gaoxinxing committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    78a51e4 View commit details
    Browse the repository at this point in the history

  2. job parser for controller

    gaoxinxing committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    525ad32 View commit details
    Browse the repository at this point in the history

  3. job parser for controller

    gaoxinxing committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    594439e View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2022

  1. controller:extract job content during upload;tune task entity(doing)

    gaoxinxing committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    639ce3f View commit details
    Browse the repository at this point in the history

  2. change eval handler to model

    gaoxinxing committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    f867dd7 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2022

  1. change eval handler to model

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    23a2f76 View commit details
    Browse the repository at this point in the history

  2. default pipeline use datastore(doing)

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    c42f496 View commit details
    Browse the repository at this point in the history

  3. draft new job scheduler

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    236ffb9 View commit details
    Browse the repository at this point in the history

  4. tune example for mnist

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    40fec7c View commit details
    Browse the repository at this point in the history

  5. model change

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    8c5c5d5 View commit details
    Browse the repository at this point in the history

  6. refactor(eval):job dag

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    0620c74 View commit details
    Browse the repository at this point in the history

  7. fix yaml dump problem

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    47beade View commit details
    Browse the repository at this point in the history

  8. run eval with existing version; single task execute

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    18cc832 View commit details
    Browse the repository at this point in the history

  9. dataset uri sent by param

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    0ed6e20 View commit details
    Browse the repository at this point in the history

  10. optimise entrypoint

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    011c583 View commit details
    Browse the repository at this point in the history

  11. code format

    @goldenxinxing
    goldenxinxing authored and gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    d626837 View commit details
    Browse the repository at this point in the history

  12. feat:add default handler for evaluation

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    22ddec6 View commit details
    Browse the repository at this point in the history

  13. update ignore

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    9f49100 View commit details
    Browse the repository at this point in the history

  14. add default handler for eval

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    240a3c7 View commit details
    Browse the repository at this point in the history

  15. change job values to list

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    743724e View commit details
    Browse the repository at this point in the history

  16. job parser for controller

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    42f927a View commit details
    Browse the repository at this point in the history

  17. job parser for controller

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    556e5b3 View commit details
    Browse the repository at this point in the history

  18. controller:extract job content during upload;tune task entity(doing)

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    b328f50 View commit details
    Browse the repository at this point in the history

  19. change eval handler to model

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    ba8e0c9 View commit details
    Browse the repository at this point in the history

  20. change eval handler to model

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    bcd4fb4 View commit details
    Browse the repository at this point in the history

  21. default pipeline use datastore(doing)

    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    aa9b3c7 View commit details
    Browse the repository at this point in the history

  22. Merge remote-tracking branch 'gxx-fork/feat/job_3' into feat/job_3 

    # Conflicts:
#	docker/Makefile
    gaoxinxing committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    fe2f362 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 8, 2022
    Configuration menu
    Copy the full SHA
    c508398 View commit details
    Browse the repository at this point in the history

  2. default pipeline support datastore

    gaoxinxing committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    862725c View commit details
    Browse the repository at this point in the history

  3. fix:datastore dir bug; add manual dump method

    gaoxinxing committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    271f9f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. fix:datastore scan can't found results

    gaoxinxing committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    e56a687 View commit details
    Browse the repository at this point in the history

  2. feat:default ppl handler support datastore

    gaoxinxing committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    f214d8f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 9, 2022
    Configuration menu
    Copy the full SHA
    1d4ac2d View commit details
    Browse the repository at this point in the history

  4. code format

    gaoxinxing committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    cf562b2 View commit details
    Browse the repository at this point in the history

  5. optimise code style

    gaoxinxing committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    8d50d78 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. add multi processing communicate:pipe

    gaoxinxing committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    7598aeb View commit details
    Browse the repository at this point in the history

  2. code format

    gaoxinxing committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    e2c1159 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2022

  1. multiprocessing use duplex pipe communication

    gaoxinxing committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    f5dfcdc View commit details
    Browse the repository at this point in the history

  2. default eval step use simple way to coding

    gaoxinxing committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    ca82147 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2022

  1. tune default ppl handler impl

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    f4b3ce3 View commit details
    Browse the repository at this point in the history

  2. skip eval unittest

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    e9c39ca View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 12, 2022
    Configuration menu
    Copy the full SHA
    4a359b0 View commit details
    Browse the repository at this point in the history

  4. fix controller var name bug

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    14b4a1c View commit details
    Browse the repository at this point in the history

  5. controller unit test ignore

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    0e33165 View commit details
    Browse the repository at this point in the history

  6. conflict resolve

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    154e129 View commit details
    Browse the repository at this point in the history

  7. fix mypy check error

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    e8d90b6 View commit details
    Browse the repository at this point in the history

  8. fix code check style problem

    gaoxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    6009ffa View commit details
    Browse the repository at this point in the history

  9. skip unit test

    @goldenxinxing
    goldenxinxing committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    36a62d3 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2022

  1. unit test code fix

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    a850a21 View commit details
    Browse the repository at this point in the history

  2. code format

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    1e5ebdd View commit details
    Browse the repository at this point in the history

  3. rollback mnist model file

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    3cf3126 View commit details
    Browse the repository at this point in the history

  4. datastore key support blank

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    d1027f7 View commit details
    Browse the repository at this point in the history

  5. rollback metric key

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    8710e67 View commit details
    Browse the repository at this point in the history

  6. change comment

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    ef3a4ca View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 15, 2022
    Configuration menu
    Copy the full SHA
    b110d6e View commit details
    Browse the repository at this point in the history

  8. remove some comment

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    78df99a View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 15, 2022
    Configuration menu
    Copy the full SHA
    2f80fd9 View commit details
    Browse the repository at this point in the history

  10. update e2e test script

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    085fcb8 View commit details
    Browse the repository at this point in the history

  11. update e2e test script

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    f4f2894 View commit details
    Browse the repository at this point in the history

  12. update comment

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    febd85b View commit details
    Browse the repository at this point in the history

  13. update some usage advice

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    3a9dfae View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 15, 2022
    Configuration menu
    Copy the full SHA
    35dd002 View commit details
    Browse the repository at this point in the history

  15. skip test for ppl

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    33e0d62 View commit details
    Browse the repository at this point in the history

  16. fix code check problem

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    b3f7496 View commit details
    Browse the repository at this point in the history

  17. fix code check problem

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    47fb644 View commit details
    Browse the repository at this point in the history

  18. fix code check problem

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    9c84f9e View commit details
    Browse the repository at this point in the history

  19. fix code check problem

    gaoxinxing committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    59ad081 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2022

  1. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 16, 2022
    Configuration menu
    Copy the full SHA
    6f79405 View commit details
    Browse the repository at this point in the history

  2. fix code check problem

    gaoxinxing committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    49a7b4c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 16, 2022
    Configuration menu
    Copy the full SHA
    d447d56 View commit details
    Browse the repository at this point in the history

  4. add logger for datastore

    gaoxinxing committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    ae4149c View commit details
    Browse the repository at this point in the history

  5. fit new datastore scan api

    gaoxinxing committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    3ee1cc4 View commit details
    Browse the repository at this point in the history

  6. support new dataset

    gaoxinxing committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    6d4f9be View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2022

  1. tune schedule way

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    1e10c73 View commit details
    Browse the repository at this point in the history

  2. fix:datastore dump problem at multiprocessing pool

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    5a4759f View commit details
    Browse the repository at this point in the history

  3. standalone debug completed

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    68a3f62 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 17, 2022
    Configuration menu
    Copy the full SHA
    52c6cbc View commit details
    Browse the repository at this point in the history

  5. add ppl unit test

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    34606ec View commit details
    Browse the repository at this point in the history

  6. fix bug for get metrics api

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    4ab200c View commit details
    Browse the repository at this point in the history

  7. add ppl unit test

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    b6ad82d View commit details
    Browse the repository at this point in the history

  8. use multithread

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    154e13d View commit details
    Browse the repository at this point in the history

  9. fix unit test bug

    gaoxinxing committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    331bc7e View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2022

  1. fix unit test bug

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    6016193 View commit details
    Browse the repository at this point in the history

  2. code format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    617b87f View commit details
    Browse the repository at this point in the history

  3. skip test

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    9587ccc View commit details
    Browse the repository at this point in the history

  4. set concurrency for pool

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    0048e2c View commit details
    Browse the repository at this point in the history

  5. fix share object bug

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    d253ad5 View commit details
    Browse the repository at this point in the history

  6. use Protocol for Callback

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    cd780ec View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 18, 2022
    Configuration menu
    Copy the full SHA
    98fe7f2 View commit details
    Browse the repository at this point in the history

  8. code format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    7f9ba31 View commit details
    Browse the repository at this point in the history

  9. @step use consts var

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    9cf6ad6 View commit details
    Browse the repository at this point in the history

  10. code format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    8e8571a View commit details
    Browse the repository at this point in the history

  11. step dependency -> needs

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    517d94b View commit details
    Browse the repository at this point in the history

  12. remove unused init method

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    762b376 View commit details
    Browse the repository at this point in the history

  13. extract common function to util package

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    00df15e View commit details
    Browse the repository at this point in the history

  14. rename for entrypoint

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    0a66293 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 18, 2022
    Configuration menu
    Copy the full SHA
    7aecd17 View commit details
    Browse the repository at this point in the history

  16. simply check

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    1dba50c View commit details
    Browse the repository at this point in the history

  17. use util's function for dump

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    200c6f7 View commit details
    Browse the repository at this point in the history

  18. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 18, 2022
    Configuration menu
    Copy the full SHA
    0029a33 View commit details
    Browse the repository at this point in the history

  19. update requirement

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    623ff84 View commit details
    Browse the repository at this point in the history

  20. code format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    4bf606a View commit details
    Browse the repository at this point in the history

  21. Merge branch 'main' into feat/job_3

    @goldenxinxing
    goldenxinxing authored Aug 18, 2022
    Configuration menu
    Copy the full SHA
    f42beed View commit details
    Browse the repository at this point in the history

  22. job yaml optimise

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    7ae9fab View commit details
    Browse the repository at this point in the history

  23. log format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    ad19dca View commit details
    Browse the repository at this point in the history

  24. optimise list

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    77fd490 View commit details
    Browse the repository at this point in the history

  25. code format

    gaoxinxing committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    b990ba5 View commit details
    Browse the repository at this point in the history