Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pulling main updates to rwkv-x-playground #89

Merged
merged 153 commits into from
Apr 10, 2024
Merged

pulling main updates to rwkv-x-playground #89

merged 153 commits into from
Apr 10, 2024

Commits on Jan 18, 2024

  1. WIP baseline run

    @pic-o
    pic-o committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    ff516a1 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. WIP torch compile test

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    a4ab031 View commit details
    Browse the repository at this point in the history

  2. Tweak torch compile to show a warning

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2a015d0 View commit details
    Browse the repository at this point in the history

  3. wip runs

    @pic-o
    pic-o committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    a8585e1 View commit details
    Browse the repository at this point in the history

  4. wip train, exp sloss train

    @pic-o
    pic-o committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5c60ab9 View commit details
    Browse the repository at this point in the history

  5. fixing the prefix masking setting

    @pic-o
    pic-o committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    ee976dd View commit details
    Browse the repository at this point in the history

  6. docker build for updated cuda version

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    783f36e View commit details
    Browse the repository at this point in the history

  7. Merge branch 'rwkv-x-selective-loss-exp' of https://github.com/RWKV/R… 

    …WKV-infctx-trainer into rwkv-x-selective-loss-exp
    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    1a772b1 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #59 from RWKV/rwkv-x-selective-loss-exp 

    Rwkv x selective loss exp
    @PicoCreator
    PicoCreator authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5288410 View commit details
    Browse the repository at this point in the history

  9. tweaking build rules

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    b1ab77f View commit details
    Browse the repository at this point in the history

  10. Merge pull request #60 from RWKV/rwkv-x-selective-loss-exp 

    tweaking build rules
    @PicoCreator
    PicoCreator authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    ccc2cb9 View commit details
    Browse the repository at this point in the history

  11. building the cuda env

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    a6eb626 View commit details
    Browse the repository at this point in the history

  12. test building on x series

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    64cf80f View commit details
    Browse the repository at this point in the history

  13. trying to trigger GH actions

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    8dd3eac View commit details
    Browse the repository at this point in the history

  14. Merge pull request #61 from RWKV/rwkv-x-selective-loss-exp 

    Rwkv x selective loss exp
    @PicoCreator
    PicoCreator authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    6aae950 View commit details
    Browse the repository at this point in the history

  15. trying to trigger the GH docker action

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4546994 View commit details
    Browse the repository at this point in the history

  16. still trying to fix the build

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    42cf1af View commit details
    Browse the repository at this point in the history

  17. 7B baseline

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f088536 View commit details
    Browse the repository at this point in the history

  18. wip 7b baseline

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    bbcb751 View commit details
    Browse the repository at this point in the history

  19. sloss perf

    @PicoCreator
    PicoCreator committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    c00a27e View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2024

  1. 7B baseline

    @PicoCreator
    PicoCreator committed Jan 20, 2024
    Configuration menu
    Copy the full SHA
    9426405 View commit details
    Browse the repository at this point in the history

  2. Fixed DS 3?

    @pic-o
    pic-o committed Jan 20, 2024
    Configuration menu
    Copy the full SHA
    38c30f6 View commit details
    Browse the repository at this point in the history

  3. deepspeed and multi gpu validation

    @pic-o
    pic-o committed Jan 20, 2024
    Configuration menu
    Copy the full SHA
    e0aad53 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #63 from RWKV/ds3-fix 

    Ds3 fix
    @PicoCreator
    PicoCreator authored Jan 20, 2024
    Configuration menu
    Copy the full SHA
    3186764 View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2024

  1. 7B benchmarking

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    1483339 View commit details
    Browse the repository at this point in the history

  2. benchmark file tweak

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    c251e14 View commit details
    Browse the repository at this point in the history

  3. tweak

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    01835f4 View commit details
    Browse the repository at this point in the history

  4. WIP 1B5 baseline

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    b237fb3 View commit details
    Browse the repository at this point in the history

  5. WIP 1B5 baseline and sloss runs

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    9438607 View commit details
    Browse the repository at this point in the history

  6. WIP experiments

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    86b3aea View commit details
    Browse the repository at this point in the history

  7. Fixing validation loss code

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    a3a8f00 View commit details
    Browse the repository at this point in the history

  8. 1B5 - enwiki / sloss runs

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    3516b74 View commit details
    Browse the repository at this point in the history

  9. 100k run

    @pic-o
    pic-o committed Jan 21, 2024
    Configuration menu
    Copy the full SHA
    5d0bcd8 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. fixing spikes in token/s tracking

    @pic-o
    pic-o committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    463b5ee View commit details
    Browse the repository at this point in the history

  2. reverted how KTokens are measure (the newer graph is wierder)

    @pic-o
    pic-o committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    5e9802f View commit details
    Browse the repository at this point in the history

  3. WIP 1B5 run

    @pic-o
    pic-o committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    695e4c2 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. wip iteration

    @pic-o
    pic-o committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    325c53a View commit details
    Browse the repository at this point in the history

  2. WIP returne

    @pic-o
    pic-o committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    ebab809 View commit details
    Browse the repository at this point in the history

  3. 1B5 memory finetune

    @PicoCreator
    PicoCreator committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    54c816a View commit details
    Browse the repository at this point in the history

  4. 1.5b and 3b runs

    @PicoCreator
    PicoCreator committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    dc99b9b View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2024

  1. 3B & 7B runs

    @PicoCreator
    PicoCreator committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    d57efd2 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #65 from RWKV/rwkv-x-selective-loss-exp 

    Rwkv x selective loss exp
    @PicoCreator
    PicoCreator authored Jan 24, 2024
    Configuration menu
    Copy the full SHA
    66e37ad View commit details
    Browse the repository at this point in the history

  3. reverting lr calc

    @pic-o
    pic-o committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    20d0411 View commit details
    Browse the repository at this point in the history

  4. example of S3 based dataset

    @pic-o
    pic-o committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    767f46e View commit details
    Browse the repository at this point in the history

  5. updating config example

    @pic-o
    pic-o committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    425e8db View commit details
    Browse the repository at this point in the history

  6. tweak example yaml

    @pic-o
    pic-o committed Jan 24, 2024
    Configuration menu
    Copy the full SHA
    47cc775 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. change default tokenizer to world

    @pic-o
    pic-o committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    a1b1046 View commit details
    Browse the repository at this point in the history

  2. Bug fixes, and feature patches from @SMERKY

    @pic-o
    pic-o committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    6e89d63 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #66 from RWKV/remote-dataset-support 

    Remote dataset support
    @PicoCreator
    PicoCreator authored Jan 25, 2024
    Configuration menu
    Copy the full SHA
    2db0b8d View commit details
    Browse the repository at this point in the history

  4. fixing total token counting, maybe?

    @pic-o
    pic-o committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    0bea4d9 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2024

  1. WIP datapack implementaiton

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    f976b7e View commit details
    Browse the repository at this point in the history

  2. WIP datapack implementation

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    5605acc View commit details
    Browse the repository at this point in the history

  3. Working datapack builder

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    41710dc View commit details
    Browse the repository at this point in the history

  4. Working datapack packing

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    53cf968 View commit details
    Browse the repository at this point in the history

  5. Format tweaking

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    2164db0 View commit details
    Browse the repository at this point in the history

  6. run tweak

    @pic-o
    pic-o committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    2ad2ef7 View commit details
    Browse the repository at this point in the history

  7. Merge pull request #68 from RWKV/rwkv-x-interweave-datapack 

    datapacking support
    @PicoCreator
    PicoCreator authored Jan 28, 2024
    Configuration menu
    Copy the full SHA
    6c35546 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #69 from RWKV/rwkv-x-interweave-datapack 

    run tweak
    @PicoCreator
    PicoCreator authored Jan 28, 2024
    Configuration menu
    Copy the full SHA
    84fb00a View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Updating exampels to v5?

    @pic-o
    pic-o committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    4ee43f2 View commit details
    Browse the repository at this point in the history

  2. bug fixes

    @pic-o
    pic-o committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    fa2c3eb View commit details
    Browse the repository at this point in the history

  3. WIP Eagle Finetune Notebook

    @pic-o
    pic-o committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    db37891 View commit details
    Browse the repository at this point in the history

  4. wip tune examples

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    0530723 View commit details
    Browse the repository at this point in the history

  5. chat format fix

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    815492b View commit details
    Browse the repository at this point in the history

  6. @SmerkyG optimized timemix

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    f6f23e5 View commit details
    Browse the repository at this point in the history

  7. Automatically disable CUDA, if its not detected

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    4deb31a View commit details
    Browse the repository at this point in the history

  8. Fixing the batching dummy passes

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    dd8998e View commit details
    Browse the repository at this point in the history

  9. fixing dummay_batch_zero

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    75f2a4c View commit details
    Browse the repository at this point in the history

  10. precision fixes 

    Signed-off-by: Daniel Goldstein <Dan@DESKTOP-UQ7GDK9>
    @SmerkyG
    SmerkyG committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    52dbb5f View commit details
    Browse the repository at this point in the history

  11. Adding no cuda auto toggle, adding torch compile (again)

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    36a4cd4 View commit details
    Browse the repository at this point in the history

  12. Empty shape skip code

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    fce7dbc View commit details
    Browse the repository at this point in the history

  13. capybara chat example

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    97c8df9 View commit details
    Browse the repository at this point in the history

  14. capybara

    @PicoCreator
    PicoCreator committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    ded473e View commit details
    Browse the repository at this point in the history

  15. simplified padding 

    Signed-off-by: Dan <Dan@DESKTOP-UQ7GDK9>
    @SmerkyG
    SmerkyG committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    fd0342f View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. Merge pull request #71 from RWKV/rwkv-x-interweave-datapack 

    Rwkv x interweave datapack
    @PicoCreator
    PicoCreator authored Feb 1, 2024
    Configuration menu
    Copy the full SHA
    d4fa285 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. various finetune examples

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    fe440bb View commit details
    Browse the repository at this point in the history

  2. multipack example tweak

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    a9a5912 View commit details
    Browse the repository at this point in the history

  3. WIP Eagle-x

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    c5b98f3 View commit details
    Browse the repository at this point in the history

  4. use chunklen128 with float64 and tighter decay clamp for JIT training… 

    … speed
    @SmerkyG
    SmerkyG committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    06e4280 View commit details
    Browse the repository at this point in the history

  5. Revert "use chunklen128 with float64 and tighter decay clamp for JIT … 

    …training speed"

This reverts commit 06e4280.
    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    7444e63 View commit details
    Browse the repository at this point in the history

  6. multipack

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    90059d3 View commit details
    Browse the repository at this point in the history

  7. Enofrcing NO_CUDA by default, and disabling torch compile by default

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    2c4fc86 View commit details
    Browse the repository at this point in the history

  8. tweaking defaults

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    6d52048 View commit details
    Browse the repository at this point in the history

  9. Merge pull request #72 from RWKV/rwkv-x-eagle-notebooks 

    Rwkv x eagle notebooks
    @PicoCreator
    PicoCreator authored Feb 2, 2024
    Configuration menu
    Copy the full SHA
    a7b090d View commit details
    Browse the repository at this point in the history

  10. typo in checkpoint size

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    7f7f61f View commit details
    Browse the repository at this point in the history

  11. updated examples

    @PicoCreator
    PicoCreator committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    786889d View commit details
    Browse the repository at this point in the history

  12. Merge pull request #73 from RWKV/rwkv-x-eagle-notebooks 

    Rwkv x eagle notebooks
    @PicoCreator
    PicoCreator authored Feb 2, 2024
    Configuration menu
    Copy the full SHA
    00274ed View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2024

  1. cuda 12-1 build

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    021424f View commit details
    Browse the repository at this point in the history

  2. updated github runner template

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    24cd174 View commit details
    Browse the repository at this point in the history

  3. Fixing cosign, for docker img

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    08d5296 View commit details
    Browse the repository at this point in the history

  4. bumping docker build process

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    aca143b View commit details
    Browse the repository at this point in the history

  5. base model tweak

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    34b3616 View commit details
    Browse the repository at this point in the history

  6. docker build bump

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    2e33f4b View commit details
    Browse the repository at this point in the history

  7. fixing docker ref

    @PicoCreator
    PicoCreator authored Feb 3, 2024
    Configuration menu
    Copy the full SHA
    d606d4a View commit details
    Browse the repository at this point in the history

  8. change logging format, to reduce confusion

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    4ddbf83 View commit details
    Browse the repository at this point in the history

  9. move to experiment folder

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    b3a32c0 View commit details
    Browse the repository at this point in the history

  10. preparing experiment notebooks

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    695545d View commit details
    Browse the repository at this point in the history

  11. Added notes on runnign the runner

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    d4e2d3e View commit details
    Browse the repository at this point in the history

  12. Merge remote-tracking branch 'origin/main' into rwkv-x-eagle-notebooks

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    fea8459 View commit details
    Browse the repository at this point in the history

  13. include any option

    @pic-o
    pic-o committed Feb 3, 2024
    Configuration menu
    Copy the full SHA
    d9a6211 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2024

  1. wip benchmarks

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    3d5fc0f View commit details
    Browse the repository at this point in the history

  2. tweak

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    9c7744c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    59dc19e View commit details
    Browse the repository at this point in the history

  4. Fixing the LR for batched

    @pic-o
    pic-o committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    c42c7c0 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @pic-o
    pic-o committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    fe9bd2c View commit details
    Browse the repository at this point in the history

  6. fixing mask sum calc

    @pic-o
    pic-o committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    475558d View commit details
    Browse the repository at this point in the history

  7. wip calibration

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    0fd1387 View commit details
    Browse the repository at this point in the history

  8. enwiki 16k test

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    a2443d7 View commit details
    Browse the repository at this point in the history

  9. Update notebook title in enwiki-16k-3e-5.ipynb

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    85ce0c1 View commit details
    Browse the repository at this point in the history

  10. Fixing multipack

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    18f52d0 View commit details
    Browse the repository at this point in the history

  11. loss validation run

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    ade9b35 View commit details
    Browse the repository at this point in the history

  12. WIP datapack fixing code

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    efced7f View commit details
    Browse the repository at this point in the history

  13. Update learning rate initialization and finalization values

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    e5deb69 View commit details
    Browse the repository at this point in the history

  14. WIP benchmarks

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    5e1b715 View commit details
    Browse the repository at this point in the history

  15. drop dataset_name and dataset_index

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    6d7ca1c View commit details
    Browse the repository at this point in the history

  16. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    dd9c79d View commit details
    Browse the repository at this point in the history

  17. the datapack code

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    0ee162d View commit details
    Browse the repository at this point in the history

  18. support custom dataset split

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    44fb0be View commit details
    Browse the repository at this point in the history

  19. config update

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    6f1e02e View commit details
    Browse the repository at this point in the history

  20. WIP tweaks

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    601986c View commit details
    Browse the repository at this point in the history

  21. wip MultiPack train

    @PicoCreator
    PicoCreator committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    dfccbc7 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. tweaks

    @PicoCreator
    PicoCreator committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    74deedb View commit details
    Browse the repository at this point in the history

  2. prototype train/test split swap - because sometimes you need that

    @pic-o
    pic-o committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    429c8c1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @pic-o
    pic-o committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    0654172 View commit details
    Browse the repository at this point in the history

  4. fixing multi-gpu sync

    @pic-o
    pic-o committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    e02907a View commit details
    Browse the repository at this point in the history

  5. Merge pull request #75 from RWKV/rwkv-x-eagle-notebooks 

    Rwkv x eagle notebooks
    @PicoCreator
    PicoCreator authored Feb 6, 2024
    Configuration menu
    Copy the full SHA
    6b04127 View commit details
    Browse the repository at this point in the history

  6. enforce DropLast for the distributed sampler to work around set issues

    @PicoCreator
    PicoCreator committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    bea9014 View commit details
    Browse the repository at this point in the history

  7. Merge pull request #76 from RWKV/rwkv-x-eagle-notebooks 

    enforce DropLast for the distributed sampler to work around set issues
    @PicoCreator
    PicoCreator authored Feb 6, 2024
    Configuration menu
    Copy the full SHA
    d9f8b8d View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Add notes to README.md and update config-example.yaml

    @PicoCreator
    PicoCreator committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    727e080 View commit details
    Browse the repository at this point in the history

  2. Added nvidia-smi safety within the container, stops and restart on mi… 

    …ssing nvidia-smi
    @pic-o
    pic-o committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    5ffe8cb View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @pic-o
    pic-o committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    6e792a2 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Merge pull request #77 from RWKV/rwkv-x-eagle-notebooks 

    Rwkv x eagle notebooks
    @PicoCreator
    PicoCreator authored Feb 8, 2024
    Configuration menu
    Copy the full SHA
    ba60880 View commit details
    Browse the repository at this point in the history

  2. fixing finetune recommendations

    @pic-o
    pic-o committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    7a92b7b View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. WIP dataset behaviour tweak

    @PicoCreator
    PicoCreator committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    b9f1f1a View commit details
    Browse the repository at this point in the history

  2. enable no cuda by default

    @PicoCreator
    PicoCreator committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    fe5f734 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV… 

    …-infctx-trainer into rwkv-x-eagle-notebooks
    @PicoCreator
    PicoCreator committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    539e8ed View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2024

  1. T2 chunk 1 build process

    @PicoCreator
    PicoCreator committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    143ff31 View commit details
    Browse the repository at this point in the history

  2. Multipack instruct tweak

    @PicoCreator
    PicoCreator committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    976e038 View commit details
    Browse the repository at this point in the history

  3. wip datastreaming test

    @PicoCreator
    PicoCreator committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    efa7737 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2024

  1. notebook validation of the continue checkpoint

    @PicoCreator
    PicoCreator committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    6861e5e View commit details
    Browse the repository at this point in the history

  2. the crazy fix - hijack LR schedule code to fix dataset offset

    @PicoCreator
    PicoCreator committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    605c9a3 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #78 from RWKV/rwkv-x-eagle-notebooks 

    Rwkv x eagle notebooks
    @PicoCreator
    PicoCreator authored Feb 11, 2024
    Configuration menu
    Copy the full SHA
    da81f80 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. jit improvement, noncuda fixes

    @SmerkyG
    SmerkyG committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    73716ed View commit details
    Browse the repository at this point in the history

  2. Merge pull request #80 from RWKV/smerky_fixes_2024_02_15 

    jit improvement, noncuda fixes
    @PicoCreator
    PicoCreator authored Feb 15, 2024
    Configuration menu
    Copy the full SHA
    40c1fb3 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. initial rwkv6 and cuda state support for v5

    @SmerkyG
    SmerkyG committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    ba2f463 View commit details
    Browse the repository at this point in the history

  2. model init version support 

    bugfixes for non-cuda v6 timemix and v6 chanmix
    @SmerkyG
    SmerkyG committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    9c3ab87 View commit details
    Browse the repository at this point in the history

  3. v6 chanmix bugfix

    @SmerkyG
    SmerkyG committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    7d443f5 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. improved v6 assertion to allow non-cuda inference

    @SmerkyG
    SmerkyG committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    d42fa37 View commit details
    Browse the repository at this point in the history

  2. add older v6 cuda

    @SmerkyG
    SmerkyG committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    fcee20d View commit details
    Browse the repository at this point in the history

  3. split v6 into separate directory

    @SmerkyG
    SmerkyG committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    31fe6b6 View commit details
    Browse the repository at this point in the history

  4. removed v6 chanmix from v5 dir

    @SmerkyG
    SmerkyG committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    e672079 View commit details
    Browse the repository at this point in the history

  5. revert usage of v6state cuda for v5

    @SmerkyG
    SmerkyG committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    4d215ef View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. import bugfix, single step non-cuda bugfix

    @SmerkyG
    SmerkyG committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    2908b58 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Merge pull request #88 from RWKV/rwkv-6-support 

    RWKV 6 support
    @SmerkyG
    SmerkyG authored Apr 3, 2024
    Configuration menu
    Copy the full SHA
    2528086 View commit details
    Browse the repository at this point in the history