Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement stochastic speculative sampling #5625

Merged
merged 17 commits into from
Mar 4, 2024
Merged

Implement stochastic speculative sampling #5625

merged 17 commits into from
Mar 4, 2024
+256 −57

Commits on Feb 21, 2024

  1. (WIP) Implement stochastic speculative decoding

    @mscheong01
    mscheong01 committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    c1bad4a View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. sample from residual distribution on draft accept failure

    @mscheong01
    mscheong01 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    a9335a5 View commit details
    Browse the repository at this point in the history

  2. fix ggerganov#5657: force greedy sampling with probs when temp is 0

    @mscheong01
    mscheong01 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    4694edd View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. remove p_accept parameter

    @mscheong01
    mscheong01 committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    fb18827 View commit details
    Browse the repository at this point in the history

  2. fix style

    @mscheong01
    mscheong01 committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    34b942a View commit details
    Browse the repository at this point in the history

  3. remove unused variables

    @mscheong01
    mscheong01 committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    875319b View commit details
    Browse the repository at this point in the history

  4. add srand() in speculative.cpp

    @mscheong01
    mscheong01 committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    6afc1f6 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. replace use of rand() with mt19937 sampling

    @mscheong01
    mscheong01 committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    94f6256 View commit details
    Browse the repository at this point in the history

  2. fixes based on review (@JohannesGaessler)

    @mscheong01
    mscheong01 committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    e4896e7 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. fix r random generation

    @mscheong01
    mscheong01 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    6b35c8b View commit details
    Browse the repository at this point in the history

  2. randomly select next sequence to verify + fix bug in memory freeing

    @mscheong01
    mscheong01 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2ad3f7c View commit details
    Browse the repository at this point in the history

  3. fix bug in active_seqs sync

    @mscheong01
    mscheong01 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    c2cd292 View commit details
    Browse the repository at this point in the history

  4. fix uniform int distribution initialization

    @mscheong01
    mscheong01 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    7463569 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2024

  1. remove warnings from comparison between int and size_t

    @mscheong01
    mscheong01 committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    c761354 View commit details
    Browse the repository at this point in the history

  2. check grammar in llama_sample_probability_distribution_impl

    @mscheong01
    mscheong01 committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    45465b2 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. remove malloc code by utilizing vectors

    @mscheong01
    mscheong01 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    67ad517 View commit details
    Browse the repository at this point in the history

  2. add PR link to README

    @mscheong01
    mscheong01 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    056bdb3 View commit details
    Browse the repository at this point in the history