Skip to content

add interactive mode #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 10, 2025
Merged

add interactive mode #11

merged 4 commits into from
May 10, 2025

Conversation

andrewkchan
Copy link
Owner

Adds an "interactive mode" that loads the model weights and then continuously prompts the user for new commands. Added because loading deepseek weights is truly horrendously slow and sometimes I want to test multiple things at a time.

Use it like so:

# ./main <checkpoint dir> -m i -L
# Load the checkpoint dir in interactive mode with -L to lock the weights in memory
./main v2-lite-f16 -m i

Then at the prompt, input commands similar to how you would input commands in the CLI, except without the checkpoint dir and omitting -m:

> c -i "Q: What is the meaning of life?" -n 16 -t 0.5
# Equivalent to ./main v2-lite-f16 -m c -L -i "Q: What is the meaning of life?" -n 16 -t 0.5

@andrewkchan andrewkchan merged commit 200fbab into main May 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant