Large prompts crashing with no error output

[trevtravtrev]

Anytime I input a prompt more than a couple of lines, alpaca crashes with no error message. I'm running this on a beefy computer (32GB RAM, 12th Gen Intel(R) Core(TM) i7-12700KF 3.61 GHz CPU).

What I've tried:

• standard run with no parameters
• upping threads parameter to 10
• upping num tokens parameter to 1600
• 7B, 13B, 30B weights

Any idea why its crashing, or if different runtime variables can be set to fix this?

[vittogialli]

Don't write multiple lines normally, instead, write one line, and before reaching the end put a \ , then press enter and continue on the next line with the prompt

[trevtravtrev]

Don't write multiple lines normally, instead, write one line, and before reaching the end put a \ , then press enter and continue on the next line with the prompt

Ahh maybe this links back to #111 then. When I say "a couple of lines" in my post above, I'm not referring to newlines. I'm referring to a string maybe 50-100 words long with no newlines and no \ that is causing it to crash with no error. Any prompts more than about 20 words from my experience.

[trevtravtrev]

I haven't tried file input yet and don't have access to a computer to test currently. Has anyone had any luck with large prompts via file input without crashes? Do you need to increase the tokens or threads (or other) parameters when running?

[4eJIoBek1]

this fixes the problem, thanks to aleksusklim

                char buf[16384] = {0};
                int n_read;
                if(params.use_color) printf(ANSI_BOLD ANSI_COLOR_GREEN);
                if (scanf("%16383[^\n]%n%*c", buf, &n_read) <= 0) {

[trevtravtrev]

Where does that go

[4eJIoBek1]

In chat.cpp, line 1044

[gcweeks]

That just extends the character limit of the line (a welcome addition) but doesn't actually address the crash, which is caused by the else clause below it, which assumes the end of the line hasn't been reached:

} else { // n_read == 255
    another_line = false;
    buf[n_read] = '\n'; // buf[255]
    buf[n_read+1] = 0; // buf[256] !!
}

A check for that would do the trick:

if (n_read > 0 && buf[n_read-1]=='\\') {
    another_line = true;
} else {
    another_line = false;
    if (n_read == sizeof(buf)) { // end of line
        --n_read; // make room for newline
    }
}
buf[n_read - 1] = '\n';
buf[n_read] = 0;

Though you would still have the limitation that scanf would miss the trailing \ due to clipping for any line longer than the limit (whether 255 or 16383).