Ability for `./main` to keep the model in memory and pass it more text

[simonw]

The ./main program currently outputs text and then quits.

How hard would it be to add a mode where it could stay running and be ready to accept more text piped to standard input?

This could help avoid the overhead of loading the model again every time the script runs.

Maybe it could output the generated text followed by a marker of some sort when it's done, so a wrapping process could see when it's finished and available to send a new prompt for evaluation.

I'm interested in wrapping it in a tiny Python web server to give myself a UI for interacting with the model.

[j-f1]

I made a fork (https://github.com/j-f1/forked-llama.cpp/tree/swift) that’s focused around working as a library rather than a standalone program. I don’t know how hard it would be to bridge that to Python but you might find some of the changes useful for writing a C++ command line program you can talk to via the command line.

[G2G2G2G]

agreed, a chat mode would be a lot better. The prompts this model generates are very bizarre lmao

[ggerganov]

Modifying main.cpp to continuously get new prompts with a single load of the model should be easy. Just have to manage the n_past and embd vars properly.

Interfacing this with the outside world will take some more effort.

[bakkot]

Interfacing this with the outside world will take some more effort.

One thing I've done previously is to drop in a single-file HTTP server, like this one, and then make an HTTP API. (Optionally also a single-file JSON parser/serializer, like this one, so that you can make the API JSON-based.)

It's a little silly, vs building as a library or adding python bindings or whatever, but it's cross-platform and very easy to get it working (~60 lines, or a little more if you want to stream the results rather than just sending them all when it's done).

example server

#include <cstdio>
#include "httplib.h"

using namespace httplib;

#define PORT 8080

std::string get_response(std::string message) {
  // call actual API here
  return "got message: " + message;
}

int main(void) {
  Server svr;

  if (!svr.is_valid()) {
    printf("server setup failed\n");
    return -1;
  }

  svr.Get("/", [=](const Request & /*req*/, Response &res) {
    res.set_content("POST api is listening on /api\n", "text/plain");
  });

  svr.Post("/api",
    [&](const Request &req, Response &res, const ContentReader &content_reader) {
      if (req.is_multipart_form_data()) {
        res.set_content("Server does not support multipart form data", "text/html");
        res.status = 500;
        return;
      }
      std::string body;
      content_reader([&](const char *data, size_t data_length) {
        body.append(data, data_length);
        return true;
      });
      // if it's JSON, change the content type to application/json
      res.set_content(get_response(body), "text/plain");
    });

  svr.set_exception_handler([](const Request& req, Response& res, std::exception_ptr ep) {
    auto fmt = "<h1>Error 500</h1><p>%s</p>";
    char buf[BUFSIZ];
    try {
      std::rethrow_exception(ep);
    } catch (std::exception &e) {
      snprintf(buf, sizeof(buf), fmt, e.what());
    } catch (...) {
      snprintf(buf, sizeof(buf), fmt, "Unknown Exception");
    }
    res.set_content(buf, "text/html");
    res.status = 500;
  });

  printf("starting server on port %d\n", PORT);
  svr.listen("localhost", PORT);

  return 0;
}

[turbo]

How hard would it be to add a mode where it could stay running and be ready to accept more text piped to standard input?

Would be awesome, because this would allow pre-prompting and spawning interactive sessions like tinygrads's LLaMa personalities (demo video).

[G2G2G2G]

well he doesn't want any deps so that's why the interfacing is the hard part, otherwise it's pretty ez part.
I think doing the

Modifying main.cpp to continuously get new prompts with a single load of the model should be easy. Just have to manage the n_past and embd vars properly.

will be most of the work, at least from there it's easy enough to ghetto hack in anyones own test bed to input stuff into stdin for the program

[DelusionalLogic]

Assuming nobody cares about windows it would be possible to allocate the giant buffer required for the model with shm_open to retain the loaded model in memory between executions. That way you could still faff about with the executable/parameters as long as they don't impact how you load the llama model.

[blackhole89]

I'm working on adding a sort of interactive mode over in a fork, where (if a flag, -i, is passed) you can either interrupt generation with SIGINT or specify a "reverse prompt" token string which results in control being passed back to the user.

It currently looks like this:

Would you be interested in a PR once I'm done with some further cleanup and testing? I'm still planning to put the colouring behind another flag, and find a solution for some papercuts (among others, it seems that spaces tend to be part of a single token with words that follow them, so you have to use User: rather than User: as a reverse prompt and force the user to enter the space following it).

[ggerganov]

@blackhole89
Yes! This is exactly the way to go

Edit: if you cannot make it run on Windows, you can #ifdef it and say it's supported only on Linux / Mac OS

Edit2: regarding spaces - the tokenizer is currently broken. Probably this is causing a lot of trouble such cases and also when Unicode characters are present in the input.

[blackhole89]

I unfortunately don't have access to a Windows machine to test it on right now. Is there a problem with the availability of signal.h/sigaction there? Either way, at least the "reverse prompt" triggered interaction should work even without the signal handler.

Unfortunate about the tokenizer. I guess I will leave the problem untouched for now, hoping for an eventual resolution :)

I think I got it to work to approximately the standard I was hoping for (few lingering issues: probably better to communicate limitations such as that the reverse prompt must be token-exact; subsequent user inputs are not counted towards the context size limit), so I'll go ahead and make a PR.