feat(api): Allow tracing of requests and responses #7609
Conversation
✅ Deploy Preview for localai ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp
force-pushed
the
feat/tracing
branch
2 times, most recently
from
2faacc1 to
670ae28
Compare
| function tracesApp() { | ||
| return { | ||
| traces: [], | ||
| selectedTrace: null, |
There was a problem hiding this comment.
should we also have an "export" to save as a jsonfile or either have a dump of the whole memory?
There was a problem hiding this comment.
implemented as a JSON file. The body of the message is base64 encoded which could be inconvenient, but most of the time I would convert it to Parquet anyway.
core/http/middleware/trace.go
Outdated
| go func() { | ||
| for exchange := range logChan { | ||
| mu.Lock() | ||
| apiLogs = append(apiLogs, exchange) |
There was a problem hiding this comment.
I wonder if this would at least need some guard, e.g. a max limit that we default with, and keeps only the latest "X" fresh conversation. Just to avoid we don't keep growing in memory indefinetly
There was a problem hiding this comment.
Implemented in a circle buffer. The max number of items are configurable.
| UploadLimitMB, Threads, ContextSize int | ||
| F16 bool | ||
| Debug bool | ||
| EnableTracing bool |
There was a problem hiding this comment.
Should this be as well a runtime setting?
| @@ -0,0 +1,184 @@ | |||
| <!DOCTYPE html> | |||
There was a problem hiding this comment.
an open point: how the user reaches this page? should it be in the navbar?
There was a problem hiding this comment.
I'm not sure whether it should go at the top level of the navbar, or whether it should go in a new menu. I think it is most similar to the system page because they both deal with monitoring.
richiejp
force-pushed
the
feat/tracing
branch
3 times, most recently
from
47a9fa2 to
c428e40
Compare
Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp
force-pushed
the
feat/tracing
branch
from
c428e40 to
3b5e6ae
Compare
2 participants
Description
When creating a data set for fine-tuning or debugging an agent, it is useful to be able to see a log of the requests and responses that took place. LocalAI currently has a text log However, this contains a lot of information in addition to the data which these use cases require.
There are many different ways in which we could record requests made to LocalAI. Including a proxy server that wouldn't require any changes to LocalAI or the client application.
However, this is another component that users must install and configure. In addition, a separate component cannot be used as the basis for further local AI features, For example we could add a button that replays requests.
This initial pull request, however, is very simple. It just adds the ability to trace requests and responses, logging them in memory. They can then be fetched through an API endpoint. It's not enabled by default.
Notes for Reviewers
Signed commits