0001: Service Config #2
Conversation
|
|
||
| ### The Config Source | ||
|
|
||
| Glide config is represented as a YAML file (R1, R2, R3). |
There was a problem hiding this comment.
I agree. Should be YAML.
| - via Glide CLI params & args | ||
| - via environment variables | ||
| - via a separate config file | ||
| - via a combination of CLI, environment variables, and/or config file |
There was a problem hiding this comment.
How is a config file accessed by the providers?
There was a problem hiding this comment.
Config is going to be a singleton inited and loaded in the Gateway class and then passed itself or specific config like HTTP Server Config to the corresponding structs that needs it. Something like that. I did not mention that detail here as the contract is more important to decide on and discuss then to focus on how we get there under the hood.
…e most minimalistic config possible. Elaborated the reqs. Added rejected alternatives & future work
| cert_path: | ||
| # other configs | ||
|
|
||
| routes: |
There was a problem hiding this comment.
@mkrueger12 our conversation around embeddings API in GEP0002 and ideas to support TTS & STT models have opened my eyes on the previously proposed strategy which is to have one uniform list of pools that support all API Glide providers.
I think that if you consider these two type of APIs, it's clear that the idea is not super viable. There is no reason to impose LLM lifecycle on something like STT/TTS. They are just too different. And we should take care of them considering their specific.
Language, embeddings, transcribers, synthesizers are just too different to treat them the same way and we might end up being in trouble trying to go that route.
So the new idea is to have type-specific model pools (e.g. language, embeddings, transcribers, synthesizers). This should aid separating logic needed (fallbacking, load balancing, unified request/response schemas, etc). for one type of models from logic that is appropriate in another.
The current config examples have reflected this approach.
Feel free to take a look and let know how does that feel 🙌
There was a problem hiding this comment.
Yep this makes complete sense. Lets go that route
No description provided.