Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Routing] Find a way to normalize latency #78

Closed
roma-glushko opened this issue Jan 14, 2024 · 0 comments · Fixed by #111
Closed

[Routing] Find a way to normalize latency #78

roma-glushko opened this issue Jan 14, 2024 · 0 comments · Fixed by #111
Assignees
@roma-glushko
Labels
area:resiliency area:routing
Milestone
Glide: Private Pr...

Comments

@roma-glushko
Copy link
Member

We need to find a way to track model latency that would be independent from the response size.

Options:

  • use the time to fist byte metric (needs to instrument clients to get that info)
  • count the generated tokens in the response and dividing it by the response time
  • something else?

Reference:
This was referenced Jan 14, 2024
Merged
Merged
@roma-glushko roma-glushko self-assigned this Jan 16, 2024
roma-glushko added a commit that referenced this issue Jan 29, 2024
@roma-glushko
#78: Normalize response latency by response token count & renamed Tok…
7301bb7 
…enCount to TokenUsage
roma-glushko added a commit that referenced this issue Jan 29, 2024
@roma-glushko
#78: FIxed broken references
593518f
@roma-glushko roma-glushko linked a pull request Jan 29, 2024 that will close this issue
🔧 #78: Normalize response latency by response token count #111
Merged
roma-glushko added a commit that referenced this issue Jan 30, 2024
@roma-glushko
🔧 #78: Normalize response latency by response token count (#111)
72f2c75 
- Normalize response latency by response token count 
- renamed TokenCount to TokenUsage
@roma-glushko roma-glushko mentioned this issue Jan 31, 2024
Merged
roma-glushko added a commit that referenced this issue Jan 31, 2024
@roma-glushko
📦 v0.0.1 (Private Preview) (#115)
b02c949 
## Changelog

### Features

- ✨ #81: Allow to chat message based for specific models (@mkrueger12)

### Improvements

- 🔧 #78: Normalize response latency by response token count (@roma-glushko)
- 📝 #112 added the CLI banner info (@roma-glushko)

### Miscellaneous 

- 📝 #114 Make links actual across the project (@roma-glushko)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roma-glushko roma-glushko

Labels
area:resiliency area:routing
Projects
None yet
Milestone
Glide: Private Preview (MVP)
Development

Successfully merging a pull request may close this issue.

🔧 #78: Normalize response latency by response token count EinStack/glide
1 participant
@roma-glushko