Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add support for new models #1391

Closed
wants to merge 12 commits into from
Closed

[WIP] Add support for new models #1391

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b8cbea3
Add support for meta-llama-3.1 8B & 8B Instruct
Smartappli Jul 27, 2024
61cfac2
Add support for Meta-Llama-3.1 70B & 70B Instruct
Smartappli Jul 27, 2024
95c22a3
Update README.md
Smartappli Jul 27, 2024
cb2c787
Update llama-cpp-python to version 0.2.84
Smartappli Jul 28, 2024
373efda
Add support for Mistral-Nemo-Instruct
Smartappli Jul 28, 2024
d7bc34a
Add support for Gemma-2 9B Instruct
Smartappli Jul 28, 2024
29761d0
Update README.md
Smartappli Jul 28, 2024
df8bf63
Add support for Codestral 22B
Smartappli Jul 28, 2024
f9597f5
Update README.md
Smartappli Jul 28, 2024
db33c3e
revert
Smartappli Jul 28, 2024
d306f1c
revert
Smartappli Jul 28, 2024
b2301a2
Update models.json
Smartappli Jul 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ The following Environment Variables are available:
| Variable Name | Description | Default Value |
|-----------------------|---------------------------------------------------------|--------------------------------------|
| `SERGE_DATABASE_URL` | Database connection string | `sqlite:////data/db/sql_app.db` |
| `SERGE_JWT_SECRET` | Key for auth token encryption. Use a random string | `uF7FGN5uzfGdFiPzR` |
| `SERGE_JWT_SECRET` | Key for auth token encryption. Use a random string | `uF7FGN5uzfGdFiPzR` |
| `SERGE_SESSION_EXPIRY`| Duration in minutes before a user must reauthenticate | `60` |
| `NODE_ENV` | Node.js running environment | `production` |
| `NODE_ENV` | Node.js running environment | `production` |

## 🖥️ Windows

Expand Down Expand Up @@ -83,7 +83,7 @@ Instructions for setting up Serge on Kubernetes can be found in the [wiki](https
| **Medalpaca** | 13B |
| **Medicine** | Chat, LLM |
| **Meditron** | 7B, 7B-Chat, 70B |
| **Meta-LlaMA-3** | 8B, 8B-Instruct, 70B, 70B-Instruct |
| **Meta-LlaMA-3** | 3-8B, 3.1-8B, 3-8B-Instruct, 3.1-8B-Instruct, 3-70B, 3.1-70B, 3-70B-Instruct, 3.1-70B-Instruct |
| **Mistral** | 7B-V0.1, 7B-Instruct-v0.2, 7B-OpenOrca |
| **MistralLite** | 7B |
| **Mixtral** | 8x7B-v0.1, 8x7B-Dolphin-2.7, 8x7B-Instruct-v0.1 |
Expand Down
44 changes: 44 additions & 0 deletions api/src/serge/data/models.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,6 +605,17 @@
}
]
},
{
"name": "Meta-Llama-3_1-8B",
"repo": "QuantFactory/Meta-Llama-3.1-8B-GGUF",
"files": [
{
"name": "q4_K_M",
"filename": "Meta-Llama-3.1-8B.Q4_K_M.gguf",
"disk_space": 4920733856.0
}
]
},
{
"name": "Meta-Llama-3-8B-Instruct",
"repo": "QuantFactory/Meta-Llama-3-8B-Instruct-GGUF",
Expand All @@ -616,6 +627,17 @@
}
]
},
{
"name": "Meta-Llama-3_1-8B-Instruct",
"repo": "QuantFactory/Meta-Llama-3.1-8B-Instruct-GGUF",
"files": [
{
"name": "q4_K_M",
"filename": "Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf",
"disk_space": 4920734240.0
}
]
},
{
"name": "Meta-Llama-3-70B",
"repo": "NousResearch/Meta-Llama-3-70B-GGUF",
Expand All @@ -627,6 +649,17 @@
}
]
},
{
"name": "Meta-Llama-3_1-70B",
"repo": "mradermacher/Meta-Llama-3.1-70B-GGUF",
"files": [
{
"name": "q4_K_M",
"filename": "Meta-Llama-3.1-70B.Q4_K_M.gguf",
"disk_space": 42520393600.0
}
]
},
{
"name": "Meta-Llama-3-70B-Instruct",
"repo": "QuantFactory/Meta-Llama-3-70B-Instruct-GGUF",
Expand All @@ -637,6 +670,17 @@
"disk_space": 42520906208.0
}
]
},
{
"name": "Meta-Llama-3_1-70B-Instruct",
"repo": "mradermacher/Meta-Llama-3.1-70B-Instruct-GGUF",
"files": [
{
"name": "q4_K_M",
"filename": "Meta-Llama-3.1-70B-Instruct.Q4_K_M.gguf",
"disk_space": 42520394080.0
}
]
}
]
},
Expand Down
2 changes: 1 addition & 1 deletion scripts/serge.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
LLAMA_PYTHON_VERSION=0.2.82
LLAMA_PYTHON_VERSION=0.2.84
SERGE_ENABLE_IPV4=true
SERGE_ENABLE_IPV6=false