Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ feat: add DEFAULT_MODEL environment variable #280

Merged
merged 3 commits into from
Mar 29, 2023
Merged

✨ feat: add DEFAULT_MODEL environment variable #280

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bffa128
:sparkles: feat: add DEFAULT_MODEL environment variable
thomasleveil Mar 28, 2023
b404127
set the model maxLength setting in the models definition
thomasleveil Mar 28, 2023
37e597c
set the model tokenLimit setting in the models definition
thomasleveil Mar 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
set the model tokenLimit setting in the models definition
  • Loading branch information
@thomasleveil
thomasleveil committed Mar 28, 2023
commit 37e597c96f3ed574729235272f6387f312d4e3ab
5 changes: 1 addition & 4 deletions pages/api/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { ChatBody, Message } from '@/types/chat';
import { OpenAIModelID } from '@/types/openai';
import { DEFAULT_SYSTEM_PROMPT } from '@/utils/app/const';
import { OpenAIStream } from '@/utils/server';
import tiktokenModel from '@dqbd/tiktoken/encoders/cl100k_base.json';
Expand All @@ -22,8 +21,6 @@ const handler = async (req: Request): Promise<Response> => {
tiktokenModel.pat_str,
);

const tokenLimit = model.id === OpenAIModelID.GPT_4 ? 6000 : 3000;

let promptToSend = prompt;
if (!promptToSend) {
promptToSend = DEFAULT_SYSTEM_PROMPT;
Expand All @@ -38,7 +35,7 @@ const handler = async (req: Request): Promise<Response> => {
const message = messages[i];
const tokens = encoding.encode(message.content);

if (tokenCount + tokens.length > tokenLimit) {
if (tokenCount + tokens.length > model.tokenLimit) {
break;
}
tokenCount += tokens.length;
Expand Down
3 changes: 3 additions & 0 deletions types/openai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ export interface OpenAIModel {
id: string;
name: string;
maxLength: number; // maximum length of a message
tokenLimit: number;
}

export enum OpenAIModelID {
Expand All @@ -17,10 +18,12 @@ export const OpenAIModels: Record<OpenAIModelID, OpenAIModel> = {
id: OpenAIModelID.GPT_3_5,
name: 'GPT-3.5',
maxLength: 12000,
tokenLimit: 3000,
},
[OpenAIModelID.GPT_4]: {
id: OpenAIModelID.GPT_4,
name: 'GPT-4',
maxLength: 24000,
tokenLimit: 6000,
},
};