Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In-context document support for Anthropic and Google models #5130

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Next Next commit
Upload files directly to models when RAG is disabled
Reused the image handling functions, as the mime type was already being
set correctly.

For now the upload only happens if RAG is disabled. A better approach
would be to re-use the agents menu for deciding context vs RAG
  • Loading branch information
alex-torregrosa committed Jan 11, 2025
commit c52e2e86e72e1bc2698ed09758f556d6b2887bc9
10 changes: 4 additions & 6 deletions api/server/services/Files/Local/images.js
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,11 @@ function encodeImage(imagePath) {
* @returns {Promise<[MongoFile, string]>} - A promise that resolves to an array of results from updateFile and encodeImage.
*/
async function prepareImagesLocal(req, file) {
const { publicPath, imageOutput } = req.app.locals.paths;
const userPath = path.join(imageOutput, req.user.id);
const { publicPath, root } = req.app.locals.paths;

if (!fs.existsSync(userPath)) {
fs.mkdirSync(userPath, { recursive: true });
}
const filepath = path.join(publicPath, file.filepath);
const startPath = file.width ? publicPath : root;

const filepath = path.join(startPath, file.filepath);

const promises = [];
promises.push(updateFile({ file_id: file.file_id }));
Expand Down
3 changes: 2 additions & 1 deletion api/server/services/Files/images/encode.js
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ async function encodeAndFormat(req, files, endpoint, mode) {
for (let file of files) {
const source = file.source ?? FileSources.local;

if (!file.height) {
if (source == FileSources.vectordb) {
// Do not try to base-64 encode the file if RAG needs to be performed
promises.push([file, null]);
continue;
}
Expand Down
6 changes: 5 additions & 1 deletion api/server/services/Files/process.js
Original file line number Diff line number Diff line change
Expand Up @@ -379,9 +379,13 @@ const uploadImageBuffer = async ({ req, context, metadata = {}, resize = true })
*/
const processFileUpload = async ({ req, res, metadata }) => {
const isAssistantUpload = isAssistantsEndpoint(metadata.endpoint);
const has_rag = !!process.env.RAG_API_URL;
const assistantSource =
metadata.endpoint === EModelEndpoint.azureAssistants ? FileSources.azure : FileSources.openai;
const source = isAssistantUpload ? assistantSource : FileSources.vectordb;
const localSource = has_rag ? FileSources.vectordb : req.app.locals.fileStrategy;

const source = isAssistantUpload ? assistantSource : localSource;

const { handleFileUpload } = getStrategyFunctions(source);
const { file_id, temp_file_id } = metadata;

Expand Down