Skip to content

Dev to staging #466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 100 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
eb2f4fe
Integration_qa test (#375)
abhishekkumar-27 May 31, 2024
29d1722
recent merges
kartikpersistent May 31, 2024
0ef6fea
pdf deletion due to out of diskspace
kartikpersistent May 31, 2024
6393bf4
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent May 31, 2024
8cbd6f1
fixed status blank issue
kartikpersistent Jun 3, 2024
0656dae
Rendering the file name instead of link for gcs and s3 sources in the…
kartikpersistent Jun 3, 2024
4da1555
Convert is_cancelled value from string to bool
praveshkumar1988 Jun 3, 2024
3f32953
added the default page size
kartikpersistent Jun 3, 2024
7205698
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 3, 2024
ff18d1d
Issue fixed Processed chunked as 0 when file re-process again
praveshkumar1988 Jun 3, 2024
133057a
Youtube timestamps (#386)
aashipandya Jun 3, 2024
f00089e
groq llm integration backend (#286)
aashipandya Jun 3, 2024
fc63584
offset in chunks (#389)
aashipandya Jun 5, 2024
ba8036e
page number in gcs loader (#393)
aashipandya Jun 5, 2024
a4603ab
added youtube timestamps (#392)
vasanthasaikalluri Jun 5, 2024
555ed69
chat pop up button (#387)
prakriti-solankey Jun 5, 2024
bea09f3
chunks create before extraction using is_pre_process variable (#383)
praveshkumar1988 Jun 5, 2024
426492b
fixed the layout issue
kartikpersistent Jun 6, 2024
428154e
Populate graph schema (#399)
praveshkumar1988 Jun 6, 2024
64db13c
conditionally-including-the-gcs-login-flow-in-gcs-as-source (#396)
kartikpersistent Jun 6, 2024
6296859
Fixed issue : Remove extra unused param
praveshkumar1988 Jun 6, 2024
423d024
get emb only if used (#278)
im-ajaymeena Jun 7, 2024
da33904
Chatbot chunks (#402)
prakriti-solankey Jun 7, 2024
13d3d1a
Settings modal to support generating the labels from the llm by using…
kartikpersistent Jun 7, 2024
9beb1cf
fixed css issue
kartikpersistent Jun 7, 2024
b2423d4
fixed status blank issue
kartikpersistent Jun 11, 2024
b669bbc
Modified response when no docs is retrived (#413)
vasanthasaikalluri Jun 11, 2024
d161459
Fixed env/docker-compose for local deployments + README doc (#410)
msenechal Jun 11, 2024
f50f410
Support for all unstructured files (#401)
aashipandya Jun 11, 2024
19c4f35
Settings modal to support generating the labels from the llm by using…
kartikpersistent Jun 11, 2024
45db324
gcs file content read via storage client (#417)
aashipandya Jun 11, 2024
a9d4ea2
pypdf2 to read files from gcs (#420)
aashipandya Jun 12, 2024
a4a30b8
407 remove driver from frontend (#416)
prakriti-solankey Jun 12, 2024
b07b336
Css handling of info modal and Tooltips (#418)
prakriti-solankey Jun 12, 2024
3ddb17e
Fixed retrival bugs (#421)
vasanthasaikalluri Jun 12, 2024
5f55498
yarn format fixes
kartikpersistent Jun 12, 2024
dc01f55
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 12, 2024
f82dbb5
changed the delete message
kartikpersistent Jun 12, 2024
b4f99f9
added the cancel button
kartikpersistent Jun 12, 2024
fdab00a
changed the message on tooltip
kartikpersistent Jun 12, 2024
c803fa1
added space
kartikpersistent Jun 12, 2024
b71693b
UI fixes
kartikpersistent Jun 12, 2024
75074cb
tooltip for setting
prakriti-solankey Jun 12, 2024
d7c0f08
updated req
prakriti-solankey Jun 13, 2024
d34e811
wikipedia URL input (#424)
kartikpersistent Jun 13, 2024
3076900
speech and copy (#422)
prakriti-solankey Jun 13, 2024
6d729dd
Fixed issue for out of range in KNN API
praveshkumar1988 Jun 13, 2024
8c00f47
solved conflicts
kartikpersistent Jun 13, 2024
e9b7e84
conflict solved
kartikpersistent Jun 13, 2024
3c72893
Remove logging info from update KNN API
praveshkumar1988 Jun 13, 2024
a66850d
tooltip changes
kartikpersistent Jun 13, 2024
7c12a82
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 13, 2024
85899b7
format and lint fixes
kartikpersistent Jun 13, 2024
a9517fa
responsiveness changes
kartikpersistent Jun 13, 2024
e795ae0
Fixed issue for total pages GCS, S3
praveshkumar1988 Jun 14, 2024
fcff39f
UI polishing (#428)
kartikpersistent Jun 14, 2024
0576eed
speech state (#426)
prakriti-solankey Jun 14, 2024
9886ad9
delete wording change
kartikpersistent Jun 14, 2024
f5eaf06
Total pages in buckets (#431)
aashipandya Jun 14, 2024
7b43e4f
Merge branch 'STAGING' into DEV
praveshkumar1988 Jun 14, 2024
4310ace
removed the max width
kartikpersistent Jun 14, 2024
7e6c2e9
Update FileTable.tsx
kartikpersistent Jun 14, 2024
a346905
Update the docker file
praveshkumar1988 Jun 17, 2024
1bcd343
Modified prompt (#438)
vasanthasaikalluri Jun 17, 2024
6d1f76e
Update Dockerfile
karanchellani Jun 17, 2024
73d123c
Update Dockerfile
karanchellani Jun 17, 2024
7fe2fa9
Update Dockerfile
karanchellani Jun 17, 2024
2013cd3
rendering Fix
prakriti-solankey Jun 18, 2024
a9ed06b
Local file upload gcs (#442)
aashipandya Jun 18, 2024
c972a2a
Modified chat length and entities used (#443)
vasanthasaikalluri Jun 18, 2024
e43f553
metadata for unstructured files (#446)
aashipandya Jun 18, 2024
9301ad1
Unstructured file metadata (#447)
aashipandya Jun 18, 2024
2a30a04
icons added to chunks (#435)
prakriti-solankey Jun 18, 2024
7827532
fixed gcs status message issue
kartikpersistent Jun 19, 2024
91fbc5c
added if check for failed count
kartikpersistent Jun 19, 2024
e3fbc1d
Null issue Fixed from backend for upload API and graph_document when …
praveshkumar1988 Jun 19, 2024
8627702
added word break issue
kartikpersistent Jun 19, 2024
51c0498
Added neo4j-rust-ext
vasanthasaikalluri Jun 20, 2024
0af4a3c
processing time estimation based on bytes
kartikpersistent Jun 20, 2024
781c5f3
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 20, 2024
45f5205
File extension upper case fixed, File delete from GCS or local based …
praveshkumar1988 Jun 20, 2024
090a6dd
timer per byte
kartikpersistent Jun 20, 2024
6336d10
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 20, 2024
0cc7586
Update Dockerfile
karanchellani Jun 20, 2024
9b00547
Adding sort rows on the table (#451)
msenechal Jun 20, 2024
110f4fa
Gcs upload folder hashed (#453)
praveshkumar1988 Jun 20, 2024
7291121
upload all unstructuredfiles to gcs (#455)
aashipandya Jun 20, 2024
0393d32
Mofified chunk query (#454)
vasanthasaikalluri Jun 20, 2024
7c8624f
Added libre office for fixing error -- soffice command was not found.…
praveshkumar1988 Jun 21, 2024
f52c31f
Fix the PARTIAL CONTENT issue
praveshkumar1988 Jun 21, 2024
7e0edc6
File-table no data found (#456)
prakriti-solankey Jun 21, 2024
59882c5
Llm format change (#459)
kartikpersistent Jun 21, 2024
a77d5ee
commented total page code (#460)
kartikpersistent Jun 21, 2024
5df46cf
format fixes
kartikpersistent Jun 21, 2024
3517cd0
removed the disabled check on dropdown
kartikpersistent Jun 21, 2024
00d0182
Large file env
kartikpersistent Jun 21, 2024
636fd41
Merge branch 'STAGING' into DEV
praveshkumar1988 Jun 21, 2024
a52b659
added upload api
kartikpersistent Jun 21, 2024
bd14067
Merge branch 'DEV' of https://github.com/neo4j-labs/llm-graph-builder…
kartikpersistent Jun 21, 2024
e522c16
changed the dropzone error message
kartikpersistent Jun 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/src/document_sources/gcs_bucket.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def get_documents_from_gcs(gcs_project_id, gcs_bucket_name, gcs_bucket_folder, g
else:
blob_name = gcs_blob_filename

logging.info(f"GCS project_id : {gcs_project_id}")
logging.info(f"GCS project_id : {gcs_project_id}")

if access_token is None:
storage_client = storage.Client(project=gcs_project_id)
Expand Down
14 changes: 4 additions & 10 deletions frontend/src/components/DropZone.tsx
Original file line number Diff line number Diff line change
@@ -1,16 +1,15 @@
import axios from 'axios';
import { Dropzone, Flex, Typography } from '@neo4j-ndl/react';
import React, { useState, useEffect, FunctionComponent } from 'react';
import Loader from '../utils/Loader';
import { v4 as uuidv4 } from 'uuid';
import { useCredentials } from '../context/UserCredentials';
import { useFileContext } from '../context/UsersFiles';
import CustomAlert from './Alert';
import { CustomFile, CustomFileBase, UploadResponse, alertStateType } from '../types';
import { CustomFile, CustomFileBase, UserCredentials, alertStateType } from '../types';
import { buttonCaptions, chunkSize } from '../utils/Constants';
import { url } from '../utils/Utils';
import { InformationCircleIconOutline } from '@neo4j-ndl/react/icons';
import IconButtonWithToolTip from './IconButtonToolTip';
import { uploadAPI } from '../utils/FileAPI';

const DropZone: FunctionComponent = () => {
const { filesData, setFilesData, model } = useFileContext();
Expand Down Expand Up @@ -122,12 +121,7 @@ const DropZone: FunctionComponent = () => {
})
);
try {
const apiResponse = await axios.post<UploadResponse>(`${url()}/upload`, formData, {
headers: {
'Content-Type': 'multipart/form-data',
},
});

const apiResponse=await uploadAPI(chunk,userCredentials as UserCredentials,model,chunkNumber,totalChunks,file.name)
if (apiResponse?.data.status === 'Failed') {
throw new Error(
JSON.stringify({ message: apiResponse.data.message, fileName: apiResponse.data.file_name })
Expand Down Expand Up @@ -283,7 +277,7 @@ const DropZone: FunctionComponent = () => {
setalertDetails({
showAlert: true,
alertType: 'error',
alertMessage: 'Failed To Upload, File is larger than 15MB',
alertMessage: 'Failed To Upload, Unsupported file extention',
});
}
},
Expand Down
5 changes: 4 additions & 1 deletion frontend/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,11 @@ export type ExtractParams = {
} & { [key: string]: any };

export type UploadParams = {
file: File;
file: Blob;
model: string;
chunkNumber:number;
totalChunks:number;
originalname:string
} & { [key: string]: any };

export type FormDataParams = ExtractParams | UploadParams;
Expand Down
6 changes: 3 additions & 3 deletions frontend/src/utils/FileAPI.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ import { UserCredentials, ExtractParams, UploadParams } from '../types';
import { apiCall } from '../services/CommonAPI';

// Upload Call
export const uploadAPI = async (file: File, userCredentials: UserCredentials, model: string): Promise<any> => {
const urlUpload = `${url()}/sources`;
export const uploadAPI = async (file: Blob, userCredentials: UserCredentials, model: string, chunkNumber: number, totalChunks: number,originalname:string): Promise<any> => {
const urlUpload = `${url()}/upload`;
const method: Method = 'post';
const commonParams: UserCredentials = userCredentials;
const additionalParams: UploadParams = { file, model };
const additionalParams: UploadParams = { file, model, chunkNumber, totalChunks ,originalname};
const response = await apiCall(urlUpload, method, commonParams, additionalParams);
return response;
};
Expand Down