Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

community: add Needle retriever and document loader integration #28157

Merged
merged 31 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6f34186
init: needle langchain integration
JANHMS Nov 15, 2024
26bf2e5
trigger change
JANHMS Nov 15, 2024
4503f86
feat: needle integraiton
JANHMS Nov 16, 2024
25e43d7
techdebt: test and format
JANHMS Nov 16, 2024
362fcef
techdebt: lint
JANHMS Nov 16, 2024
2df0963
techdebt: lint
JANHMS Nov 16, 2024
5a42ad6
fix: make headers in docs
JANHMS Nov 16, 2024
1df8674
fix: whitespace
JANHMS Nov 16, 2024
883170d
fix: docstrings
JANHMS Nov 16, 2024
170f77e
format
JANHMS Nov 16, 2024
f1d3815
fix: add API reference header
JANHMS Nov 16, 2024
c31a2bd
fix: remove API refernece
JANHMS Nov 16, 2024
1035707
fix: API reference at footer
JANHMS Nov 16, 2024
106a52d
Merge branch 'master' into Integration-Needle
JANHMS Nov 16, 2024
7280ef5
fix: make lint
JANHMS Nov 16, 2024
e7cfb61
fix: import not on top since community package
JANHMS Nov 16, 2024
e9f56b7
fix: lint
JANHMS Nov 16, 2024
7719ad3
lint test
JANHMS Nov 16, 2024
8491a19
techdebt: make tests working with import
JANHMS Nov 16, 2024
c50bcb5
failing doc imports
JANHMS Nov 16, 2024
e736d66
techdebt: make test corrections
JANHMS Nov 17, 2024
d3361f4
fix: test imports
JANHMS Nov 17, 2024
8e2481f
fix: test imports
JANHMS Nov 17, 2024
dcd12ed
fix: pytest.mark.requires
JANHMS Nov 17, 2024
c205603
techdebt: add needle-python to extended_testing_deps
JANHMS Nov 17, 2024
066884f
fix: make extended tests work
JANHMS Nov 17, 2024
6974ed9
lint
JANHMS Nov 17, 2024
09e2b59
Merge branch 'master' into Integration-Needle
JANHMS Nov 17, 2024
032c796
Merge branch 'master' into Integration-Needle
efriis Nov 19, 2024
11f38a2
Merge branch 'master' into Integration-Needle
JANHMS Nov 20, 2024
528502e
Merge branch 'master' into Integration-Needle
efriis Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: import not on top since community package
  • Loading branch information
JANHMS committed Nov 16, 2024
commit e7cfb6163bbbe751e3cb81e0d71ff523929b35e4
5 changes: 3 additions & 2 deletions libs/community/langchain_community/document_loaders/needle.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

from langchain_core.document_loaders.base import BaseLoader
from langchain_core.documents import Document
from needle.v1 import NeedleClient
from needle.v1.models import FileToAdd


class NeedleLoader(BaseLoader):
Expand Down Expand Up @@ -57,6 +55,7 @@ def __init__(
Raises:
ValueError: If the collection ID is not provided.
"""
from needle.v1 import NeedleClient
super().__init__()
self.needle_api_key = needle_api_key
self.collection_id = collection_id
Expand Down Expand Up @@ -92,6 +91,8 @@ def add_files(self, files: dict) -> None:
Raises:
ValueError: If the collection is not properly initialized.
"""
from needle.v1.models import FileToAdd

self._get_collection()
assert self.client is not None, "NeedleClient must be initialized."

Expand Down
4 changes: 3 additions & 1 deletion libs/community/langchain_community/retrievers/needle.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from langchain_core.callbacks import CallbackManagerForRetrieverRun
from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever
from needle.v1 import NeedleClient
from pydantic import BaseModel, Field


Expand Down Expand Up @@ -39,6 +38,7 @@ class NeedleRetriever(BaseRetriever, BaseModel):
for doc in results:
print(doc.page_content)
"""
from needle.v1 import NeedleClient

needle_api_key: Optional[str] = Field(None, description="Needle API Key")
collection_id: Optional[str] = Field(
Expand All @@ -52,6 +52,8 @@ def _initialize_client(self) -> None:

If a client instance is already provided, this method does nothing.
"""
from needle.v1 import NeedleClient

if not self.client:
self.client = NeedleClient(api_key=self.needle_api_key)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def list(self, collection_id: str):

@pytest.mark.requires("needle-python")
def test_add_and_fetch_files(mocker: MockerFixture):
"""Test adding and fetching files using the NeedleLoader with a mock NeedleClient."""
"""Test adding and fetching files using the NeedleLoader with a mock."""
# Mock the NeedleClient to use the mock implementation
mocker.patch("needle.v1.NeedleClient", new=MockNeedleClient)

Expand Down
Loading