community: add Needle retriever and document loader integration

[JANHMS]

• PR title: "community: add Needle retriever and document loader integration"

  • Where "package" is whichever of langchain, community, core, etc. is being modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI changes.
  • Example: "community: add foobar LLM"

• PR message: Delete this entire checklist and replace with

  • Description: This PR adds a new integration for Needle, which includes:
    • NeedleRetriever: A retriever for fetching documents from Needle collections.
    • NeedleLoader: A document loader for managing and loading documents into Needle collections.
    • Example notebooks demonstrating usage have been added in:
      • docs/docs/integrations/retrievers/needle.ipynb
      • docs/docs/integrations/document_loaders/needle.ipynb.
  • Dependencies: The needle-python package is required as an external dependency for accessing Needle's API. It has been added to the extended testing dependencies list.
  • Twitter handle: Feel free to mention me if this PR gets announced: needlexai.

• Add tests and docs: If you're adding a new integration, please include

  1. Unit tests have been added for both NeedleRetriever and NeedleLoader in libs/community/tests/unit_tests. These tests mock API calls to avoid relying on network access.
  2. Example notebooks have been added to docs/docs/integrations/, showcasing both retriever and loader functionality.

• Lint and test: Run make format, make lint, and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

  • make format: Passed
  • make lint: Passed
  • make test: Passed (requires needle-python to be installed locally; this package is not added to LangChain dependencies).

Additional guidelines:

• Optional dependencies are imported only within functions.
• No dependencies have been added to pyproject.toml files except for those required for unit tests.
• The PR does not touch more than one package.
• Changes are fully backwards compatible.
• Community additions are not re-imported into LangChain core.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 3, 2024 10:06pm

[JANHMS]

@efriis should be ok, Vercel deployment fails due to

Error: Docs markdown link couldn't be resolved: (MIGRATION.md) in source file "/vercel/path0/docs/docs/langserve.md" for version current

I guess that is a FP? Should be all good with the PR.

[oeken]

Hi Langchain team, @efriis 👋 This is Onur from Needle. We are waiting on this review for more than 2 months. There was another PR previously which we had to close and open this one since the review flow changed.

Could you finalize reviewing this PR? Obviously we want to avoid same scenario happening again. Thanks for your efforts!

[efriis]

hey sorry for the delay! would you be interested in integrating like this instead? then you don't have to deal with the "needs support" status for the integration, and docs PRs are much easier for us to review.

Please take a look at the CI failures as well (looks like it's on the docs build, which would apply regardless of format of the integration!)

https://python.langchain.com/docs/contributing/how_to/integrations/

[efriis]

actually merging in master to rebuild the docs - @JANHMS ' comment was addressed earlier, so it may fix itself

[JANHMS]

Thank @efriis it did solve the issue by itself.

I need this to be a part of the langchain_community package for another integration I am making.

Are we ready to merge?

hey sorry for the delay! would you be interested in integrating like this instead? then you don't have to deal with the "needs support" status for the integration, and docs PRs are much easier for us to review.

Please take a look at the CI failures as well (looks like it's on the docs build, which would apply regardless of format of the integration!)

[JANHMS]

Hey @efriis sorry to bump again, but we actually need this asap and are waiting for a long time. It blocks us on another side. Please let's go ahead and merge so we have it available as the community package.

[JANHMS]

@efriis

[JANHMS]

@efriis

[JANHMS]

@eyurtsev @efriis

[oeken]

@eyurtsev Hi Eugene - sorry about chasing, but it's ready to be merged from our side. Can you take a look? We understand you guys are busy these days, we appreciate the effort! 🙏