unstructured, community, initialize langchain-unstructured package

[Coniferish]

Update (2):

A single UnstructuredLoader is added to handle both local and api partitioning. This loader also handles single or multiple documents.

Changes in community:

Changes here do not affect users. In the initial process of using the SDK for the API Loaders, the Loaders in community were refactored.
Other changes include:
The UnstructuredBaseLoader has a new check to see if both mode="paged" and chunking_strategy="by_page". It also now has Element.element_id added to the Document.metadata. UnstructuredAPIFileLoader and UnstructuredAPIFileIOLoader. As such, now both directly inherit from UnstructuredBaseLoader and initialize their file_path/file attributes respectively and implement their own _post_process_elements methods.

---

Update:

New SDK Loaders in a partner package are introduced to prevent breaking changes for users (see discussion below).

TODO:

• Test docstring examples

---

• Description: UnstructuredAPIFileIOLoader and UnstructuredAPIFileLoader calls to the unstructured api are now made using the unstructured-client sdk.

• New Dependencies: unstructured-client

• Add tests and docs: If you're adding a new integration, please include

  • a test for the integration, preferably unit tests that do not rely on network access,
  • update the description in docs/docs/integrations/providers/unstructured.mdx

• Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

• Make sure optional dependencies are imported within a function.
• Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
• Most PRs should not touch more than one package.
• Changes should be backwards compatible.
• If you are adding something to community, do not re-import it in langchain.

TODO:

• Update https://python.langchain.com/v0.1/docs/integrations/document_loaders/unstructured_file/#unstructured-api
  • langchain/docs/docs/integrations/document_loaders/unstructured_file.ipynb
  • The description here needs to indicate that users should install unstructured-client instead of unstructured. Read over closely to look for any other changes that need to be made.
• Update the lazy_load method in UnstructuredBaseLoader to handle json responses from the API instead of just lists of elements.
  • This method may need to be overwritten by the API loaders instead of changing it in the UnstructuredBaseLoader.
• Update the documentation links in the class docstrings (the Unstructured documents have moved)
• Update Document.metadata to include element_id (see thread here)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 24, 2024 11:18pm