Open-Source Pre-Processing Tools for Unstructured Data

The unstructured library provides open-source components for pre-processing text documents such as PDFs, HTML and Word Documents. These components are packaged as bricks 🧱, which provide users the building blocks they need to build pipelines targeted at the documents they care about. Bricks in the library fall into three categories:

• 🧩 Partitioning bricks that break raw documents down into standard, structured elements.
• 🧹 Cleaning bricks that remove unwanted text from documents, such as boilerplate and sentence fragments.
• 🎭 Staging bricks that format data for downstream tasks, such as ML inference and data labeling.
  
  

✴️ Installation

To install the library, run pip install unstructured.

☕ Getting Started

• Using pyenv to manage virtualenv's is recommended but not necessary

  • Mac install instructions. See here for more detailed instructions.
    • brew install pyenv-virtualenv
    • pyenv install 3.8.15
  • Linux instructions are available here.

• Create a virtualenv to work in and activate it, e.g. for one named unstructured:

  pyenv virtualenv 3.8.15 unstructured
pyenv activate unstructured

• Run make install-project-local

👏 Quick Tour

You can run this Colab notebook to run the examples below.

The following examples show how to get started with the unstructured library. You can parse HTML and PDF documents with one line of code!

See our documentation page for a full description of the features in the library.

HTML Parsing

You can parse an HTML document using the following workflow:

from unstructured.partition.html import partition_html

elements = partition_html("example-docs/example-10k.html")
print("\n\n".join([str(el) for el in elements[:5]]))

The print statement will show the following text:

UNITED STATES

SECURITIES AND EXCHANGE COMMISSION

Washington, D.C. 20549

FORM 10-K

ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934

And elements will be a list of elements in the HTML document, similar to the following:

[<unstructured.documents.elements.Title at 0x169cbe820>,
 <unstructured.documents.elements.NarrativeText at 0x169cbe8e0>,
 <unstructured.documents.elements.NarrativeText at 0x169cbe3a0>]

PDF Parsing

You can use the following workflow to parse PDF documents.

from unstructured.partition.pdf import partition_pdf

elements = partition_pdf("example-docs/layout-parser-paper.pdf")
print(doc)

At this point, print(elements) will print out a string representation of the PDF file. The first page of output looks like the following:

LayoutParser : A Unified Toolkit for Deep Learning Based Document Image Analysis

Zejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and
Weining Li 5

Abstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural
networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.
However, various factors like loosely organized codebases and sophisticated model configurations complicate the easy
reuse of im- portant innovations by a wide audience. Though there have been on-going efforts to improve reusability and
simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none
of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA
is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper
introduces LayoutParser , an open-source library for streamlining the usage of DL in DIA research and applica- tions.
The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models
for layout de- tection, character recognition, and many other document processing tasks. To promote extensibility,
LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digiti- zation
pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in
real-word use cases. The library is publicly available at https://layout-parser.github.io

Keywords: Document Image Analysis · Deep Learning · Layout Analysis · Character Recognition · Open Source library ·
Toolkit.

Introduction

Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks
including document image classification [11,

The Document has a pages attribute consisting of Page object and the Page object has an element attribute consisting of Element objects. Sub-types of the Element class represent different components of a document, such as NarrativeText and Title. You can use these normalized elements to zero in on the components of a document you most care about.

💂‍♂️ Security Policy

See our security policy for information on how to report security vulnerabilities.

📚 Learn more

Section	Description
Company Website	Unstructured.io product and company info
Documentation	Full API documentation