Full Text Search On Local Files With Inverted Index

Demo

Features

• Text extraction from PDFs, Microsoft Word DOCX and text-based formats
• Disk-persistence of inverted index
• Validation of inverted index
• Command-line utility

Setup

Make sure Java is installed on your system, with JAVA_HOME pointing to a JDK installation. You may clone the project from the GitHub repository, and build it with gradlew present in the root of the repository,

$> git clone https://github.com/shubham0204/full-text-search
$> cd full-text-search
$> ./gradlew build

To execute tests,

$> ./gradlew test

To build the fat/uber JAR,

$> ./gradlew shadowJar

Usage

Index

$> java -jar fulltextsearch.jar index build [dir]
$> java -jar fulltextsearch.jar index info [dir]
$> java -jar fulltextsearch.jar index rm [dir]

Use fulltextsearch index --help for description of each command.

Query

$> fulltextsearch query [dir]

Dependencies

• Apache PdfBox
• Apache POI
• picocli
• shadow

Useful Resources

• Wikipedia - Full Text Search
• Let's build a Full-Text Search engine
• Building a full-text search engine in 150 lines of Python code
• Building a Full Text Search Engine
• How to Implement Inverted Index Data Structure in Java
• Reddit discussion on full-text-search