NLP for Oriya

This repository contains State of the Art Language models and Classifier for Oriya, which is spoken in the Indian state of Odisha.

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

1. Oriya Wikipedia Articles

2. Oriya News Dataset

Open Source Datasets

1. IndicNLP News Article Classification Dataset - Oriya

Results

Language Model Perplexity (on validation set)

Architecture/Dataset	Oriya Wikipedia Articles
ULMFiT	26.57
TransformerXL	26.81

Classification Metrics

ULMFiT

Dataset	Accuracy	MCC	Notebook to Reproduce results
IndicNLP News Article Classification Dataset - Oriya	98.83	98.44	Link

Visualizations

Word Embeddings

Architecture	Visualization
ULMFiT	Embeddings projection
TransformerXL	Embeddings projection

Pretrained Models

Language Models

Download pretrained Language Models from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary from here