Skip to content
#

text-tokenization

Here are 17 public repositories matching this topic...

Successfully fine-tuned a pretrained DistilBERT transformer model that can classify social media text data into one of 4 cyberbullying labels i.e. ethnicity/race, gender/sexual, religion and not cyberbullying with a remarkable accuracy of 99%.

  • Updated Jun 10, 2024
  • Jupyter Notebook

Successfully established a text summarization model using Seq2Seq modeling with Luong Attention, which can give a short and concise summary of the global news headlines.

  • Updated May 6, 2024
  • Jupyter Notebook

Extract text content from an HTML page, process it, and extract unique words from the processed text. This notebook utilizes various text processing techniques including cleaning, normalization, tokenization, lemmatization or stemming, and stop words removal.

  • Updated Apr 5, 2024
  • Jupyter Notebook

ISPY ChatBot ISPY is a chatbot designed for ISP customer service, providing automated responses and assistance for various queries such as connection issues, payments, and service requests. Built using Python with libraries like nltk and newspaper3k, it simulates conversation and handles customer interactions effectively.

  • Updated Apr 14, 2024
  • Jupyter Notebook

Nihotip is a web app that lets users explore Japanese text through interactive tokenization and detailed insights. Built with React and Python, it offers a dynamic way to analyze words and symbols with tooltips for deeper understanding.

  • Updated Sep 26, 2024
  • JavaScript

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

  • Updated Oct 16, 2024
  • Python

Improve this page

Add a description, image, and links to the text-tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the text-tokenization topic, visit your repo's landing page and select "manage topics."

Learn more