Up to 100x faster strings for C, C++, CUDA, Python, Rust, Swift, JS, & Go, leveraging NEON, AVX2, AVX-512, SVE, GPGPU, & SWAR to accelerate search, hashing, sorting, edit distances, sketches, and memory ops 🦖

Dingo: A Comprehensive AI Data Quality Evaluation Tool

Process Common Crawl data with Python and Spark

News crawling with StormCrawler - stores content as WARC

Statistics of Common Crawl monthly archives mined from URL index files

🕷️ The pipeline for the OSCAR corpus

Drill into WARC web archives

Tools to construct and process Common Crawl webgraphs

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

Various Jupyter notebooks about Common Crawl data

A dataset for knowledge base population research using Common Crawl and DBpedia.

🕸 GlotCC Dataset and Pipline -- NeurIPS 2024

German small and large versions of GPT2.

The website of the Oscar Project

Common Crawl's processing tools

We explore data by using Big Data Analysis and Visualization skills. To obtain this, we perform 3 main operations. i.e. i)Data Aggregation through different sources. ii) Big Data Analysis using MapReduce and iii) Visualization through Tableau. Data Analysis is very critical in understanding the data, and what we can do with the data. For small d…

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

Various Common Crawl utilities in Clojure.

Distributed download scripts for Common Crawl data

Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's data concurrently with Python and Scrapy.