Commits

Branch selector

 main

User selector

soldni

Datepicker

All time

Commit History

Commits on Sep 4, 2024

• Update deduper.rs (allenai#197)

  soldni

  authored
  621a6f4

  Copy full SHA for 621a6f4

Commits on Aug 28, 2024

• force numpy to be below v2 (allenai#195)

  soldni

  authored
  2a4ef70

  Copy full SHA for 2a4ef70

Commits on Aug 26, 2024

• tokenizers (allenai#189)

  soldni

  authored
  6cb88c0

  Copy full SHA for 6cb88c0

Commits on Aug 22, 2024

• Changed entrypoint to increase IntelliJ compatibility (allenai#188)

  soldni

  authored
  16452bb

  Copy full SHA for 16452bb

Commits on Aug 17, 2024

• Count Bytes and Docs (allenai#186)

  Show description for f365470
  soldni

  authored
  f365470

  Copy full SHA for f365470

Commits on Aug 11, 2024

• fixed tests to match new mixer (allenai#184)

  soldni

  authored
  c4fa395

  Copy full SHA for c4fa395

Commits on Aug 10, 2024

• fixed empty attributes bug

  soldni

  committed
  8ab1d1d

  Copy full SHA for 8ab1d1d

Commits on Aug 6, 2024

• Better Filters Error Handling (allenai#171)

  Show description for b406546
  soldni

  authored
  b406546

  Copy full SHA for b406546

Commits on Aug 1, 2024

• V2 of Gopher tagger (allenai#181)

  Show description for a72c76b
  soldni

  authored
  a72c76b

  Copy full SHA for a72c76b

Commits on Jul 24, 2024

• Version bump for new release (1.0.4) (allenai#179)

  Show description for 5868453
  soldni

  authored
  5868453

  Copy full SHA for 5868453

Commits on Jul 13, 2024

• added option for tokenizer to split on special tokens (allenai#176)

  soldni

  authored
  ac91597

  Copy full SHA for ac91597

Commits on Jun 6, 2024

• Workaround to fix memory leak in HuggingFace tokenizer (allenai#169)

  Show description for 64886d9
  soldni

  authored
  64886d9

  Copy full SHA for 64886d9

• Adds ZST support in Deduper and Mixer (allenai#170)

  Show description for bf80be0
  soldni

  authored
  bf80be0

  Copy full SHA for bf80be0

Commits on May 21, 2024

• Adding Quality Classifier from Dolma 1.7 (allenai#163)

  Show description for a9eaf29
  soldni

  authored
  a9eaf29

  Copy full SHA for a9eaf29

Commits on May 2, 2024

• Add support for parsing WARC (allenai#153)

  Show description for 3bbea39
  soldni

  authored
  3bbea39

  Copy full SHA for 3bbea39

• fixing dtype option not being correctly propagated (allenai#154)

  soldni

  authored
  c1a4430

  Copy full SHA for c1a4430

Commits on Apr 5, 2024

• Add tests for multi-byte unicode spans in deduper. (allenai#145)

  soldni

  authored
  4ae0aef

  Copy full SHA for 4ae0aef

Commits on Apr 2, 2024

• Optionally add total/sum to output of analyzer (allenai#144)

  Show description for b775636
  soldni

  authored
  b775636

  Copy full SHA for b775636

Commits on Mar 25, 2024

• Add an option to improve tokenization shuffling (allenai#141)

  soldni

  authored
  c3d6243

  Copy full SHA for c3d6243

Commits on Mar 22, 2024

• test & fix for shuffling bug (allenai#140)

  soldni

  authored
  de24038

  Copy full SHA for de24038

Commits on Mar 21, 2024

• preparing release

  soldni

  committed
  c732011

  Copy full SHA for c732011

Commits on Mar 17, 2024

• Improves tool to compute statistics; adds deduplication options. (allenai#135)

  Show description for 58dedd3
  soldni

  authored
  58dedd3

  Copy full SHA for 58dedd3

Commits on Mar 6, 2024

• Added JQ syntax for replacements + min/max score. (allenai#133)

  Show description for a75bf09
  soldni

  authored
  a75bf09

  Copy full SHA for a75bf09

Commits on Mar 1, 2024

• Added Support for JQ syntax in include/exclude mixer config (allenai#131)

  soldni

  authored
  cd7d983

  Copy full SHA for cd7d983

Commits on Feb 21, 2024

• Tagger modules import (fix allenai#128) (allenai#129)

  soldni

  authored
  5b8109d

  Copy full SHA for 5b8109d

Commits on Feb 16, 2024

• Update CITATION.cff (allenai#126)

  soldni

  authored
  e7657e4

  Copy full SHA for e7657e4

Commits on Feb 13, 2024

• Add preliminary Dolma v1.7 configurations, fix corner case in tokens. (allenai#120)

  Show description for c6c0b47
  soldni

  authored
  c6c0b47

  Copy full SHA for c6c0b47

Commits on Feb 10, 2024

• upated cff and bibtex (allenai#118)

  soldni

  authored
  b9a8b94

  Copy full SHA for b9a8b94

• handle None and ""

  soldni

  committed
  6ad02ef

  Copy full SHA for 6ad02ef

• Taggers for URL Filtering (allenai#112)

  Show description for 70206e9
  soldni

  authored
  70206e9

  Copy full SHA for 70206e9

Commits on Feb 7, 2024

• Modify CI to build when version is incremented; increment to v1.0.1 (allenai#116)

  Show description for f6970d5
  soldni

  authored
  f6970d5

  Copy full SHA for f6970d5

Commits on Feb 1, 2024

• Fixing Errors in Linux Build (allenai#105)

  Show description for a74b78a
  soldni

  authored
  a74b78a

  Copy full SHA for a74b78a

• V1.0 candidate; new deduper options, new taggers (allenai#100)

  Show description for 2daae12
  

  3 peopleauthored
  2daae12

  Copy full SHA for 2daae12

Commits on Jan 21, 2024

• Fixed Dangling CLI Options; E2E Tokenizer Tests (allenai#103)

  Show description for a44489f
  soldni

  authored
  a44489f

  Copy full SHA for a44489f

Commits on Jan 20, 2024

• BOS/EOS/PAD options in tokens cli; speed up tokenization by segmenting paragraphs. (allenai#102)

  Show description for 45b5eea
  soldni

  authored
  45b5eea

  Copy full SHA for 45b5eea

Pagination

Previous

Next