Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Sep 24, 2024

• Also pin maturin in action (allenai#208)

  undfined

  authored
  c029e94

  Copy full SHA for c029e94

• Pin maturin in CI (allenai#207)

  undfined

  authored
  35379cc

  Copy full SHA for 35379cc

• Bump version with postfix for PyPI (allenai#206)

  undfined

  authored
  333d204

  Copy full SHA for 333d204

• Adds dclm fasttext classifier (allenai#205)

  Show description for 06d521d
  undfined

  authored
  06d521d

  Copy full SHA for 06d521d

Commits on Sep 20, 2024

• Undfined/runner v3 (allenai#204)

  Show description for 8a3c058
  undfined

  authored
  8a3c058

  Copy full SHA for 8a3c058

• Revert upload/download to v3 for now (allenai#203)

  Show description for 6efd592
  

  undfined

  and

  soldni

  authored
  6efd592

  Copy full SHA for 6efd592

• [Json fooramt error in line 133] Update getting-started.md (allenai#196)

  Show description for 4c2e54b
  

  yushengsu-thu

  and

  soldni

  authored
  4c2e54b

  Copy full SHA for 4c2e54b

• Dependabot fail (allenai#202)

  undfined

  authored
  f4057bd

  Copy full SHA for f4057bd

• Polymorphic span replacement (allenai#200)

  Show description for 2436a6c
  undfined

  authored
  2436a6c

  Copy full SHA for 2436a6c

Commits on Sep 4, 2024

• Update deduper.rs (allenai#197)

  soldni

  authored
  621a6f4

  Copy full SHA for 621a6f4

Commits on Sep 3, 2024

• Bump actions/download-artifact (allenai#198)

  Show description for 1f72ce3
  dependabot[bot]

  authored
  1f72ce3

  Copy full SHA for 1f72ce3

Commits on Aug 28, 2024

• force numpy to be below v2 (allenai#195)

  soldni

  authored
  2a4ef70

  Copy full SHA for 2a4ef70

• Bump nltk from 3.8.1 to 3.9 in the pip group across 1 directory (allenai#187)

  Show description for 59fe87c
  

  dependabot[bot]

  and

  soldni

  authored
  59fe87c

  Copy full SHA for 59fe87c

• Update getting-started.md (allenai#193)

  Show description for 3b86772
  yushengsu-thu

  authored
  3b86772

  Copy full SHA for 3b86772

Commits on Aug 26, 2024

• tokenizers (allenai#189)

  soldni

  authored
  6cb88c0

  Copy full SHA for 6cb88c0

Commits on Aug 22, 2024

• Changed entrypoint to increase IntelliJ compatibility (allenai#188)

  soldni

  authored
  16452bb

  Copy full SHA for 16452bb

Commits on Aug 17, 2024

• Count Bytes and Docs (allenai#186)

  Show description for f365470
  soldni

  authored
  f365470

  Copy full SHA for f365470

Commits on Aug 16, 2024

• Merge pull request allenai#185 from allenai/epwalsh/install-patch

  Show description for 4cf2d17
  epwalsh

  authored
  4cf2d17

  Copy full SHA for 4cf2d17

Commits on Aug 15, 2024

• Fix local installation on MacOS

  epwalsh

  committed
  fae2e2e

  Copy full SHA for fae2e2e

Commits on Aug 11, 2024

• fixed tests to match new mixer (allenai#184)

  soldni

  authored
  c4fa395

  Copy full SHA for c4fa395

Commits on Aug 10, 2024

• fixed empty attributes bug

  soldni

  committed
  8ab1d1d

  Copy full SHA for 8ab1d1d

Commits on Aug 6, 2024

• Always use inferred extension (allenai#183)

  Show description for 4322ad0
  undfined

  authored
  4322ad0

  Copy full SHA for 4322ad0

• Bump to 1.0.7 (allenai#182)

  undfined

  authored
  abb67f7

  Copy full SHA for abb67f7

• Bump openssl from 0.10.64 to 0.10.66 in the cargo group (allenai#178)

  Show description for 2e7ade6
  

  dependabot[bot]

  and

  soldni

  authored
  2e7ade6

  Copy full SHA for 2e7ade6

• Better Filters Error Handling (allenai#171)

  Show description for b406546
  soldni

  authored
  b406546

  Copy full SHA for b406546

Commits on Aug 1, 2024

• V2 of Gopher tagger (allenai#181)

  Show description for a72c76b
  soldni

  authored
  a72c76b

  Copy full SHA for a72c76b

Commits on Jul 26, 2024

• Cherry pick zstd compressor (allenai#180)

  Show description for a01a222
  undfined

  authored
  a01a222

  Copy full SHA for a01a222

Commits on Jul 24, 2024

• Version bump for new release (1.0.4) (allenai#179)

  Show description for 5868453
  soldni

  authored
  5868453

  Copy full SHA for 5868453

Commits on Jul 13, 2024

• added option for tokenizer to split on special tokens (allenai#176)

  soldni

  authored
  ac91597

  Copy full SHA for ac91597

Commits on Jul 6, 2024

• Adding partition logic (allenai#161)

  Show description for 7318c44
  

  Whattabatt

  and

  soldni

  authored
  7318c44

  Copy full SHA for 7318c44

Commits on Jun 6, 2024

• Workaround to fix memory leak in HuggingFace tokenizer (allenai#169)

  Show description for 64886d9
  soldni

  authored
  64886d9

  Copy full SHA for 64886d9

• Adds ZST support in Deduper and Mixer (allenai#170)

  Show description for bf80be0
  soldni

  authored
  bf80be0

  Copy full SHA for bf80be0

Commits on May 21, 2024

• Adding Quality Classifier from Dolma 1.7 (allenai#163)

  Show description for a9eaf29
  soldni

  authored
  a9eaf29

  Copy full SHA for a9eaf29

Commits on May 8, 2024

• Bump rustls in the cargo group across 1 directory (allenai#155)

  Show description for 09eff3d
  

  dependabot[bot]

  and

  soldni

  authored
  09eff3d

  Copy full SHA for 09eff3d

Commits on May 6, 2024

• Merge pull request allenai#156 from allenai/optimization

  Show description for 2dd256b
  Whattabatt

  authored
  2dd256b

  Copy full SHA for 2dd256b

Pagination

Previous

Next