Commits

Branch selector

main

User selector

All users

Datepicker

All time

Commit history

Commits on May 19, 2025

• Add report for DDP

  RamonKaspar

  committed
  e257c2fView commit details

  Copy full SHA for e257c2f

  Browse repository at this point

• Update README

  RamonKaspar

  committed
  38f969bView commit details

  Copy full SHA for 38f969b

  Browse repository at this point

• Update Report Pretokenization with new plot path

  RamonKaspar

  committed
  f817873View commit details

  Copy full SHA for f817873

  Browse repository at this point

• DDP: Plot result for report

  RamonKaspar

  committed
  6ee6372View commit details

  Copy full SHA for 6ee6372

  Browse repository at this point

• DDP: Scripts and logs for running padding-free approaches (on-the-fly and pretokenized ones)

  RamonKaspar

  committed
  7d41ab2View commit details

  Copy full SHA for 7d41ab2

  Browse repository at this point

• Enhance IterableParquetDataset and IterablePreTokenizedDataset for DDP compatibility by implementing rank and world size initialization, sharding logic, and buffer management.

  RamonKaspar

  committed
  dc8d9cbView commit details

  Copy full SHA for dc8d9cb

  Browse repository at this point

• DDP: Log files for runing baseline padded and pretokenized padded (with world_size 4, 8, 12, and 16; batch_size=8 and batch_size=1)

  RamonKaspar

  committed
  65c8f56View commit details

  Copy full SHA for 65c8f56

  Browse repository at this point

• DDP: Use a smaller model so we can test larger batch sizes (e.g., batch_size=8)

  RamonKaspar

  committed
  d039538View commit details

  Copy full SHA for d039538

  Browse repository at this point

• Add scripts for distributed training (DDP) with padding and padding-free options

  RamonKaspar

  committed
  8dc0481View commit details

  Copy full SHA for 8dc0481

  Browse repository at this point

Commits on May 18, 2025

• Add DDP validation script

  RamonKaspar

  committed
  fcebe29View commit details

  Copy full SHA for fcebe29

  Browse repository at this point

• Feaure 2: Implement data distributed parallel (DDP)

  RamonKaspar

  committed
  14ad29bView commit details

  Copy full SHA for 14ad29b

  Browse repository at this point

• Move plots for pre-tokenization analysis to dedicated folder

  RamonKaspar

  committed
  3338409View commit details

  Copy full SHA for 3338409

  Browse repository at this point

• Update Repot

  RamonKaspar

  committed
  49749caView commit details

  Copy full SHA for 49749ca

  Browse repository at this point

• Update README

  RamonKaspar

  committed
  767cdbcView commit details

  Copy full SHA for 767cdbc

  Browse repository at this point

• Update README

  RamonKaspar

  committed
  1ed19e3View commit details

  Copy full SHA for 1ed19e3

  Browse repository at this point

• Update README

  RamonKaspar

  committed
  d243ad8View commit details

  Copy full SHA for d243ad8

  Browse repository at this point

Commits on May 7, 2025

• Rendered report to PDF (using EIsvogel template)

  RamonKaspar

  committed
  85e3902View commit details

  Copy full SHA for 85e3902

  Browse repository at this point

• Add report on pretokenization

  RamonKaspar

  committed
  7f6c6c3View commit details

  Copy full SHA for 7f6c6c3

  Browse repository at this point

• Refactor benchmark_datasets.py

  RamonKaspar

  committed
  500c761View commit details

  Copy full SHA for 500c761

  Browse repository at this point

• Update README

  RamonKaspar

  committed
  2ea4593View commit details

  Copy full SHA for 2ea4593

  Browse repository at this point

Commits on May 6, 2025

• Refactor PreTokenizedDataset and IterablePreTokenizedDataset for clarity

  RamonKaspar

  committed
  7642aa7View commit details

  Copy full SHA for 7642aa7

  Browse repository at this point

Commits on May 3, 2025

• Update README

  RamonKaspar

  committed
  6a7e730View commit details

  Copy full SHA for 6a7e730

  Browse repository at this point

• Remove plotting in main with actual plotting entry function (analyze_results.py)

  RamonKaspar

  committed
  04eb6c2View commit details

  Copy full SHA for 04eb6c2

  Browse repository at this point

• Enhance plotting functions: increase figure size, add batch size comparison plot

  RamonKaspar

  committed
  67a4d61View commit details

  Copy full SHA for 67a4d61

  Browse repository at this point

• Add tokenization benchmark script for performance evaluation of datasets (raw data processing speed)

  RamonKaspar

  committed
  e7086caView commit details

  Copy full SHA for e7086ca

  Browse repository at this point

• Run all 4 (2x baseline, 2x pretokenization) with batch_size=2 (NOTE: Larger is not possible due to GPU memory limits)

  RamonKaspar

  committed
  e91781aView commit details

  Copy full SHA for e91781a

  Browse repository at this point

• Script to verifiy the two pretokenizations (padded and loken-list)

  RamonKaspar

  committed
  55a6ed1View commit details

  Copy full SHA for 55a6ed1

  Browse repository at this point

• Use color palette which is more distinctive, split labels along multiple lines

  RamonKaspar

  committed
  d759ce4View commit details

  Copy full SHA for d759ce4

  Browse repository at this point

• Plot results between fair comparison: Pretokenization much worse!

  RamonKaspar

  committed
  c129a6aView commit details

  Copy full SHA for c129a6a

  Browse repository at this point

• Simple approach: Create fair comparison by comparing directly without any differences between on-the-fly tokenization and pretokenization. Result:

  Show description for 6567019
  RamonKaspar

  committed
  6567019View commit details

  Copy full SHA for 6567019

  Browse repository at this point

• Switch to snappy conversion

  RamonKaspar

  committed
  dee54e5View commit details

  Copy full SHA for dee54e5

  Browse repository at this point

• Remove packed version in preprcoessing script

  RamonKaspar

  committed
  dda1fddView commit details

  Copy full SHA for dda1fdd

  Browse repository at this point

• Remove not working packed version

  RamonKaspar

  committed
  518fa5fView commit details

  Copy full SHA for 518fa5f

  Browse repository at this point

• Run padded and token-list with snappy compression

  RamonKaspar

  committed
  ad03c2dView commit details

  Copy full SHA for ad03c2d

  Browse repository at this point

• Fix: Use snappy compression to match original compression algorithm, also with row_group_size=1000.

  RamonKaspar

  committed
  1f35a49View commit details

  Copy full SHA for 1f35a49

  Browse repository at this point