Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

EleutherAI / clt-training Public

forked from EleutherAI/sparsify

Notifications You must be signed in to change notification settings
Fork 5
Star 18

Code
Pull requests 2
Actions
Projects
Security
Insights

Additional navigation options

Code
Pull requests
Actions
Projects
Security
Insights

Matryoshka#3

Open

zgoel317 wants to merge 70 commits intoEleutherAI:cltEleutherAI/clt-training:cltfrom

minlu21:matryoshkaminlu21/eai-sparsify:matryoshkaCopy head branch name to clipboard

Conversation Commits70 (70)Checks Files changed

Open

Matryoshka#3
zgoel317 wants to merge 70 commits intoEleutherAI:cltfrom
minlu21:matryoshka

Commits

Commits on Jul 2, 2025

added set up help and modified package dependency list

minlu21
authored and
zgoel317
committed
file was looking for cfg.json for mlp, changed to config.json since that is what is used in most recent eleuther-AI transcoder upload for llama
zgoel317
committed
torch >= 2.7.0
zgoel317
committed
goofy
zgoel317
committed
added matryoshka runner class to runner.py, changed config py to accomodate matryoshka args
zgoel317
committed
forgot bias slicing oopsies
zgoel317
committed
spelling error boooo
zgoel317
committed
more spelling errors :(
zgoel317
committed
we were trying to assign tensor slice directly to a parameter, which was causing a type error, so we now clone the original parameter data, temporarily modify it, then restore original data.
zgoel317
committed
hopefully fixed dimension issues with bias slicing
zgoel317
committed
the parameter replacement approach broke triton
zgoel317
committed
the parameter replacement approach broke triton
zgoel317
committed
many changes made
zgoel317
committed
spelling error
zgoel317
committed
annotations
zgoel317
committed
added debug output
zgoel317
committed
issues with slicing logic
zgoel317
committed
issues with slicing logic
zgoel317
committed
we have to select activations using topk or whatever before every single slice, not just across the whole encoder onece, also added debug statements for sparsecoder for educational reasons.
zgoel317
committed
issue with debug statements
zgoel317
committed
issue with pre_acts shape
zgoel317
committed
type error issue
zgoel317
committed
parameters incorrectly defined for decode with masking
zgoel317
committed
got rid of check that was defaulting to masking approach
zgoel317
committed
more dtype error
zgoel317
committed
gradient tracking to mid decoder for matryoshka
zgoel317
committed
more gradient tracking
zgoel317
committed
trying to fix gradient tracking issues
zgoel317
committed
changing the way batchtopk works in matryoshka to more closely align w crosslayer runner - ie we are applying batchtopk+topk to each slice
zgoel317
committed
changing the way batchtopk works in matryoshka to more closely align w crosslayer runner - ie we are applying batchtopk+topk to each slice
zgoel317
committed
we only use one mid_out for all slices instead of making a new mid_out for each slice
zgoel317
committed
create new middecoder copy for each slice, but they are all the same size
zgoel317
committed
original activations should match shape of the activations used in forward pass, not mid_out
zgoel317
committed
shape issues continued
zgoel317
committed
shape issues continued
zgoel317
committed
we create only one middecoder, single gradient tracking setup, but we still slice
zgoel317
committed
the only real difference between matryoshka and crosslayer runner is that we need the losses at different slices, but the way in which activations get chosen is identical to crosslayer
zgoel317
committed
the only real difference between matryoshka and crosslayer runner is that we need the losses at different slices, but the way in which activations get chosen is identical to crosslayer
zgoel317
committed
mid decoder copy for each slice ?
zgoel317
committed
we do cross layer coalescing only once for the largest slice, and then for the other slices we only carry out loss computations.
zgoel317
committed
we do cross layer coalescing only once for the largest slice, and then for the other slices we only carry out loss computations.
zgoel317
committed
added time related debug statements to identify bottleneck in matryoshka
zgoel317
committed
was the ghastly training time actually because of how often we were saving checkpoints?
zgoel317
committed
cleaned up most debug statements
zgoel317
committed
cleaned up most debug statements
zgoel317
committed
cleaned up most debug statements
zgoel317
committed
changed where pre_acts is calcualted so that it doesnt break the rest of my code
zgoel317
committed
moved loss per slice calculation to sparse_coder
zgoel317
committed
got rid of any dependencies on pre_acts, all computation happens in sparse_coder, coalescing only happens in the final slice, jsut like sparsecoder
zgoel317
committed
cosmetic changes
zgoel317
committed

Commits on Jul 8, 2025

scrapped everything - all matryoshka related code lives in matryoshkamiddecoder now
zgoel317
committed
careless error, matryoshka mid decoder wasnt reached
zgoel317
committed
changed some debug statementst
zgoel317
committed
when copy appplied to mid decoder it wasnt preserving matryoshka property.
zgoel317
committed
more debug statements
zgoel317
committed
merged with clt
zgoel317
committed
issues with calculating latent acts for matryoshka mid decoder
zgoel317
committed

Commits on Jul 16, 2025

merged from upstream
zgoel317
committed
got rid of the comparison to mid decoder debug statement bc it needs an extra forward pass
zgoel317
committed
moved logging to wanddb and fixed docstring issues
zgoel317
committed
debug for logging issues
zgoel317
committed
debug for logging issues
zgoel317
committed
debug for logging issues
zgoel317
committed
debug for logging issues
zgoel317
committed
no more debug statements
zgoel317
committed

Commits on Jul 23, 2025

Merge branch 'clt' into matryoshka
zgoel317
authored

Commits on Jul 29, 2025

rebase and added vectorized loss calculation
zgoel317
committed
rebase and added vectorized loss calculation
zgoel317
committed
rebase and added vectorized loss calculation
zgoel317
committed
removed remnant from previous matryoshka implementation
zgoel317
committed

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.