Beta testers program todo

[EiffL]

• Repackage in a clean way the tokenizers (comments from @LTMeyer)
• Include somehow the tokenizers with AION, to make going from data to tokens very easy
• Implement a couple of nicely explained and detailed notebook (@al-jshen will give it a go)
• Test out weight upload behind HF wall

✅ Codec Checklist

• tok_image
• catalog
• tok_segmap
• tok_ebv
• tok_flux_g
• tok_flux_r
• tok_flux_i
• tok_flux_z
• tok_flux_w1
• tok_flux_w2
• tok_flux_w3
• tok_flux_w4
• tok_shape_e1
• tok_shape_e2
• tok_shape_r
• tok_image_hsc
• tok_mag_g
• tok_mag_r
• tok_mag_i
• tok_mag_z
• tok_mag_y
• tok_a_g
• tok_a_r
• tok_a_i
• tok_a_z
• tok_a_y
• tok_shape11
• tok_shape22
• tok_shape12
• tok_spectrum_sdss
• tok_z
• tok_spectrum_desi
• tok_xp_bp
• tok_xp_rp
• tok_flux_g_gaia
• tok_flux_bp_gaia
• tok_flux_rp_gaia
• tok_parallax
• tok_ra
• tok_dec

[LTMeyer]

Repackage in a clean way the tokenizers

Context

Because at the time of development of the tokenizers, we were making many concurrent updates we designed a solution to store the tokenizer model with all their dependencies. At that time it was a good solution, but now it prevents us from loading the tokenizer simply. They require many dependencies that could be remove. Currently tokenizers are loaded through the following custom-made function.

from torch import package

importer = package.PackageImporter(tokenizer_path)
tokenizer = importer.load_pickle("network", "network.pkl", map_location="cpu")

Instead we would like to have tokenizer as standalone class and load the weights in a traditional fashion.

tokenizer = MyTokenizerModule(...)  # Instantiate the model
tokenizer.load_state_dict(...)  # Load the weights

If we have such implementation, the next step will be supporting hugging face and have other simple ways to load the model.

tokenizer = MyTokenizerModule.from_pretrained(...)  # Instantiate the model from HF hosted weights

How to Proceed

We must rewrite the tokenizer models as standalone classes.
We can retrieve the original code that was stored with inspect.

from torch import package

importer = package.PackageImporter(tokenizer_path)
tokenizer = importer.load_pickle("network", "network.pkl", map_location="cpu")
tokenizer_code = inspect.getsource(tokenizer.__class__)

Questions

Can we list the tokenizers we need to support and give the path to where we stored them @EiffL ?

[LTMeyer]

The list of tokenizers is given here https://github.com/PolymathicAI/AION_ICML/blob/main/scripts/tokenizers.yaml

[EiffL]

Alright, only 2 tokenizers left, but these are the hardest ones...

[EiffL]

Done :-)