How to save retrained model

[AllenWT512]

Dear SCimilarity Developer,

I have retrained SCimilarity with our data but encountered some issues regarding model preservation. I downloaded the original model file but was unable to find instructions on how to properly save the annotation and cellsearch folders in the official tutorial.

Specifically, I’m referring to files such as:

labelled_kNN.bin in the annotation folder
full_KNN.bin, __Fragment_metadata.tdb, and a0.tdb in the cell_embedding folder
a0.tdb through a23_var.tdb in the cell_metadata folder
Could you please provide guidance on how to save these files effectively during the retraining process?

Thank you very much for your help!

[tony-kuo]

Hello. Those files aren't exactly training specific. Rather it uses the trained model to calculate embeddings on all training data and unlabeled data to build a knn used for cell annotation and cell search respectively. So you would need to have access to the original count data in order to embed it and then create the knn and metadata tables.

We cannot distribute the original counts due to technical limitations, though they are all public data and you could potentially download them all and organize it yourself. We have some scripts to aggregate and build knns and tiledb metadata tables but they are still being worked on for when we release a training tutorial.

If you are only interested in the data you retrained on and do not require the data we provided there are things you can do. Take a look at the build_knn function in CellAnnotation to get an idea of how to build the knn. For the metadata, you will need to provide the csv of the full metadata table. Take a look at the CellQuery constructor to see how that might work for you, though keep in mind you will need some specific column names and also a numpy arrary of embeddings.

We are still working on streamlining training and building these precomputed structures.

[AllenWT512]

Dear SCimilarity Developer,

Thanks for your detailed reply.

In the CellAnnotation class, the build_knn function is used to create the labelled_kNN.bin. However, this function needs to be called after the CellAnnotation class is instantiated, and the class instantiation requires a model path as a parameter. The issue arises because the model path provided for instantiation already contains the labelled_kNN.bin file.

The problem is that to create the labelled_kNN.bin, we must instantiate the CellAnnotation class, but the model path required for instantiation already includes the labelled_kNN.bin. This creates a logical conflict: we need to use build_knn function to generate the knn.bin file, but the model path we use for instantiation already contains this file.

Could you please provide guidance on how to solve this problem?

Thank you very much for your help!

[tony-kuo]

Hello. Assuming you have retrained the model, you would have saved it in a new model directory. Lets call that /opt/new_model. In a freshly trained model, there are model files:

decoder.ckpt  encoder.ckpt  gene_order.tsv  hyperparameters.json  label_ints.csv  layer_sizes.json  metadata.json

These are the only files needed to load a model for perform embedding.

When you construct a cell annotation knn, it will create a new subdirectory: /opt/new_model/annotation. Then the knn files

labelled_kNN.bin  reference_labels.tsv

will be saved in there.

So if you have retrained, I would hope you saved the model in a new directory. If not you will want to move it to a new directory.

[AllenWT512]

Dear SCimilarity Developer,
Thank you very much for your suggestions, which have brought my model reasoning process one step closer.
I successfully instantiated CellAnnotationl, however when I build_knn function, the error occurs:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[211], line 1
----> 1 ca.build_knn(input_data = zarr_list)

Cell In[206], line 130, in CellAnnotation.build_knn(self, input_data, knn_filename, celltype_labels_filename, obs_field, ef_construction, M, target_labels)
    126     if len(data.obs) == 0:
    127         continue
    129     embeddings_list.append(
--> 130         self.get_embeddings(align_dataset(data, self.gene_order).X)
    131     )
    132     labels.extend(data.obs[obs_field].tolist())
    133 embeddings = np.concatenate(embeddings_list)

File ~/miniconda3/lib/python3.10/site-packages/scimilarity/utils.py:87, in align_dataset(data, target_gene_order, keep_obsm, gene_overlap_threshold)
     85 # raise an error if not enough genes from target_gene_order exists
     86 if sum(data.var.index.isin(target_gene_order)) < gene_overlap_threshold:
---> 87     raise RuntimeError(
     88         f"Dataset incompatible: gene overlap less than {gene_overlap_threshold}. Check that var.index uses gene symbols."
     89     )
     91 # check if X is dense, convert to csr_matrix if so
     92 if isinstance(data.X, np.ndarray):

RuntimeError: Dataset incompatible: gene overlap less than 5000. Check that var.index uses gene symbols.

Which two dataset should be incompatible?
Could you please provide guidance on how to solve this problem?

Thank you very much for your help!

[tony-kuo]

Hello. There is a gene_order attribute in CellAnnotation that shows what genes the model was trained with. These are in gene symbols and not ensemble IDs. The error is saying it is not finding enough genes in the dataset you're using that overlap with the genes the model was trained on. Typically this is because your genes are in ensemble and need to be converted to symbol.

This is usually not an issue because the annotation knn is built with the dataset you trained on.

1. Did you train on a dataset using gene symbols?
2. How did you save the model, typically this saves the gene order when you properly save the model.

Generally, we train with pytorch-lightning as follows (in broad terms):

• Datamodule is zarr_data_models.MetricLearningDataModule
• Training module is training_models.MetricLearning
• Do training
• training module has a save_all method that saves all model and hyperparameter files, as well as the gene order.

[hilal-t]

Hello

I trained the model on my data.
After training, these files were generated:
decoder.ckpt encoder.ckpt gene_order.tsv hyperparameters.json label_ints.csv layer_sizes.json metadata.json

I built the knn and the content of annotation filled by labeled_knn.bin and reference_labels.tsv

It is not clear how to generate the files in cellsearch folder.
Any suggestion?

Thank you

[tony-kuo]

Hello. The cellsearch files generation is quite complicated unfortunately. We are still in the process of streamlining it.
The basic procedure is as follows:

You have a corpus of all labeled and unlabeled cells as anndatas (at the study level or sample level), for which you will need the counts (not log normed).

1. run scanpy QC to get QC annotations
2. For every anndata, align the gene space and generate embeddings (this is the reason you need counts)
3. Using the embedding and annotation knn, do cell type prediction
4. Aggregate all the metadata for every anndata, which requires some mandatory columns, the most import of which are "study", "sample", "disease", "tissue"
5. Write the aggregate metadata into tiledb
6. Aggregate all the embeddings and write into a tiledb
7. Using the embeddings, create a knn using hnswlib

At the moment we don't have public functions or scripts to perform this workflow, but we are working on it as part of a training tutorial.

[hilal-t]

OK. I appreciate the detailed reply.

[AllenWT512]

Dear SCimilarity Developer,

Thank you for your detailed feedback earlier.

An additional question is whether SCimilarity can be retrained on the CPU?

Thank you very much for your help!

[tony-kuo]

It could be trained on CPU, though I haven't tried and don't know how slow that will be.

[sojichld]

Hello, when training our own data, should the data be normalized or just aligned? should data be normalized when creating the cellsearch knn?

[tony-kuo]

Hello, the data should be log normalized. We usually use the scanpy method with a target sum of 10,000.
The cellsearch knn should be created on embeddings.