Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode #259

ivababukova · 2023-07-11T08:08:36Z

Fixes #1228

Changes

A new file under vitessce/ containing a script that takes in project_name and output_dir, downloads all files and configurations for that project and creates an Anndata object out of them.
The script provides 2 ways to export the project data:
- 1. As an object saved locally and written to the Anndata-Zarr store. The saved object can be loaded with Vitessce zero config mode functionality.
- 2. Vitessce view config that can be loaded to Vitessce directly.

NOTE: For both cases, the user will need to use a local http server to load the files (issue #1278)
NOTE: For the second option, the user will need to add the correct URL to their file and also define coordinationValues and options.

Tested with the following projects:

Projects are taken from: https://github.com/ucscGenomeBrowser/cellbrowser-confs
They are available in https://cells.ucsc.edu, for example: https://cells.ucsc.edu/?ds=adultPancreas

Successfully processed:

adultPancreas (2542 number of cells) –- good
brain-dronc-seq (13067 number of cells) – good
adult-retina (19694 number of cells) – good
chporg (32464 number of cells) – good, but have to wait about 20 mins because the dataset is big
dros-brain+normal (4407 number of cells) – good
dros-brain+merge (8696 number of cells) – good

Successfully processed, but there are no cell set colours:

dental-cells+human-adult-molars (41673) – no cell colours and no heatmap
cortex-dev — cell colours are not appearing (no suitable fields in obs)
cross-tissue-maps+immune (14156 number of cells) - cell colours are not appearing (no suitable fields in obs)
covid19-smoking (19361 number of cells) - cell colours are not appearing. There are Cluster and Cell Type fields. It is strange that it is not working.
dros-olfac (3833 number of cells) - cell colours not appearing because obs doesn’t contain supported props
dental-cells+mouse-incisor (2888 number of cells) – cell set colours don’t show up (no valid obs prop)
chi-10x-mouse-cardiomyocytes (9072 number of cells) – the colours of the cells are not appearing (Cluster is in obs and in Vitessce view config, but colors still not showing up)
covid19-cellular-targets+kidney (33872 number of cells) – the cellSet colors are not showing up: there are no obsSets under options, but even if I add them (adding cellId or donor), the app crashes and doesn’t work. Requires 10+ mins wait time to process
cardiac-differentiation+trajectory+cm-combined-trajectory (35902) - no cell colours and no heatmap

Unsuccessful, because it took too long to load the matrix file:

covid19-periph-immuno (44721 number of cells) - couldn’t process it after 9 mins of waiting
covid19-immuno+b-cells (11377 number of cells) — took more than 30 mins to process
covid19-bronch-epi — dataset is really big (expr matrix unzipped is 3.86 GB). Couldn’t process it after half an hour of waiting

NOTE: I only ran the script for smaller datasets. For datasets with more than 40 000 cells, loading the expression matrix takes too long time (more than half an hour).

tests/data/smaller_expr_matrix.tsv.gz

vitessce/config_converter.py

keller-mark · 2023-07-12T13:29:03Z

vitessce/config_converter.py

+                print(f"obsm {key} is an instance of DataFrame, converting it to numpy array.")
+                self.adata.obsm[key] = self.adata.obsm[key].to_numpy()
+
+        self.adata = optimize_adata(


Suggested change

self.adata = optimize_adata(

return optimize_adata(

To make things simpler and easier to test, I would not worry about writing to the Zarr format in this converter, and instead would return the AnnData object

Do you suggest that we delete the following lines:

os.makedirs(os.path.dirname(data_dir), exist_ok=True) self.adata.write_zarr(zarr_filepath, chunks=[self.adata.shape[0], VAR_CHUNK_SIZE])

If we don't write things to the Zarr format and just return the adata object, written to the store, then how will the zero config mode functionality in the Vitessce website work? How will it be able to pick up the local Anndata-Zarr object and generate the view config? Or am I misunderstanding your suggestion?

vitessce/config_converter.py

keller-mark · 2023-07-12T13:49:25Z

Can we add an example notebook in docs/notebooks (and linked from https://github.com/vitessce/vitessce-python/blob/main/docs/widget_examples.rst so that it gets included in the documentation website)

keller-mark · 2023-07-12T13:49:57Z

Looking good, and thanks for all of the tests! A few minor comments

…. Added jupyter notebooks

ivababukova · 2023-07-17T09:07:53Z

Can we add an example notebook in docs/notebooks (and linked from https://github.com/vitessce/vitessce-python/blob/main/docs/widget_examples.rst so that it gets included in the documentation website)

That is done now

ivababukova · 2023-07-17T09:08:37Z

docs/notebooks/environment.yml

+ - pandas>=1.5.3
+ - anndata==0.8.0


these versions were incompatible and the new scripts were erroring out on the step where I call optimize_adata

ivababukova · 2023-07-17T09:10:29Z

docs/notebooks/environment.yml

 - numba>=0.53.0
 - scanpy>=1.6.0
 - jupyterlab>=3
 - zarr>=2.5.0
 - boto3>=1.16.30
+ - starlette==0.30.0


this dependency is needed when we call anndata_wrapper_inst.auto_view_config(vc), part of the convert_cellbrowser_project_to_vitessce_config function. I upgraded straight to 0.30, instead of 0.14 (as it is in the vitessce-python-dev environment) to avoid having to install 'aiofiles>=0.6.0'

I think we need to address this first #254

ivababukova · 2023-07-17T09:11:01Z

pyproject.toml

  'ome-zarr==0.2.1',
  'tifffile>=2020.10.1',
+  'jsonschema>=3.2'


this is used for the validator of the CellBrowser config

ivababukova · 2023-07-17T09:11:39Z

vitessce/data_utils/entities.py

@@ -38,7 +38,7 @@ def add_mapping(self, name, coords):
        if len(coords) != len(self._cell_ids):
            raise ArgumentLengthDoesNotMatchCellIdsException(
                'Coordinates length does not match Cell IDs Length')
-        if type(name) != str:
+        if not isinstance(name, str):


auto corrected by the linter

ivababukova added 5 commits July 3, 2023 15:14

Added a script to convert Cellbrowser config to Vitessce

bdc85ad

Clean up code and print statements

706f6bb

add file

100a33c

Adding scanpy to dependencies

2fd8334

Add unit tests

c938e90

ivababukova changed the title ~~Create a script that converts CellBrowser config to Vitessce view config~~ Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode Jul 11, 2023

ivababukova added 3 commits July 12, 2023 09:39

didn't commit the test file before

599ab6b

100% test coverage and fix linting errors

7e1181b

fix linting

d5efa57

ivababukova commented Jul 12, 2023

View reviewed changes

tests/data/smaller_expr_matrix.tsv.gz Outdated Show resolved Hide resolved

ivababukova added 3 commits July 12, 2023 13:03

Reorganise files in unit tests

29993ee

Improve the class interface and add comments on what each function does

5c6874e

Add a functionality to obtain vitessce view config directly

126ab7b

keller-mark reviewed Jul 12, 2023

View reviewed changes

vitessce/config_converter.py Outdated Show resolved Hide resolved

ivababukova added 5 commits July 13, 2023 08:42

unit tests for the new function

0b4e6e2

Support special cases of config

2d2dbc5

Fix of naming convention that - should be translated to Minus

21b3a21

Add capability to run script starting with preexisting anndata objest…

349de82

…. Added jupyter notebooks

forgot to commit the jupyter notebook in prev commit

bfcf87c

ivababukova commented Jul 17, 2023

View reviewed changes

ivababukova added 2 commits July 17, 2023 11:50

fix error

c6b61d7

fix problem

e08c982

ivababukova merged commit b5c5795 into main Jul 18, 2023
6 of 10 checks passed

ivababukova deleted the ivababukova/issue-1228 branch July 18, 2023 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode #259

Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode #259

ivababukova commented Jul 11, 2023 •

edited

Loading

keller-mark Jul 12, 2023

ivababukova Jul 13, 2023

keller-mark commented Jul 12, 2023

keller-mark commented Jul 12, 2023

ivababukova commented Jul 17, 2023

ivababukova Jul 17, 2023

ivababukova Jul 17, 2023

keller-mark Jul 18, 2023

ivababukova Jul 17, 2023

ivababukova Jul 17, 2023

Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode #259

Create a script that converts CellBrowser config to Anndata-Zarr file which is digestable by Vitessce zero config mode #259

Conversation

ivababukova commented Jul 11, 2023 • edited Loading

Tested with the following projects:

Successfully processed:

Successfully processed, but there are no cell set colours:

Unsuccessful, because it took too long to load the matrix file:

keller-mark Jul 12, 2023

Choose a reason for hiding this comment

ivababukova Jul 13, 2023

Choose a reason for hiding this comment

keller-mark commented Jul 12, 2023

keller-mark commented Jul 12, 2023

ivababukova commented Jul 17, 2023

ivababukova Jul 17, 2023

Choose a reason for hiding this comment

ivababukova Jul 17, 2023

Choose a reason for hiding this comment

keller-mark Jul 18, 2023

Choose a reason for hiding this comment

ivababukova Jul 17, 2023

Choose a reason for hiding this comment

ivababukova Jul 17, 2023

Choose a reason for hiding this comment

ivababukova commented Jul 11, 2023 •

edited

Loading