'Unspecified' fields on browser

[pnejad]

We noticed a lot of 'Unspecified' fields in the browser in places that had information previously.

For example:

Organ, organ_part, and disease are showing 'Unspecified' for the Human Hematopoietic Profiling dataset.

Tagging @HumanCellAtlas/wranglers to list other issues we noticed.

[theathorn]

@hannes-ucsc @NoopDog Is this a metadata issue, indexing issue or data browser issue?

[zperova]

@pnejad @hannes-ucsc @NoopDog @theathorn
This dataset uses cell lines so there is a model_organ which is displayed correctly in the Explore Data view (where all the projects are listed).
I bet the other datasets that also have Unspecified in the Organ and Organ parts are those with cell lines or organoids. We introduced the Model organ because this is a correct way to represent the data however we have not discussed what should be displayed in the Explore Project view for such datasets. I think this needs a different ticket - calling for a correct and harmonized representation of such datasets. Displaying the actual values for Organ and Organ parts that are in the metadata is not a sensible solution. Adding Model organ to the Explore Project view would help. I found related ticket: #805

@hannes-ucsc could you please comment on what is the algorithm used by the Data Browser to populate Organ, Organ parts and Known diseases (Specimen) in the Explore Project view and in the Explore Data view?

More related tickets for reference here:
#338
#302

[pnejad]

@zperova Good catch, I think that is why it does not appear in the main table!

The lab did purchase a cell line, but they did not differentiate them. So the specimen organ/organ_part is the same as the model_organ/model_organ_part.

Here is how it currently looks in the browser:

[hannes-ucsc]

@hannes-ucsc could you please comment on what is the algorithm used by the Data Browser to populate Organ, Organ parts and Known diseases (Specimen) in the Explore Project view and in the Explore Data view?

Known diseases (Specimen) is populated from specimen_from_organism

Azul also exposes the donor diseases from donor_organism.disease but the Data Browser does not display that. @NoopDog should be able to clarify as to if this was intentional and if so what the intention is.

Organ is populated from the sample.organ field. A sample is the biomaterial from which the cell suspension was created. It can be a specimen_from_organism, organoid or cell_line. The problem is that only specimen_from_organism actually has a organ property, so for the other two types, the result is Unspecified.

Filtering by sample type organoid or cell_line illustrates the problem well.

https://data.humancellatlas.org/explore/projects?filter=%5B%7B%22facetName%22%3A%22sampleEntityType%22%2C%22terms%22%3A%5B%22cell_lines%22%2C%22organoids%22%5D%7D%5D

Only one of the projects has a value other than Unspecified for Organ and that's because that project has samples that are specimens, meaning that some cell suspensions in that project were derived from specimens directly while other cell suspensions were derived from cell lines that were derived from specimens. We see the organ of the sample specimens but we don't see the organ of specimens that aren't samples.

TL;DR: The DB seems to compute the values in Organ / Model Organ column as in (samples[].organ or "Unspecified") + "/" + (samples[].modelOrgan or "Unspecified". One resolution would be to make that (specimens[].organ or "Unspecified") + "/" + (samples[].modelOrgan or "Unspecified").

Organ Part is handled identically to organ. Everything I said about organ applies to organ part.

You can stop reading now because I think the above solution will work except for freaky graphs: That solution does introduce the possibility of including the organ of specimens that aren't connected to a cell suspension. If that turns out to be a problem the solution in Azul would be to propagate specimen.organ into the connected sample. That would mean that organoids and cell suspension would have both organ and modelOrgan properties in the response. We've stopped pushing properties down like that because it created problems with aggregation by organ. Another solution would be to disregard specimens that aren't connected to a sample.

[theathorn]

Discuss with @hannes-ucsc

[NoopDog]

Azul also exposes the donor diseases from donor_organism.disease but the Data Browser does not display that. @NoopDog should be able to clarify as to if this was intentional and if so what the intention is.

Correct, the donor_organism.disease is not currently displayed. We can display but this just needs to be clearly distinguished from the sample disease. (the donor may have a disease but the organ the sample was taken from was "normal")

[NoopDog]

Regarding:

TL;DR: The DB seems to compute the values in Organ / Model Organ column as in (samples[].organ or "Unspecified") + "/" + (samples[].modelOrgan or "Unspecified". One resolution would be to make that (specimens[].organ or "Unspecified") + "/" + (samples[].modelOrgan or "Unspecified").

We have validated the Data Browser is working as originally designed. I need to look a bit deeper into the TLDR to see which solution makes sense to me. Stay tuned.