not applicable in multiplex experiments with pooled and reference samples.

[ypriverol]

In multiplex experiments, the Reference/Pooled, whatever we want to call the sample, will need to provide age, sex, etc, for some of the sample properties. In the current specification/validation, all those experiments are failing because, for example, age can't be not applicable. I don't know if it is better to do the following:

• Highlight that the sample in some way is a reference, pooled, etc, then validation applies differently to it.
• Relax the rule and make clear how this needs to be written for multiplex reference samples

@noatgnu this is one of the main reasons a lot of experiments are failing with the new rules and validators. @bigbio/sdrf-community ideas?

[ypriverol]

Related with issue #748

[noatgnu]

For the reference pool or row with SN= in pooled sample column, we will need custom validation of them. The way they should be treated is as if they are a separate abstract of the samples contributed into the pools that we can see from SN= value. When validating, the validator should split the sdrf table into the reference pool and the normal sample. Normal samples go through typical validation rule in the yml. Then maybe we have an additional collection of rule in yml specifically for validating reference portions where columns can have different rules from normal samples.

This would make it possible to reuse a lot of the current validator and schema. The only difference is that on read, the sdrfdataframe object would store two dfs in case we can call them sdrf.df and sdrf.reference_pooled_df . these would be validated differently. The rule may be in a single combined yml or it would be two different yml where the validator can call "minimum" for normal sample and "minimum_pooled" if it see sdrf.reference_pooled_df is not empty. this would make it possible to have this be changed on the backend instead of any significant spec changes. We can update a bit of the text for more guidance on how the row with SN value would be treated.

Ideally we should have 3 dfs attached to the object, the full table, the normal sample table and the reference pooled table. the full table is important for che king uniqueness of the whole table while while the other two would follow the logic above.

[ypriverol]

In the current specification, we do not require writing pooled or reference samples in a way that predicts SN=?.. In fact, the reference and pooled samples I have seen are usually written with their respective names; most fields list pooled and not applicable or not available.

The original idea was to write all the contained subsamples for each pooled sample, but I now find it quite complex to annotate manually and not very helpful or useful for extracting information.

We can explicitly require that reference and pooled samples have a structured format and then implement your approach @noatgnu , making it clear that, for example, age and other fields are not applicable for pooled or reference samples.

[noatgnu]

I think my approach does have a few cons as it would no longer just be pooled or not pooled values. Idealy the sample source name they are refering to in the SN= value of the reference row should also be in the table with the original characteristics of that biological sample. I can see that currently in the spec it does not mention that the sample included in SN= have to be in the table. We will need to update the guideline around this to clarify these.

This would be a major change though and may need to redo lot of current sdrf. So this is where implementation of sdrf schema versioning and spec versioning maybe needed.

[TineClaeys]

If feasible to tweak the validator, I think the flagging approach will work best and be more sustainable on the long term.

[ypriverol]

I think, we should try the following approach:

• pooled sample: as a reserved word, in the similar way than not available and not applicable. Pooled sample is an ontology based word: https://www.ebi.ac.uk/ols4/ontologies/ncit/classes/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C165587
• For pooled sample samples, if the pooled samples share the same characteristics, then use only one; if not, then use pooled sample.
• In the pooled sample column the user can add the samples as SN={};SN={}... as we are suggesting now, this will be only possible for tools annotators.

[levitsky]

From the Zoom discussion, I think we agreed that the pooled sample column should be required. I think if we don't require the list of samples there, we should specify what can be put there instead.

Given that there will always be this column, I don't see the necessity for having pooled sample as a special value in other columns. I think we can just allow not applicable in pooled samples and it should not be hard to validate. In my opinion pooled samples are only one possible exception to allow not applicable, and this value doesn't really add information, because the validator can already see that the sample is pooled. However I don't see serious problems with it either, other than creating another reason for all existing SDRF files with pooled samples to be invalid according to the new specification (but we will probably have a lot of those anyway).

[noatgnu]

If someone is writing the file manually, writing a very long cell if you have complex source name could be a big problem and potentially a large source of errors. So I think I would support both. Columns with the SN= array in pooled sample would automatically be understood as pooled and it would disregard of most of the data in the characteristics cells. The validator should be updated to scan first for these rows and validate the content of this cell. If the validator detects a "pooled sample" value in one of the cell then it would assume that that row is a pooled row and disregard of most of the characteristics column.