update sample sheet generation to handle project shortname

[tanaes]

We're using the sample_proj field in the Sample Sheets as a unique reference for demultiplexing to assign demuxed fastq files automatically to their project folder.

We need to update the sample sheet generation to read in a per-sample value:
https://github.com/jdereus/labman/blob/0b36cfb63e8222f078f2479c9651e225864fb9c1/labman/db/process.py#L2577

And we may also need to add a column to the projects table to allow a project shortname in a format like 'PI_qiitaID' or similar.

[AmandaBirmingham]

@tanaes Some clarification questions:

1. You note "we may also need to add a column to the projects table", but I can't find a projects table in the labman/qiita db! Are we perchance talking about the study table? (I ask because I see that the fix for issue Plates List screen (Project names) #32, Plates List screen (Project names), got the "project names" it displays from the qiita.study.title value). If not, has a project table been added at some point but didn't make it into the labman db patch scripts? Or does it have a name other than projects?
2. If we are talking about the qiita.study table, is the project shortname already in there? For example, is it the study_id field or the study_alias field?
3. Right now the sample_proj parameter to _format_sample_sheet_data is being set to the sequencing process's run name. After we change sample_proj to instead hold the per-sample project shortname, does the sequencing process's run name need to be included anywhere (else) in the sample sheet output?

[tanaes]

I don't think that this is a field that exists in Qiita yet necessarily. CC @jdereus — the thing is we need something that can link up to a consistent folder name on Barnacle. We've had some discussions about how best to do this but I don't know that we've formally adopted a consensus yet!

So, 1 & 2: Correct, it doesn't exist yet in the Qiita db; and 3: I think it would be good to add an additional column to the sample name titled "Project_Name" that includes the long name while the protected column "Sample_Project" contains the short name.

[jdereus]

We need to be very careful on the definition of "consistent folder name" and how it applies to a sample sheet. Reading some of this, it sounds like somehow we are expect to know the run folder information prior to being able to generate the sample sheet. I am not aware of a project short name. other than what we add in to the sample sheet. This should be something along the lines of <PI>_<qiita_ID>. That will allow for per project, per plate data aggregation when we do the bcl conversion. Please correct me if I am misunderstanding the issue here. JD From: Jon Sanders <notifications@github.com<mailto:notifications@github.com>> Reply-To: jdereus/labman <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Tuesday, March 13, 2018 at 12:56 PM To: jdereus/labman <labman@noreply.github.com<mailto:labman@noreply.github.com>> Cc: "Dereus, Jeff" <jdereus@ucsd.edu<mailto:jdereus@ucsd.edu>>, Mention <mention@noreply.github.com<mailto:mention@noreply.github.com>> Subject: Re: [jdereus/labman] update sample sheet generation to handle project shortname (#204) I don't think that this is a field that exists in Qiita yet necessarily. CC @jdereus<https://github.com/jdereus> - the thing is we need something that can link up to a consistent folder name on Barnacle. We've had some discussions about how best to do this but I don't know that we've formally adopted a consensus yet! So, 1 & 2: Correct, it doesn't exist yet in the Qiita db; and 3: I think it would be good to add an additional column to the sample name titled "Project_Name" that includes the long name while the protected column "Sample_Project" contains the short name. - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#204 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ANq27eeIAzcpfhBHdWi60_0AZo57WP9pks5teDKEgaJpZM4SMpYh>.

[AmandaBirmingham]

@tanaes @jdereus Here are the conclusions we reached in our discussion today; please correct anything I've gotten wrong:

• A project shortname will be generated programmatically from the study information for each sample; Labman users will not need to enter (or even know) the shortname.
• Project shortname format will be either <PI_name>_<study_ID> or <Lab_person_name>_<PI_name>_<study_ID> ; Amanda will generate samples for some existing studies in each of these formats and run them by Greg for a decision on which is more usable.
• The project shortname will NOT be stored in the database anywhere. Instead, the gold-standard repository of the project shortname of any sample is the sample sheet for that sample's sequencing run. These sample sheets are expected to be saved indefinitely; as long as this is the case, it will always be possible to go back and find the project shortname used for a sample (even if the rule used to generate project shortnames from the database has changed since that sample was sequenced).

[AmandaBirmingham]

@tanaes @jdereus As I look at our conclusions, I find I have one concern: are samples from any study ever sequenced more than once? Like, maybe if the original sequencing run produced bad data?? If so, the naming convention we laid out above will not keep the sequencing results of the multiple runs separate.

In the approach we chose, every sequencing run for a given study X will have the same project shortname, which could result in data from multiple runs being aggregated together into the same barnacle folder after demuxing (if folders persist between runs). Depending on the how smart the naming conventions used in the demuxing code are (of which I know nothing), it is even possible that old demuxed data might be overwritten with new demuxed data.

Do these possibilities concern you? If so, how would you like to proceed?

[tanaes]

@AmandaBirmingham @jdereus and I when discussing this system envisioned that within the output project folder, each run would get it's own run folder, retaining the Illumina-encoded run folder information (which includes things like the sequencing instrument serial number and flow cell serial number) so that we'd be able to maintain this per-run information, so I think it's ok to not have the sample names themselves be globally unique.

[jdereus]

I think that depends on how the data is pulled back. If, as I believe, the samples are pulled back based off of individual wells on specific plates, then it should not be an issue. But if there is some check somewhere that checks for uniqueness, then we might have an issue. I also think we might be discussing two different things here. Output location after bcl conversion and sample naming on _any_ plate and well throughout entire system. From: Jon Sanders <notifications@github.com<mailto:notifications@github.com>> Reply-To: jdereus/labman <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Wednesday, March 21, 2018 at 11:59 AM To: jdereus/labman <labman@noreply.github.com<mailto:labman@noreply.github.com>> Cc: "Dereus, Jeff" <jdereus@ucsd.edu<mailto:jdereus@ucsd.edu>>, Mention <mention@noreply.github.com<mailto:mention@noreply.github.com>> Subject: Re: [jdereus/labman] update sample sheet generation to handle project shortname (#204) @AmandaBirmingham<https://github.com/amandabirmingham>@jdereus<https://github.com/jdereus> and I when discussing this system envisioned that within the output project folder, each run would get it's own run folder, retaining the Illumina-encoded run folder information (which includes things like the sequencing instrument serial number and flow cell serial number) so that we'd be able to maintain this per-run information, so I think it's ok to not have the sample names themselves be globally unique. - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#204 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ANq27Tb6Pex8DJ4-jlhIy4SHCvjh_VSSks5tgrEHgaJpZM4SMpYh>.