Mechanism for modifying metadata in (meta)analysis

[adswafford]

Putting an issue here to collect ideas and track.

I'd advocate still for being able to download, modify, upload, and update metadata to a specific analysis, leaving the original study metadata untouched. This will enable flexibility for meta-analysis and prevent users from the data-duplication model of downloading all the raw data for studies of interest and then uploading as a new merged study with fixed metadata.

[antgonza]

This is actually not difficult in principle as analysis already support multiple metadata files and we could simply add it, the issues (opportunities?) are:

1. should we allow this, in the base that any user can make any analysis public and there is no actual curation of the metadata
2. should we have a validator? what should it validate? That all samples match? Anything else?
3. once you upload the new mapping file, should all steps be redone? should it be a new analysis?
4. should we keep track of all of the new/old mapping files?

Perhaps a way to do this is that you can reuse a biom by creating a new analysis with your own mapping file, which can't be made public until admin revision - or with a big banner that says: different from metadata in original studies.

[adswafford]

I like the idea of allowing users to reuse a biom with their own mapping file and a warning banner that says the metadata is different from the original study. Or to Jose's point, we could only allow them to add additional columns to the metadata (though ideally not with an in-Qiita GUI) and then append/prepend the added metadata categories with something to flag that they are not present in the original study, e.g. "custom_" or "_12345" (where 12345 = the analysis ID).

For the specific issues:

1. with 1 or more of the above safety mechs, I think it's fine for it to be made public
2. Just all samples matching and no db violations should be fine right?
3. this is a good point. it would be great if there was an option for users to choose to rerun all of the analyses with the new metadata, but perhaps the new analysis should start branching from a fresh node so that the older analyses are maintained? i'm wondering if the older analyses should be marked read-only once the metadata is updated?
4. yes, but just using the same tracking we now use for metadata and prep files, unless we need to hang onto them in some other way to prevent the older analyses from breaking.

[antgonza]

Thanks. Perhaps the less intrusive and "easiest" implementation will be for the user to generate/copy an analysis with their own mapping file, with these steps:

• The user selects an analysis and clicks to reprocess with own mapping file.
• The mapping file is uploaded, validated (only sample names), and a new analysis is created - the user has the option to rerun all steps from the copied analysis. Note that the mapping file info is not actually stored in the DB and it's only used in the analysis so makes things easier.