Add specimen_id attribute to the study object

[ElDeveloper]

This PR adds a new attribute specimen_id_column to the Study object. This attribute holds the name of the column that is used to match tubes (or specimens) to sample identifiers in Qiita. The user interface has been updated to include a drop-down menu that lets the user select from any of the categories in the sample information that have unique values.

I have also modified the SampleTemplate.delete_column to prevent the deletion of columns that are currently selected as a specimen_id_column.

You may notice that there are a few unrelated changes, this is because the functionality to handle PATCH requests in the Study object was limited to only operate on tags. This PR has extended that functionality to also operate on the specimen_id_column. While making these changes I noticed that the AJAX requests were being formatted incorrectly. The PATCH standard states that the contents be delivered via the body of the request in JSON format, not via the request's arguments. I only changed this for the requests pertaining to the study object. However, there's many other instances of this problem elsewhere in the code base. For now, I've opened #2648.

The functionality itself can be seen here (note that only unique columns are selectable from the dropdown menu):

And when the user tries to delete that column:

cc @AmandaBirmingham since this is related to biocore/LabControl#273

[codecov-io]

Codecov Report

Merging #2649 into dev will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev    #2649      +/-   ##
==========================================
+ Coverage   94.36%   94.38%   +0.02%     
==========================================
  Files         166      166              
  Lines       19475    19563      +88     
==========================================
+ Hits        18377    18465      +88     
  Misses       1098     1098

Impacted Files	Coverage Δ
qiita_pet/handlers/api_proxy/__init__.py	100% <ø> (ø)	⬆️
qiita_pet/handlers/study_handlers/__init__.py	100% <ø> (ø)	⬆️
qiita_pet/webserver.py	100% <ø> (ø)	⬆️
...dlers/study_handlers/tests/test_sample_template.py	99.53% <ø> (ø)	⬆️
...ita_pet/handlers/study_handlers/tests/test_base.py	98.5% <100%> (+0.35%)	⬆️
...ita_db/metadata_template/base_metadata_template.py	95.08% <100%> (+0.03%)	⬆️
qiita_pet/handlers/study_handlers/base.py	71.42% <100%> (+1.15%)	⬆️
qiita_pet/handlers/api_proxy/studies.py	87.7% <100%> (+0.5%)	⬆️
qiita_pet/handlers/api_proxy/tests/test_studies.py	99.45% <100%> (+0.05%)	⬆️
qiita_db/test/test_study.py	99.77% <100%> (ø)	⬆️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03883e1...4f0e112. Read the comment docs.

[antgonza]

Code looks good. However, there are clear problems with performance. Also, I'm wondering if the changes affected the tags functionality via JS.

[antgonza]

This is gonna be extremely slow for large info files. Examples from the real system:

from qiita_db.metadata_template.sample_template import SampleTemplate

def unique_columns(st):
    df = st.to_dataframe()
    n = len(df)
    return {k for k, v in df.nunique().iteritems() if v == n}

In [22]: %timeit unique_columns(SampleTemplate(550)) # moving pics
1 loop, best of 3: 662 ms per loop

In [23]: %timeit unique_columns(SampleTemplate(10317)) # AGP
1 loop, best of 3: 1min 16s per loop

As you can imagine something blocking for 1min 16s is not desirable and more in the sample info page which is pretty common and we normally expect fasts replies.

Options:

• Do everything in DB - preferred but not sure if much faster
• Do not display only unique values but all columns - and option but don't love it
• Store unique values in redis and change every time the sample id is replaced as part of the update - kind of cool to store all summaries as part of the update
• Other?

[antgonza]

items -> iteritems cause items doesn't exist in the pandas version we have in qiita

[antgonza]

Wondering if this step (as it has to verify that the column is unique and set the SQL values) should be a job vs. a simple set ...

[ElDeveloper]

This is only checking one column, so I don't think it will take that much time. The request should also be asynchronous from the UI so it shouldn't block any interaction. Maybe I'm overlooking something?

[wasade]

A minor fix and a question

[wasade]

For this to always clean up, it may be worth wrapping the assertion in a try/finally

[ElDeveloper]

This can indeed become a problem. I looked into following your suggestion, however this entire test suite follows the pattern I implemented. For now I've left it as is and opened an issue with a proposed solution here #2654

[wasade]

Should this be 404?

[ElDeveloper]

The handler implementing this functionality returned a 200 regardless of the status, hence I left that as is.

[wasade]

That seems surprising right? If you agree, perhaps worth an issue?

[ElDeveloper]

Yes, I agree with you. Opened here: #2655

[ElDeveloper]

Code looks good. However, there are clear problems with performance.

Thanks to some further investigation by @antgonza we've concluded that we can't compute the "unique columns" in a fast way for large studies (namely the american gut project). Hence the UI now shows all the available columns and errors if a non-unique column is selected.

[antgonza]

Code looks good, a few more questions/requests:

• Can you add explanation in the help or even beside the GUI selector of what specimen_id is? I can see users being confused about this and think is a requirement.
• Is it possible, via the GUI, to unselect specimen_id? I think this is desirable, in case a user sets it by mistake, and if this is possible, what are the implications for when it has been used for platting?

[ElDeveloper]

Can you add explanation in the help or even beside the GUI selector of what specimen_id is? I can see users being confused about this and think is a requirement.

Great idea, I've added a small ? badge with a tooltip.

Is it possible, via the GUI, to unselect specimen_id? I think this is desirable, in case a user sets it by mistake

Yes, now it is possible.

.... what are the implications for when it has been used for platting?

That should be fine since this category is mainly used for mapping the tube identifiers to the Qiita sample ids. Hence if this category ceased to exist (or was deselected) the plate viewer would instead show qiita sample identifiers.

---

For reference here's how the UI currently looks. Things to note (1) the question mark badge and the tooltip, (2) the new default in the select i.e. None (not available), and (3) how this information can be updated to either a column or to None.

[antgonza]

Thanks! Did you notice that the button beside the selector is not even, it gets cut, I guess this depends on the size of the window and the solution is to make sure the number of "bootstrap" columns is correct, can you fix?

[ElDeveloper]

Good catch, fixed. Looks like I had incorrectly set the column width of the button.

[antgonza]

Sorry, just realized that: biocore/LabControl#175; so either we decide on a name and use it here or we remove the name from the tooltip ...

[ElDeveloper]

I would prefer removing the name. What about "Only for use with external LIMS systems".

[ElDeveloper]

Thanks for the feedback!