File format filter dropped from cURL manifest request

[nadove-ucsc]

Steps to reproduce (using Firefox 138.0.3 (64-bit) on Ubuntu 24.10):

1. Go to https://explore.data.humancellatlas.org/projects
2. Sign in
3. Filter by project title:
   • "Asian Immune Diversity Atlas (AIDA): Single-cell analysis of human diversity in circulating immune cells (Japan cells)
   • "A Cellular Anatomy of the Normal Adult Human Prostate and Prostatic Urethra"
4. Filter by file format: "xlsx"
5. Click "Export"
6. Click "Request cURL Command"
7. Select "Species: Unspecified"
8. Click "Request cURL Command"
9. Copy the generated cURL command to clipboard
10. Execute the command in a bash terminal

Expected behavior: only two .xlsx files are downloaded, as indicated by the browser UI before and during the generation of the cURL command.

Observed behavior: every file in the selected projects is downloaded, regardless of format, including many large fastq files.

The inspect UI indicates that the fileFormat parameter is missing from the browser's request to Azul.

[NoopDog]

Thanks will take a look.

[NoopDog]

Hi @nadove-ucsc, @bvizzier-ucsc @hannes-ucsc, this is happening because of the way we (incorrectly) implemented the "drop all redundant filters" for the verbatim PFB export. I want to clarify the requirement with you and pass by an approach to see if you think the proposed update will do the trick.

Requirement: To support the Azul verbatim PFB, the front end needs to detect requests for "whole projects" (all of the files in the project(s)), regardless of the user filter state. If the request is for a full project, drop any filters (besides the project filter). If the request is for a partial project, send the user filters as is.

If that is what we are trying to do, I can suggest the following approach:

Approach:

1. If the filter does not explicitly contain a project filter, then send all filters.
2. If the request explicitly contains one or more project filters:
   1. Construct a summary query with only the project filters (dropping all other filters) and note the file count.
   2. Compare that file count with the file count with all filters applied.
   3. If the file counts are equal, drop the filters (except the project filters); otherwise, send all user filters.

Do you think that can work?

Another option is that the front end always sends what the user selects, and the back end (Azul) tries optimizing the request.

WDYT?

Note that understanding what is going on is complicated by two things:

1. We do not have access to the AIDA dataset (do you?), so this is effectively a request for only the "A Cellular Anatomy of the Normal Adult Human Prostate and Prostatic Urethra" project.
2. The xlsx file is tagged with species "Unspecified" and all other files in "Cellular Anatomy" are tagged with "Homo sapiens".

[hannes-ucsc]

I don't think your description of the requirement is correct. The requirement is that orphans will only be included in the verbatim PFB manifest if all filters are dataset filters. That's it. You shouldn't need to "optimize" by detecting and dropping redundant non-dataset filters, i.e., non-dataset filters that don't further restrict the result set. It's OK for the orphans to be absent from the PFB if the user specifies any non-dataset filters, even redundant ones. A dataset filter is a filter for a property of the AnVIL dataset entity. A non-dataset filter is a filter for a property of any other entity.

More importantly, orphans are only an AnVIL feature so this consideration shouldn't even affect HCA. This is a critical bug because it yields manifests with files that don't match the user's selection. This bug causes them to download many more files than they initially intended to download.

If the user selects all file types and species on the last page of the export flow, you should send no filters on file type or species. This is because specifying a filter for every possible value of the filtered property is equivalent to specifying no filter on that property at all. This will eliminate the need of having to "optimize" the query, which is complicated, inefficient and error prone, as evidenced by this bug.

If you are unable to fix it this way, consider disabling the "optimization" for HCA, since HCA has no orphans.

[NoopDog]

If the user selects all file types and species on the last page of the export flow, you should send no filters on file type or species.

Yes, but to do this, we need to know what "ALL" is for the selected file types and species. So we need to hit Azul to do a summary request with any file types and species filters dropped to know the full set of file types and species implied by any other filters. Then we can compare the selected file types and species with the "ALL" file types and species and decide whether the file type and species filters are redundant.

We need the original list because file type selections can restrict species selections and vice versa. Once the user starts filtering by file type or species on the export page, we may no longer have the full list to compare to.

This determination can be made easier by comparing the file count without file types or species filters applied with the file count with the filters applied. If the file count matches, the current file type and species filters are ineffective, and we can drop them.

Making this customizable so it only applies to AnVIL and not HCA adds extra complexity, which, of course, we would like to avoid. Do you see any downside to dropping redundant file type and species filters during export for HCA as well?

Additionally, it seems we do not need to concern ourselves with "dataset" vs "non-dataset" filters, as our logic does not depend on them.

WDYT? @hannes-ucsc

[hannes-ucsc]

The 30k foot view is that there is a critical bug in the Data Browser manifest UI flow. I'm pretty sure the same bug applies to AnVIL as well, and all other manifest types. We need to fix this now, at the very least for HCA.

My recommendation to you is that you temporarily disable the complex and broken logic of dropping redundant filters until you have a firm grasp of how it should work. If you can't bear the "complexity" of disabling it for HCA only, you can disable it for both AnVIL and HCA. The collateral damage of not getting orphans in the verbatim manifests on AnVIL is an acceptable price to pay for not having this huge bug affect HCA, which gets much more traffic and exposure. But again, HCA does not have orphans, so disabling the logic for HCA only would be the pragmatic thing to do.

Another, alternative recommendation is to drop the post-filter UI completely and trust the users to use the side bar filters before they click on Export. That way you can send exactly the filters the users specify and all of this complexity goes away, and with it the bug. You'd only need to show them a warning that they're not getting orphans if they use non-dataset filters on AnVIL.