collection read_qc_analysis_activity_set is empty

[aclum]

This collection should be populated in mongo prod with ~4000 records, it was correctly populated earlier in the week. Note that collection read_QC_analysis_activity_set IS expected to be empty. read_qc_analysis_activity_set (lower case qc) is defined in the schema and should be populated. Please look into what happened and if needed restore from a mongo backup.

@eecavanna @shreddd @Michal-Babins @turbomam @dwinston @ssarrafan

This is a high priority issue.

FYI @naglepuff @marySalvi

[dwinston]

I see a drop_collection command in the oplog after the last backup, which dropped 2923 records, not ~4000. @eecavanna does the latest backup have ~4000 records or more like ~1000?

# db.getCollection("oplog.rs").find({"ns": "nmdc.$cmd", "op": "c", "o.drop": "read_qc_analysis_activity_set", "wall": {"$gt": ISODate("2023-11-01T13:00:00-0700")}})

{
    "op" : "c",
    "ns" : "nmdc.$cmd",
    "ui" : UUID("f901047e-6d80-40d6-ba46-ee36aaef4c50"),
    "o" : {
        "drop" : "read_qc_analysis_activity_set"
    },
    "o2" : {
        "numRecords" : 2923
    },
    "ts" : Timestamp({ t: 1698937887, i: 1625 }),
    "t" : Long("18"),
    "v" : Long("2"),
    "wall" : ISODate("2023-11-02T15:11:27.179+0000")
}

[aclum]

@dwinston 2923 records is probably the right number, its should about the same as the metagenome_assembly_set collection which is a downstream task and that has 2807 records.

[eecavanna]

I'm working on the restore step now (to a temporary database) and will let you know once I know.

[dwinston]

completed targeted restore of the collection only. as yet unknown how this happened.