Sample evaluation dataset - ICD-11

[filiperochalopes]

@paynejd :

Sample evaluation dataset -- 50-100 rows; closer in structure to a country dataset (more than just a name, so that we can also test out the AI Assistant); already mapped with an answer key, so that we can use for the ICD-11 Evaluation E2E Workflow (e.g. this could be CIEL concepts that are already mapped to ICD-11)

Following the same #2302 (comment) rarity distribution:

icd11_ciel_100_concepts.csv

[filiperochalopes]

Pending the full metadata: what is needed to be included? @paynejd @askanter

[paynejd]

is there any additional metadata that we might want to include in the spreadsheet?

[filiperochalopes]

I need a real-world example of a country dataset sample. For now, I can't know about the needed metadata. Do you think we need any?

[askanter]

I think we could use the Uganda or Malawi dataset. However, these were approved for the concept prevalence study and not for publication elsewhere. I think it is OK for us to use it as part of our ICD-11 mapping for them. Malawi does not have mappings (so would just be class and datatype). Uganda has mappings:

[filiperochalopes]

For this second phase of the 50–100 row sample, do you think we need to include the datatype (which is always N/A for ICD-11) and the class (almost always “Diagnosis,” since we are not treating extensions alone as standalone mappings such as anatomy, etc.)?

Are mappings relevant for this test?

@paynejd @askanter

[filiperochalopes]

From what I’m seeing, I can consider the CIEL repository itself as a real-world, country-level dataset for this purpose.

[askanter]

We should filter the Uganda set based on class, perhaps to create an ICD-11-specific source. CIEL is not really a good test for real world situations (except where CIEL was the basis of the country dataset :) )