Exclude unsupported EFO branches from Open Targets term mapping

[apriltuesday]

Open Targets uses the slim version of EFO, which excludes some branches such as material entity. We should ensure the automation fetches terms only from this subset when we are running Open Targets processing.

[apriltuesday]

One way to do this is to just use the regex OT uses during validation to filter the terms we fetch, but this doesn't help us catch potential issues with that regex.

[apriltuesday]

Some additional context: Currently terms in EFO but not slim will only be caught when we validate evidence, which is quite late in our processing. We also usually have to check these cases manually, since the regex used for diseaseFromSourceMappedId does not exactly correspond to presence in EFO slim (i.e. occasionally terms fail that regex but are actually in EFO slim). This sometimes causes last-minute schema change requests, which isn't ideal.

Unfortunately, I didn't realise EFO slim is a bit more complicated than anticipated, as it doesn't just exclude branches but also restructures EFO slightly. So restricting our searches to slim is not so straightforward.

We've come up with two possible solutions to this. The first is to add EFO slim as a supported ontology in OLS. This is the technically simplest option for us, but also doesn't make much sense as EFO slim is almost identical to EFO, and supporting it in OLS might have some undesirable implications.

The second approach is to use the regex from the JSON schema to filter out invalid terms earlier in our pipelines, so they don't make it to curation or evidence generation. We could fetch this regex from the current version of the schema every time to ensure it's in sync. This is not too difficult for us, but it assumes that OT has its own way of keeping that regex up-to-date and/or removing non-slim terms later on in their pipeline.

@ireneisdoomed @DSuveges @vivienho Let me know your thoughts, particularly on the second approach. Feel free to post here, otherwise we can discuss at our next meeting.

cc @tcezard