xCRG failing on CI?

[edeutsch]

Max has noted that ARAX answer quality has dramatically degraded on CI
https://arax.ncats.io/?systest=235

One hypothesis was that it was due to the RTX-KG2 PloverDB CI instance being down for a week over the break, but it seems to be back and the problem persists.

The error seems to be:

 File "/mnt/data/orangeboard/production/RTX/code/ARAX/ARAXQuery/Infer/scripts/creativeCRG.py", line 110, 
in load_ML_CRGmodel\n fitModel = joblib.load(file_path)\n ^^^^^^^^^^^^^^^^^^^^^^\n', ' 
File "/usr/local/lib/python3.12/dist-packages/joblib/numpy_pickle.py", line 658, 
in load\n obj = _unpickle(fobj, filename, mmap_mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' 
File "/usr/local/lib/python3.12/dist-packages/joblib/numpy_pickle.py", line 577, 
in _unpickle\n obj = unpickler.load()\n ^^^^^^^^^^^^^^^^\n', ' 
File "/usr/lib/python3.12/pickle.py", line 1203, in load\n raise EOFError\n', 'EOFError\n'

Seems to be an End of File error with creativeCRG?

When CI started on 2025-12-24T07:24:44.003522, it reported:

-rwxrwxrwx. 1 rt rt  1630233926 May 17  2025 chemical_gene_embeddings_v1.0.KG2.10.0_refreshedTo_KG2.10.2.npz

I assume that this is right file?

On arax.ncats.io:

-rw-r--r-- 1 ubuntu ubuntu  1630233926 May 17  2025 chemical_gene_embeddings_v1.0.KG2.10.0_refreshedTo_KG2.10.2.npz

i.e. the same size and date.

Here is an example of a query that failed:
https://arax.ci.transltr.io/?r=0750ab15-608e-4501-809b-f809cec45e5f

If I rerun the query today, I still also get an error.

Here is a query_graph:

{
  "edges": {
    "t_edge": {
      "attribute_constraints": [],
      "knowledge_type": "inferred",
      "object": "ON",
      "predicates": [
        "biolink:affects"
      ],
      "qualifier_constraints": [
        {
          "qualifier_set": [
            {
              "qualifier_type_id": "biolink:object_aspect_qualifier",
              "qualifier_value": "activity_or_abundance"
            },
            {
              "qualifier_type_id": "biolink:object_direction_qualifier",
              "qualifier_value": "decreased"
            }
          ]
        }
      ],
      "subject": "SN"
    }
  },
  "nodes": {
    "ON": {
      "categories": [
        "biolink:Gene"
      ],
      "constraints": [],
      "is_set": false
    },
    "SN": {
      "categories": [
        "biolink:ChemicalEntity"
      ],
      "constraints": [],
      "ids": [
        "CHEBI:142437"
      ],
      "is_set": false
    }
  }
}

Running this on arax.ncats.io works:
https://arax.ncats.io/?r=424701

But fails on https://arax.ci.transltr.io/

Why is that? I do not know. Any ideas?

[edeutsch]

One thing I'm wondering is: what is a .npz file? Is it possible that the new Python 3.12 docker container as built on CI does not have the libraries needed for .npz files? @bazarkua ?

[bazarkua]

Good question @edeutsch
.npz is just zipped format that numpy uses to save compressed files, to open .npz files within a Docker container, the primary required is numpy.
CICD-Dockerfile line 62:
RUN cd /mnt/data/orangeboard/production/RTX && python3 -m pip install -r requirements.txt --break-system-packages
file that Jenkins uses to build a docker container on CI uses requirements.txt that installs numpy version compatible with python3.12
So I think it shouldn't be an issue

[bazarkua]

Another reason to believe that it's not the issue is that the same code with the same requirements.txt file runs on arax.ncats.io packages versions are the same for numpy and scipy and it seems to be working

[edeutsch]

Thanks @bazarkua , current conversation and thinking over in Slack devops is that while the KG2.10.2 directory seems good, it may be that the KG2.10.0 directory is missing. Waiting for response from Kanna

[edeutsch]

More specifically, current thinking is that files 2 and 3 here:

      "xcrg_embeddings": "/translator/data/orangeboard/databases/KG2.10.2/chemical_gene_embeddings_v1.0.KG2.10.0_refreshedTo_KG2.10.2.npz",
      "xcrg_increase_model": "/translator/data/orangeboard/databases/KG2.10.0/xcrg_increase_model_v1.0.KG2.10.0_new_version.pt",
      "xcrg_decrease_model": "/translator/data/orangeboard/databases/KG2.10.0/xcrg_decrease_model_v1.0.KG2.10.0_new_version.pt"

May be missing in CI. Waiting for response from Kanna

[bazarkua]

Now the setup follows @edeutsch guidance:
Reverted from -f DockerBuild/CICD-Dockerfile to -f DockerBuild/Dockerfile
Changed config.domain to arax.ci.transltr.io
Dockerfile is now used by Jenkins for ITRB CI (arax.ci.transltr.io)
CICD-Dockerfile Used for CICD instance (Github Actions test suite)

[edeutsch]

Great, thanks, this is a good step forward.

but it did not solve the main problem of this issue. xCRG is still failing:
https://arax.ci.transltr.io/?r=424929

[edeutsch]

I have been able to get a listing of the KG2.10.0 directory, and find that the needed artifact files are zero-length files:

--- Directory listing of /mnt/data/orangeboard/databases/KG2.10.0
total 3401904
-rwxrwxrwx. 1 rt rt 2146578816 Aug 26  2024 xcrg_decrease_model_v1.0.KG2.10.0.pt
-rwxrwxrwx. 1 rt rt          0 Dec 17 22:38 xcrg_decrease_model_v1.0.KG2.10.0_new_version.pt
-rwxrwxrwx. 1 rt rt 1336957664 Aug 26  2024 xcrg_increase_model_v1.0.KG2.10.0.pt
-rwxrwxrwx. 1 rt rt          0 Dec 17 22:38 xcrg_increase_model_v1.0.KG2.10.0_new_version.pt

on arax.ci.transltr.io. I have requested that Kanna clear and re-download them, although I am not certain of the details of how best to do this.

[edeutsch]

Problem is likely that the 2 "new_version" files need to be uploaded to sftp.transltr.io
@sundareswarpullela has the power to do this?

[sundareswarpullela]

Hey @edeutsch , I think I do have access. I will work with @hodgesf to get the new database files into the sftp server.

[hodgesf]

@edeutsch the files are now on the ITRB sftp server.