upgrade version python used for ARAX to python3.12

[saramsey]

I know this was brought up at an ARAX AHM recently, but I am resurfacing the matter as an issue.

Hi all, we're currently using python3.9 in ARAX's flask application, and that is a pretty old version of cpython. Since 3.9, the PEP484 type hinting system has been vastly improved in terms of ergnonomics and expressiveness, enabling much more precise pre-runtime analysis of python code, to catch bugs before testing inside flask and before unit testing. So for that reason alone, I think the system could benefit by transitioning to a new version of python. If we can bump all the way to 3.12, we might get a speed improvement, as a side bonus.

[edeutsch]

sure, this would be fine by me. We'd have to hope that our tangle of dependencies would be good on Python 3.12.

Is 3.12 everyone else's vote as well? Any reason to select a different version?

[saramsey]

3.12 is fine with me

[bazarkua]

I’ve investigated RTX/code/UI/OpenAPI/python-flask-server and confirmed it imports ARAX modules (e.g. RTXConfiguration, ARAX_database_manager, ARAX_background_tasker). This means Flask app must run in the same Python interpreter. And since ARAX is pinned to Python 3.9, this forces the Flask process to run under 3.9. Because of that we can't upgrade Python version just for ARAX Flask app.
__main__.py:

`sys.path.append(os.path.dirname(os.path.abspath(__file__)) +
                "/../../../../ARAX/ARAXQuery")
sys.path.append(os.path.dirname(os.path.abspath(__file__)) +
                "/../../../..")

from RTXConfiguration import RTXConfiguration
from ARAX_database_manager import ARAXDatabaseManager

I think there are two possible ways:

1. We would need to upgrade full ARAX codebase to run under Python 3.12 (a lot of dependencies would need to be upgraded, and changes to dockerfile would need to be made as well)
2. Another solution which I'm not sure about is to isolate the Flask app via running it on it's own docker container to keep ARAX core on 3.9, we would need to some workarounds so Flask app doesn't use modules from ARAX directly.

[saramsey]

We have a bunch of dependabot warnings about dependencies that need to be updated anyhow. I lean toward trying to upgrade everything to run in Python 3.12.

Since the full ARAX already runs inside a Docker container, trying to change ARAX so that only the Flask app runs in a Docker container seems like an unappealing option. I'm not saying it couldn't be done. But rather, I think trying to get the full ARAX working on Python 3.12 sounds like the more.... appealing of the two options, by comparison.

Just my off-the-cuff $0.02.

[edeutsch]

I never considered option 2 as a viable option, and assumed we were discussing option 1.

[bazarkua]

Migrated ARAX from Python 3.9 to Python 3.12 version code in issue2521: https://github.com/RTXteam/RTX/tree/issue2521

Updated requirements.txt with Python 3.12 compatible versions

• Upgraded numpy (1.26.4), pandas (2.2.1), scipy (1.13.1), scikit-learn (1.4.2)
• Updated Flask (3.0.2), SQLAlchemy (2.0.28), neo4j-driver (5.18.0)
• Removed asyncio (now built-in to Python 3.12)

Code Fixes:

• Fixed Flask JSONEncoder import for Flask 2.3 compatibility
• Resolved pkgutil.ImpImporter removal by updating dependencies
• Handled distutils removal using ensurepip
• Updated Docker containers to install Python 3.12

Run pytest suite 143 tests pass

Also tested Flask app startup and functionality
Next I will try to test issue2521 branch on arax.rtx.ai or arax.ncats.io/beta

[bazarkua]

@chunyuma After testing example 3 query on arax.rtx.ai as an effort to verify successful migration from python3.9 to python3.12, I've encountered this error:

2025-07-25T06:39:21.212167 INFO: (57283) [] loading embeddings and models into memory
/home/ubuntu/venv312/lib/python3.12/site-packages/sklearn/base.py:442: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeClassifier from version 1.0.2 when using version 1.7.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
2025-07-25T06:39:27.692841 ERROR: (57283) [UncaughtARAXiError] An uncaught error occurred: node array from the pickle has an incompatible dtype:
- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]: ['Traceback (most recent call last):\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_query.py", line 791, in execute_processing_plan\n    expander.apply(response, action[\'parameters\'], mode=mode)\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_expander.py", line 328, in apply\n    response, overarching_kg = self.get_inferred_answers(inferred_qedge_keys, query_graph, response)\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_expander.py", line 722, in get_inferred_answers\n    infer_response = inferer.apply(response, infer_input_parameters)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_infer.py", line 366, in apply\n    getattr(self, \'_\' + self.__class__.__name__ + \'__\' + parameters[\'action\'])()  # thank you https://stackoverflow.com/questions/11649848/call-methods-by-string\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_infer.py", line 619, in __chemical_gene_regulation_graph_expansion\n    XCRG = creativeCRG(self.response, os.path.sep.join([*pathlist[:(RTXindex + 1)], \'code\', \'ARAX\', \'ARAXQuery\', \'Infer\', \'data\', \'xCRG_data\']))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/ARAX/ARAXQuery/Infer/scripts/creativeCRG.py", line 216, in __init__\n    self.increase_model = load_ML_CRGmodel(self.response, self.data_path, \'increase\')\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/ARAX/ARAXQuery/Infer/scripts/creativeCRG.py", line 110, in load_ML_CRGmodel\n    fitModel = joblib.load(file_path)\n               ^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 749, in load\n    obj = _unpickle(\n          ^^^^^^^^^^\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 626, in _unpickle\n    obj = unpickler.load()\n          ^^^^^^^^^^^^^^^^\n', '  File "/usr/lib/python3.12/pickle.py", line 1256, in load\n    dispatch[key[0]](self)\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 446, in load_build\n    Unpickler.load_build(self)\n', '  File "/usr/lib/python3.12/pickle.py", line 1760, in load_build\n    setstate(state)\n', '  File "sklearn/tree/_tree.pyx", line 836, in sklearn.tree._tree.Tree.__setstate__\n', '  File "sklearn/tree/_tree.pyx", line 1542, in sklearn.tree._tree._check_node_ndarray\n', "ValueError: node array from the pickle has an incompatible dtype:\n- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}\n- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]\n"]

it's happening because XCRG models were created with scikit-learn 1.0.2 in a python 3.9 environment (old)
and scikit-learn 1.0.2 is very old and not compatible with python3.12 the oldest we can use is scikit-learn==1.2.2 that's still compatible with python3.12 but it's causing this error, because the internal data structures changed in newer versions of scikit-learn

[chunyuma]

Hi @bazarkua，thanks for bringing up this issue. It seems like this problem might not be resolved until we retrained the xCRG model with the updated python and scikit-learn version. It may need to take up to two weeks to make the retraining process done because the GraphSage model, which was used to train xCRG, has been updated for a while. I am not sure whether it can be compatible with python 3.12 and the updated scikit-learn package. If not, I need to find the alternative model. Is it urgent for this update? Is it ok to allow me to retrain the model in the next two w? @saramsey

[edeutsch]

I am wondering if it might be easier/reasonable simply to write a trivial script to unpickle the file(s) and write to JSON under Python 3.9. And then another trivial script to read the JSON and write out pickle files and run under Python 3.12?