Error 500 on the ITRB-deployed Plover service

[bazarkua]

Error we see:

500 ERROR: /home/nobody: repository path '/home/nobody' is not owned by current user. Traceback: Traceback (most recent call last):
File "/app/app/main.py", line 211, in run_code_version
repo = pygit2.Repository(os.environ["HOME"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pygit2/repository.py", line 1791, in __init__
path_backend = init_file_backend(path, int(flags))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pygit2.GitError: /home/nobody: repository path '/home/nobody' is not owned by current user

Same code and Docker image work when we run the container ourselves (e.g. via run.sh on our AWS instance).
The failure happens only when the app runs in ITRB’s environment (deployed via Jenkins + Kubernetes to EKS).
We are not sure how the pod is configured on ITRB’s side (user, HOME, volume mounts, securityContext).

[bazarkua]

Plover runs on ITRB in Kubernetes. After a redeploy, the /code_version and /logs endpoints started returning 500 with: “repository path /home/nobody is not owned by current user” (pygit2 opening os.environ["HOME"] as a git repo). Self-hosted Plover (same code, run via run.sh on AWS) still works. @hodgesf traced the break to uncommenting uid/gid in uwsgi.ini: the process was forced to run as nobody, but in the pod that user doesn’t own /home/nobody (or the pod doesn’t allow that user), so pygit2 fails.

the current base image (FROM tiangolo/uwsgi-nginx-flask:python3.11) is deprecated on Docker Hub, and the docs say not to use it with Kubernetes they recommend building from scratch and running a single process, with replication handled by the cluster. So the immediate failure is a pod/uwsgi config (uid/gid) issue and the longer term solution is to stop using that base and move to a plain Python base and a single-process setup.

So my plan is to try the switch to FROM python:3.12 and building the image from scratch (single process, no uwsgi-nginx base).Plan would be to change the Dockerfile to use python3.12, get it running and test on our self-hosted AWS instance first, and only then try it on ITRB. Since Docker Hub doesn’t recommend using that uwsgi-nginx image with Kubernetes (and the image is deprecated anyway)

[bazarkua]

Currently kg2cplover.rtx.ai runs on the docker container that was built under new image using python.3.12
It's up on https://kg2cplover.rtx.ai:9990, so far I didn't notice any issues

All tests pass:

I merged code into main as experimental step to see if it works on ITRB, waiting for the triggered rebuild of the new image using new DockerFile

[hodgesf]

https://kg2cploverdb.ci.transltr.io is officially up and running. Here are the stats and results of running:

pytest -v test/test_kg2c.py --endpoint https://kg2cploverdb.ci.transltr.io

Time to run increase:

• old code: 29.28s
• new code: 324.73s
• 1009.05% or 11x slower

Failing one test:

========================== test session starts ==========================
platform darwin -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0 -- /Users/fmh/Desktop/translator/code/venv/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/fmh/Desktop/translator/code/PloverDB
plugins: anyio-4.12.1, locust-2.43.1
collected 30 items

test/test_kg2c.py::test_simple PASSED
test/test_kg2c.py::test_unconstrained_output_node PASSED
test/test_kg2c.py::test_unconstrained_predicate PASSED
test/test_kg2c.py::test_multiple_output_categories PASSED
test/test_kg2c.py::test_multiple_predicates PASSED
test/test_kg2c.py::test_doubly_pinned_query PASSED
test/test_kg2c.py::test_multiple_input_ids PASSED
test/test_kg2c.py::test_single_node_query PASSED
test/test_kg2c.py::test_single_node_query_with_multiple_ids PASSED
test/test_kg2c.py::test_catching_multihop_query PASSED
test/test_kg2c.py::test_symmetric_predicate PASSED
test/test_kg2c.py::test_asymmetric_predicate_1 PASSED
test/test_kg2c.py::test_asymmetric_predicate_2 PASSED
test/test_kg2c.py::test_concept_subclasses_single_node_query PASSED
test/test_kg2c.py::test_mixins_in_query PASSED
test/test_kg2c.py::test_canonical_predicate_handling PASSED
test/test_kg2c.py::test_hierarchical_category_reasoning PASSED
test/test_kg2c.py::test_hierarchical_predicate_reasoning PASSED
test/test_kg2c.py::test_query_id_mapping_in_results PASSED
test/test_kg2c.py::test_qualified_direction PASSED
test/test_kg2c.py::test_qualified_aspect PASSED
test/test_kg2c.py::test_qualified_predicate_overrides_regular_predicate PASSED
test/test_kg2c.py::test_qualified_predicate_only PASSED
test/test_kg2c.py::test_qualified_edge_queried_only_by_regular_predicate PASSED
test/test_kg2c.py::test_qualified_edge_queried_only_by_wrong_regular_predicate PASSED
test/test_kg2c.py::test_is_set_handling FAILED
test/test_kg2c.py::test_undirected_related_to_for_underlying_treats_edge PASSED
test/test_kg2c.py::test_get_edges PASSED
test/test_kg2c.py::test_get_neighbors PASSED
test/test_kg2c.py::test_version PASSED

========================== FAILURES ==========================

__________________________ test_is_set_handling __________________________

def test_is_set_handling():
    # Test is_set handling
    query = {
        "edges": {
            "e00": {
                "subject": "n01",
                "object": "n00",
                "predicates": ["biolink:treats"]
            }
        },
        "nodes": {
            "n00": {
                "ids": [DIABETES_CURIE]
            },
            "n01": {
                "categories": ["biolink:ChemicalEntity"]
            }
        }
    }

    query["nodes"]["n00"]["is_set"] = True
    query["nodes"]["n01"]["is_set"] = True
    trapi_response_issettrue = tester.run_query(query)
    results_issettrue = trapi_response_issettrue["message"].get("results")
    assert results_issettrue
    assert len(results_issettrue) == 1

    query["nodes"]["n00"]["is_set"] = False
    query["nodes"]["n01"]["is_set"] = True
    trapi_response_objectset = tester.run_query(query)

  results_objectset = trapi_response_objectset["message"].get("results")

                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

E KeyError: 'message'

test/test_kg2c.py:672: KeyError
__________________________ Captured stdout call __________________________
Request elapsed time: 0.390758 sec
Returned 1 results.
Response status code was 504. Response was:

<title>504 Gateway Time-out</title>

504 Gateway Time-out

========================== short test summary info ==========================
FAILED test/test_kg2c.py::test_is_set_handling - KeyError: 'message'

========================== 1 failed, 29 passed in 324.73s (0:05:24) ==========================

[bazarkua]

Thank you @hodgesf for running the tests and flagging the performance issue with gunicorn.

Test results (30 tests):

• Old code: 29.28s
• New code (gunicorn): 324.73s (~11x slower)

I've switched back to uWSGI on the new python:3.12.12 base image. This keeps:

• The ownership fix (plover user, uid 1000) for pygit2/ITRB compatibility
• uWSGI performance (C-based, much faster than pure-Python gunicorn)
• 8-16 workers via uWSGI's cheaper algorithm (scales with load)
• Worker recycling (max-requests, max-worker-lifetime) to prevent leaks

Setup comparison:

Aspect	Old (tiangolo)	New (python:3.12)
Base image	tiangolo/uwsgi-nginx-flask:python3.11	python:3.12.12
WSGI server	uWSGI	uWSGI
Workers	8-16 (cheaper)	8-16 (cheaper)
Threads	1 per worker	1 per worker
Container user	nobody	plover (uid 1000)

Testing locally on kg2cplover.rtx.ai now. Will push to main when stable.

[bazarkua]

Update: uWSGI + python 3.12 Build Ready for ITRB

Tested successfully on kg2cplover.rtx.ai. Performance matches the original setup (~15-45s for test suite And memory stays around ~95 GB matches old build). Pushing to main now.

Switched from gunicorn back to uWSGI on the new python:3.12.12 base image. This preserves the ownership fix while restoring original performance.

Metric	Gunicorn	uWSGI
Test suite (30 tests)	324.73s	~15-45s
Implementation	Pure Python	C-based
Performance	~11x slower	Matches original

uWSGI is written in C and optimized for high-concurrency workloads. Gunicorn (pure Python) couldn't match the performance for Plover's large in-memory indexes.

File	Change	Why
Dockerfile	Base image python:3.12.12 (pinned)	Reproducible builds, avoids breaking changes
Dockerfile	uWSGI instead of gunicorn	Performance (see above)
Dockerfile	UWSGI_PROCESSES=16, UWSGI_CHEAPER=8	8-16 workers via cheaper algorithm, tunable at runtime
requirements.txt	uwsgi==2.0.31 (pinned)	Replaces gunicorn
main.py	gc.freeze() after loading indexes	Critical: Preserves copy-on-write memory sharing across workers
main.py	Rewrote /debug endpoint	Uses /proc file reads instead of psutil (avoids memory overhead)
main.py	Rewrote /logs endpoint	Tail-reads last N lines instead of loading entire file (avoids OOM)
main.py	BatchSpanProcessor for Jaeger	Async span export, won't stall requests if Jaeger is slow
uwsgi.ini	Added http-socket, master, die-on-term	Required settings for standalone uWSGI (tiangolo provided these internally)
run.sh	Default --no-cache builds	Ensures code changes are picked up
run.sh	Port 9991 (container HTTP)	SSL handled externally by nginx/Ingress

gc.freeze()

This was the critical fix for memory stability. Without it:

• Python 3.12 may have more aggressive GC than 3.11, making COW breakage more frequent
• Worker recycling in uWSGI eventually cleaned up bloated workers, masking the gradual memory growth
• Each uWSGI worker would accumulate private copies of the 90GB indexes
• Memory ballooned from ~90GB to 200GB+ under load
• Workers got OOM-killed

Python's reference counting modifies objects when accessed, breaking Linux copy-on-write sharing. gc.freeze() tells Python's GC to leave the loaded indexes alone, preserving memory sharing across all workers.

[saramsey]

Outstanding debugging! Specifically, that you uncovered (what I understand is) the contributor to OOM failures under python3.12. In hindsight, I suppose that relying on the kernel's copy-on-write feature to minimize memory consumed by the indexes was always kind of a fragile thing. So I think I understand what gc.freeze is doing, but let me check. In the following code, x would not (for the remainder of the life of the cpython invocation) be touched by GC but y potentially would?

x = [i for i in range(100)]
gc.freeze()
y = [i for i in range(100)]

[bazarkua]

Thank you @saramsey, and good question!
x Created before gc.freeze(), so it is part of the set of objects that get frozen (plover_objs_map). It is moved into the permanent generation. For the rest of the interpreter’s lifetime, the GC will not examine or trace x during collections. It is effectively excluded from the GC machinery (reference counting still applies).
y Created after gc.freeze(), so it is not frozen. It stays in the normal generations (0, 1, 2) and is subject to normal GC behavior. we do not want to freeze) request data, response dicts, temporary objects created per request and should be collected afterward

In PloverDB we intentionally do not freeze again, because we want only the long‑lived, read‑only index data should be frozen so it stays shared across workers (copy‑on‑write).

Things created per request (response dicts, query results, temp objects, etc.) should stay under normal GC. We want them reclaimed when the request ends.

If we called gc.freeze() after requests, we’d freeze those short‑lived objects too. They’d never be collected, and memory would grow unbounded with traffic.

So we call gc.freeze() once right after loading the indexes, so only that startup data gets frozen, and everything created later during requests stays under normal GC.

[saramsey]

Thank you for the detailed explanation, @bazarkua. Very helpful.