Global String Objects are Interned Only in the First Interpreter

[ericsnowcurrently]

When a string object is interned via _PyUnicode_InternInPlace(), its "state.interned" field is set. Afterward, subsequent calls to _PyUnicode_InternInPlace() will skip that string. The problem is that some strings may be used in multiple interpreters, which each have their own interned dict. The string is shared between the interpreters, along with its "state.interned" field. That means the string will only be interned in the first interpreter where _PyUnicode_InternInPlace() is called (ignoring races in the function, e.g. gh-106930).

We need to fix it so one of the following is true:

• there should be one global interned "dict" shared by all interpreters (we tried this already and it is very tricky)
• strings are always interned in every interpreter, regardless of the "state.interned" value

Linked PRs

• gh-106931: Intern Statically Allocated Strings Globally #107272
• [3.12] gh-106931: Intern Statically Allocated Strings Globally (gh-107272) #107358
• gh-106931: Fix the WASM Buildbots #107362
• [3.12] gh-106931: Intern Statically Allocated Strings Globally (gh-107272) #110713

[ericsnowcurrently]

CC @kumaraditya303

[kumaraditya303]

I prefer option 1 and would like to try make it work after I am in better health.

[ericsnowcurrently]

I likewise prefer the first option, though with some other data structure than a dict.

[ericsnowcurrently]

Solutions I've considered:

1. use a global _Py_hashtable for statically allocated strings and keep the per-interpreter dict for strings allocated by each interpreter
2. keep all per-interpreter dicts in sync, making a copy of the string in each one (or no copy if statically allocated)
3. use a global _Py_hashtable for all strings; store only main-interpreter-owned strings (i.e. copy if needed)