Merge pull request #1946 from fractal-analytics-platform/1874-placeho…

…lder-remove-support-for-python39

Remove py3.9 + add py3.12

.github/workflows/ci.yml

@@ -15,7 +15,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.9", "3.10"]
+        python-version: ["3.10"]
         db: ["postgres-psycopg"]
 
     services:
@@ -89,18 +89,14 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.9", "3.10", "3.11"]
-        db: ["sqlite", "postgres", "postgres-psycopg"]
+        python-version: ["3.10", "3.11", "3.12"]
+        db: ["sqlite", "postgres-psycopg"]
         exclude:
 
-          - python-version: "3.9"
-            db: "postgres"
-          - python-version: "3.9"
-            db: "sqlite"
-          - python-version: "3.11"
-            db: "postgres"
           - python-version: "3.11"
             db: "sqlite"
+          - python-version: "3.12"
+            db: "sqlite"
 
     services:
       postgres:

.github/workflows/migrations.yml

@@ -31,7 +31,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.9", "3.10", "3.11"]
+        python-version: ["3.10", "3.11", "3.12"]
 
     services:
       postgres:

CHANGELOG.md

@@ -3,16 +3,17 @@
 
 # 2.7.0 (unreleased)
 
-> WARNING: This release requires running `fractalctl update-db-data` (after
-> `fractalctl set-db`).
-
-> WARNING: when running `fractalctl update-db-data`, the environment variable
-> `FRACTAL_V27_DEFAULT_USER_EMAIL` must be set, e.g. as in
-> ```
-> FRACTAL_V27_DEFAULT_USER_EMAIL=admin@fractal.yx fractalctl update-db-data
-> ```
-> This user must exist, and they will own all previously-common
-> tasks/task-groups.
+> WARNING: This release comes with several specific notes:
+>
+> 1. It requires running `fractalctl update-db-data` (after `fractalctl set-db`).
+> 2. When running `fractalctl update-db-data`, the environment variable
+>    `FRACTAL_V27_DEFAULT_USER_EMAIL` must be set, e.g. as in
+>    `FRACTAL_V27_DEFAULT_USER_EMAIL=admin@fractal.yx fractalctl
+>    update-db-data`. This user must exist, and they will own all
+>    previously-common tasks/task-groups.
+> 3. The pip extra `postgres` is deprecated, in favor of `postgres-psycopg-binary`.
+> 4. The configuration variable `DB_ENGINE="postgres"` is deprecated, in favor of `DB_ENGINE="postgres-psycopg"`.
+> 5. Python3.9 is deprecated.
 
 * API:
     * Users and user groups:
@@ -76,7 +77,8 @@
     * Add `TaskGroupV2` columns `wheel_path`, `pinned_package_versions` (\#1861).
     * Clean up `alembic` migration scripts (\#1894).
     * Verify task-group non-duplication constraint in `2.7.0` data-migration script (\#1927).
-    * Normalize pkg_name in `2.7.0` data-migration script (\#1930).
+    * Normalize `pkg_name` in `2.7.0` data-migration script (\#1930).
+    * Deprecate `DB_ENGINE="postgres"` configuration variable (\#1946).
 * Runner:
     * Do not create local folders with 755 permissions unless `FRACTAL_BACKEND_RUNNER="slurm"` (\#1923).
     * Fix bug of SSH/SFTP commands not acquiring lock (\#1949).
@@ -94,8 +96,11 @@
     * Update `_create_first_group` so that it only searches for `UserGroups` with a given name (\#1964).
 * Dependencies:
     * Bump fastapi to `0.115` (\#1942).
+    * Remove pip extra `postgres`, corresponding to `psycopg2+asyncpg` (\#1946).
+    * Deprecate python3.9 (\#1946).
 * Testing:
-    *  Benchmark `GET /api/v2/task-group/` (\#1922).
+    * Benchmark `GET /api/v2/task-group/` (\#1922).
+    * Use new `ubuntu22-slurm-multipy` image, with Python3.12 and with Python-version specific venvs (\#1946).
 
 # 2.6.4
 
@@ -1016,7 +1021,7 @@ Warning: updating to this version requires changes to the configuration variable
 
 # 1.2.4
 
-* Review setup for database URLs, especially to allow using UNIX-socket connections for postgresl (\#657).
+* Review setup for database URLs, especially to allow using UNIX-socket connections for postgresql (\#657).
 
 # 1.2.3
 

docs/configuration.md

@@ -10,6 +10,6 @@ Some are optional and sensible defaults are provided.
 > FRACTAL_RUNNER_WORKING_BASE_DIR
 > ```
 >
-> together with the database-specific variables: `SQLITE_PATH` for SQLite, or `DB_ENGINE=postgres` and `POSTGRES_DB` for Postgres.
+> together with the database-specific variables: `SQLITE_PATH` for SQLite, or `DB_ENGINE=postgres-psycopg` and `POSTGRES_DB` for Postgres.
 
 ::: fractal_server.config

docs/install_and_deploy.md

@@ -25,7 +25,7 @@ visible from a private network) for communicating with the Fractal client.
 
 ## How to install
 
-> ⚠️  The minimum supported Python version for fractal-server is 3.9.
+> ⚠️  The minimum supported Python version for fractal-server is 3.10.
 
 Fractal Server is hosted on [the PyPI
 index](https://pypi.org/project/fractal-server), and it can be installed with

docs/internals/database_interface.md

@@ -42,19 +42,12 @@ set-db`.
 ### Requirements
 
 To use PostgreSQL as a database, Fractal Server must be installed with the
-`postgres` extra:
+`postgres-psycopg-binary` extra:
 
 ```console
-$ pip install "fractal-server[postgres]"
+$ pip install "fractal-server[postgres-psycopg-binary]"
 ```
-which will install two additional Python libraries (`asyncpg` and `psycopg2`).
-
-**NOTE**: the following system libraries are required:
-
-- `postgresql`,
-- `postgresql-contrib`,
-- `libpq-dev`,
-- `gcc`.
+which will install an additional Python library (`psycopg[binary]`).
 
 
 ### Setup
@@ -91,7 +84,7 @@ is done by setting the following configuration variables (before running
 - Required:
 
     ```
-    DB_ENGINE=postgres
+    DB_ENGINE=postgres-psycopg
     POSTGRES_DB=fractal_db
     ```
 
@@ -110,7 +103,7 @@ from `SQLalchemy` to generate the appropriate URL to connect to:
 
 ```python
 URL.create(
-    drivername="postgresql+asyncpg",
+    drivername="postgresql+psycopg",
     username=self.POSTGRES_USER,
     password=self.POSTGRES_PASSWORD,
     host=self.POSTGRES_HOST,

fractal_server/config.py

@@ -167,9 +167,9 @@ def collect_oauth_clients(cls, values):
     ###########################################################################
     # DATABASE
     ###########################################################################
-    DB_ENGINE: Literal["sqlite", "postgres", "postgres-psycopg"] = "sqlite"
+    DB_ENGINE: Literal["sqlite", "postgres-psycopg"] = "sqlite"
     """
-    Select which database engine to use (supported: `sqlite` and `postgres`).
+    Database engine to use (supported: `sqlite`, `postgres-psycopg`).
     """
     DB_ECHO: bool = False
     """
@@ -203,16 +203,7 @@ def collect_oauth_clients(cls, values):
 
     @property
     def DATABASE_ASYNC_URL(self) -> URL:
-        if self.DB_ENGINE == "postgres":
-            url = URL.create(
-                drivername="postgresql+asyncpg",
-                username=self.POSTGRES_USER,
-                password=self.POSTGRES_PASSWORD,
-                host=self.POSTGRES_HOST,
-                port=self.POSTGRES_PORT,
-                database=self.POSTGRES_DB,
-            )
-        elif self.DB_ENGINE == "postgres-psycopg":
+        if self.DB_ENGINE == "postgres-psycopg":
             url = URL.create(
                 drivername="postgresql+psycopg",
                 username=self.POSTGRES_USER,
@@ -235,11 +226,7 @@ def DATABASE_ASYNC_URL(self) -> URL:
 
     @property
     def DATABASE_SYNC_URL(self):
-        if self.DB_ENGINE == "postgres":
-            return self.DATABASE_ASYNC_URL.set(
-                drivername="postgresql+psycopg2"
-            )
-        elif self.DB_ENGINE == "postgres-psycopg":
+        if self.DB_ENGINE == "postgres-psycopg":
             return self.DATABASE_ASYNC_URL.set(drivername="postgresql+psycopg")
         else:
             if not self.SQLITE_PATH:
@@ -546,20 +533,13 @@ def check_db(self) -> None:
         """
         Checks that db environment variables are properly set.
         """
-        if self.DB_ENGINE == "postgres":
+        if self.DB_ENGINE == "postgres-psycopg":
             if not self.POSTGRES_DB:
                 raise FractalConfigurationError(
-                    "POSTGRES_DB cannot be None when DB_ENGINE=postgres."
-                )
-            try:
-                import psycopg2  # noqa: F401
-                import asyncpg  # noqa: F401
-            except ModuleNotFoundError:
-                raise FractalConfigurationError(
-                    "DB engine is `postgres` but `psycopg2` or `asyncpg` "
-                    "are not available"
+                    "POSTGRES_DB cannot be None when DB_ENGINE="
+                    "postgres-psycopg."
                 )
-        elif self.DB_ENGINE == "postgres-psycopg":
+
             try:
                 import psycopg  # noqa: F401
             except ModuleNotFoundError:

fractal_server/tasks/v2/_venv_pip.py

@@ -73,6 +73,11 @@ async def _pip_install(
         ),
         logger_name=logger_name,
     )
+    await execute_command(
+        cwd=Path(task_group.venv_path),
+        command=f"{python_bin} -m pip install setuptools",
+        logger_name=logger_name,
+    )
     await execute_command(
         cwd=Path(task_group.venv_path),
         command=f"{python_bin} -m pip install {pip_install_str}",

fractal_server/tasks/v2/background_operations_ssh.py

@@ -205,7 +205,7 @@ def background_collect_pip_ssh(
                 remove_venv_folder_upon_failure = True
 
                 stdout = _customize_and_run_template(
-                    script_filename="_2_upgrade_pip.sh",
+                    script_filename="_2_preliminary_pip_operations.sh",
                     **common_args,
                 )
                 stdout = _customize_and_run_template(

fractal_server/tasks/v2/templates/_2_upgrade_pip.sh → fractal_server/tasks/v2/templates/_2_preliminary_pip_operations.sh

@@ -15,6 +15,7 @@ VENVPYTHON=${PACKAGE_ENV_DIR}/bin/python
 # Upgrade pip
 write_log "START upgrade pip"
 "$VENVPYTHON" -m pip install "pip<=__FRACTAL_MAX_PIP_VERSION__" --upgrade
+"$VENVPYTHON" -m pip install setuptools
 write_log "END   upgrade pip"
 echo