Merge branch 'development' of github.com:hotosm/fmtm into feat/entity…

…-creation

compose.yaml

@@ -289,7 +289,7 @@ services:
       retries: 3
 
   electric:
-    image: "electricsql/electric:${ELECTRIC_TAG:-0.9.3}"
+    image: "electricsql/electric:${ELECTRIC_TAG:-1.0.0-beta.10}"
     depends_on:
       fmtm-db:
         condition: service_healthy

deploy/compose.development.yaml

@@ -235,7 +235,7 @@ services:
       retries: 3
 
   electric:
-    image: "electricsql/electric:${ELECTRIC_TAG:-0.9.3}"
+    image: "electricsql/electric:${ELECTRIC_TAG:-1.0.0-beta.10}"
     depends_on:
       fmtm-db:
         condition: service_healthy

deploy/compose.main.yaml

@@ -160,7 +160,7 @@ services:
       retries: 3
 
   electric:
-    image: "electricsql/electric:${ELECTRIC_TAG:-0.9.3}"
+    image: "electricsql/electric:${ELECTRIC_TAG:-1.0.0-beta.10}"
     depends_on:
       fmtm-db:
         condition: service_healthy

docs/decisions/0005-translations.md

@@ -0,0 +1,64 @@
+# Use ParaglideJS for frontend internationalisation (translations)
+
+## Context and Problem Statement
+
+We need to serve FMTM in many languages, not just English, as mappers are
+global.
+
+Ideally these translations are contributed by the **community** and not
+just the HOT team.
+
+We will likely ask the community to contribute translations via **Weblate**,
+or another similar tool.
+
+We will not consider closed-source / paid tools for this, such as:
+
+- Transifex
+- Tolgee (although this looks good!)
+- Crowdin
+
+## Considered Options
+
+- Any library that is framework specific, such as **react-i18next**,
+  **react-intl**, or **svelte-i18n**. These won't be used to avoid lock-in and
+  to aid transfer of translations across projects.
+- **i18next** is one of the most prominent, but other tools have
+  since superseded this in terms of usability and performance.
+- **lingui** looks like an excellent choice, with many nice features
+  such as being platform agnostic, good community, and semantic key
+  translations (meaning the actual english text to be translated
+  is present in the code, instead of a key like home.banner.hello).
+- **ParaglideJS** is the newest here, with most of the same advantages of
+  lingui, including a few extra such as code splitting / tree shaking.
+
+## Decision Outcome
+
+We chose ParaglideJS for two primary reasons:
+
+- Support for tree shaking of translations. Typically the translations
+  for an app are all loaded into the frontend as a large bundle. Paraglide
+  works differently, in that the frontend only receives the translations
+  that are used on the current page (i.e. are split per component / page).
+  This uses advantages provided by the bundler (e.g. Vite) to significantly
+  reduce bundle size! It's truly a next generation i18n solution that will
+  no doubt be copied by others.
+
+- Some other nice features in the tool ecosystem, such as the 'fink'
+  translation editor. This is an interactive GUI for less technical users
+  to edit the translations, then push them to Github (without needing to
+  understand Git and make changes in the code / files directly).
+
+### Consequences
+
+- Good, because it's the most performant of all the solutions.
+- Good, because the DevEx is excellent, being very easy to integrate.
+- Good, because the files are simple JSON that is easy to edit.
+- Good, because supports any web framework through `paraglide-js` package.
+- Good, because there is already a nice wrapper available for simpler
+  use with Svelte, so we can get integrating into the mapper frontend easily.
+- Good, because non-developers can easily use Fink UI to edit translations.
+- Bad, because it's the newest tool here and does not have
+  significant testing across many production apps.
+- Bad, because it doesn't integrate easily with crowdsource translation
+  apps such as Weblate, due to lack of an interchange format like
+  gettext PO or XLIFF. This may change in future.

docs/manuals/mapping.md

@@ -1,6 +1,6 @@
-# Mapper Frontend Documentation
+# Mapper Page Documentation
 
-The **Mapper Frontend** was developed to provide a simpler, more intuitive
+The **Mapper Page** was developed to provide a simpler, more intuitive
 mapping experience.
 
 ## Prerequisites
@@ -38,7 +38,7 @@ for field mapping:
 ### Step 1: Install The Custom ODK Collect Mobile App
 
 - The first time you load the project, you should be prompted to download
-  the custom `.apk` from the sidebar.
+  the custom ODK `.apk` from the sidebar.
 - Once downloaded, you should install the custom ODK Collect application.
 
   ![highlighted-sidebar](https://github.com/user-attachments/assets/53de2d80-2709-45b0-bb82-32f0190c7859)
@@ -52,18 +52,18 @@ for field mapping:
       You may have to enable installing from unknown sources in your device
       settings too.
 
-### Step 2: Access the Mapper Frontend
+### Step 2: Access the Mapper Page
 
-- **Option 1:** Click the **Start Mapping** button on the project cards of
-  explore project page.
-- **Option 2:** Click the **Start Mapping** button on the project details page.
+- **Option 1:** Click on the project cards from your mobile device
 - **Option 3:** Go to `https://fmtm.hotosm.org/mapnow/<project_id>` to open
-  the Mapper Frontend for a specific project.
+  the Mapper Page for a specific project.
 
 !!! note
 
       This functionality is designed for mappers in the field, so it is
-      recommended to use a mobile device to access it.
+      recommended to use a mobile device to access it. If you use a computer
+      browser, you will be redirected to the project details page instead of
+      mapper's page.
 
 !!! warning
 
@@ -74,7 +74,7 @@ for field mapping:
 
 ### Step 3: Configure ODK Collect (once only)
 
-- **Option 1:** Scan the QR code displayed on the Mapper Frontend using
+- **Option 1:** Scan the QR code displayed on the Mapper page using
   the custom ODK Collect mobile application.
 
 - **Option 2:** Download the QR code and import it into ODK Collect to
@@ -89,7 +89,7 @@ for field mapping:
 !!! tip
 
       For a demonstration of the process above, click the **i** info icon
-      on the QR Code tab of the mapper frontend.
+      on the QR Code tab of the mapper page.
 
 ### Step 4: Load Imagery (optional)
 
@@ -129,6 +129,8 @@ In most cases we are submitting a survey about a feature that already exists.
 - Now click on a feature you wish to map: a popup will appear.
 - Now click 'Map Feature In ODK': ODK Collect will open, with the
   feature pre-selected in the survey (no need to open the ODK map!).
+- If the feature you are trying to map is further than 50m away, you will
+  be prompted with warning message to ensure you made the correct selection.
 - Complete the survey and submit.
 
 ![IMG_20250109_160742](https://github.com/user-attachments/assets/bf350d1c-c80e-42ee-970b-ca71a3713a9f)
@@ -155,7 +157,8 @@ Sometimes the feature does not exist on the map yet!
 - Click on a task area: a popup will display.
 - At the top right, there is a button **Map New Feature**.
 - Click on the map to create a new geometry.
-- ODK Collect will be opened automatically to fill out the survey
+- Once the geometry is drawn on FMTM, save and confirm to be redirected to ODK.
+- ODK Collect will then be opened to fill out the survey
   data for the newly created feature.
   ![IMG_20250109_160816](https://github.com/user-attachments/assets/98b70f5a-4db8-46cb-84ae-58bec07c82c1)
 

src/backend/app/central/central_crud.py

@@ -20,6 +20,7 @@
 import csv
 import json
 from asyncio import gather
+from datetime import datetime
 from io import BytesIO, StringIO
 from typing import Optional, Union
 from uuid import uuid4
@@ -722,6 +723,7 @@ async def get_entities_data(
     odk_id: int,
     dataset_name: str = "features",
     fields: str = "__system/updatedAt, osm_id, status, task_id, submission_ids",
+    filter_date: Optional[datetime] = None,
 ) -> list:
     """Get all the entity mapping statuses.
 
@@ -733,17 +735,26 @@ async def get_entities_data(
         dataset_name (str): The dataset / Entity list name in ODK Central.
         fields (str): Extra fields to include in $select filter.
             __id is included by default.
+        filter_date (datetime): Filter entities last updated after this date.
 
     Returns:
         list: JSON list containing Entity info. If updated_at is included,
             the format is string 2022-01-31T23:59:59.999Z.
     """
     try:
+        url_params = f"$select=__id{',' if fields else ''} {fields}"
+
+        filters = []
+        if filter_date:
+            filters.append(f"__system/updatedAt gt {filter_date}")
+        if filters:
+            url_params += f"&$filter={' and '.join(filters)}"
+
         async with central_deps.get_odk_dataset(odk_creds) as odk_central:
             entities = await odk_central.getEntityData(
                 odk_id,
                 dataset_name,
-                url_params=f"$select=__id{',' if fields else ''} {fields}",
+                url_params=url_params,
             )
     except Exception as e:
         log.exception(f"Error: {e}", stack_info=True)

src/backend/app/db/enums.py

@@ -134,6 +134,7 @@ class TaskEvent(StrEnum, Enum):
     MERGE = "MERGE"
     ASSIGN = "ASSIGN"
     COMMENT = "COMMENT"
+    RESET = "RESET"
 
 
 class MappingState(StrEnum, Enum):

src/backend/app/tasks/task_crud.py

@@ -17,11 +17,14 @@
 #
 """Logic for FMTM tasks."""
 
-from datetime import timedelta
+from datetime import datetime, timedelta, timezone
 
 from psycopg import Connection
 from psycopg.rows import class_row
 
+from app.central.central_crud import get_entities_data, update_entity_mapping_status
+from app.db.enums import EntityState
+from app.db.models import DbOdkEntities, DbProject
 from app.db.postgis_utils import timestamp
 from app.tasks import task_schemas
 
@@ -63,3 +66,115 @@ async def get_project_task_activity(
     async with db.cursor(row_factory=class_row(task_schemas.TaskEventCount)) as cur:
         await cur.execute(sql, {"project_id": project_id, "end_date": end_date})
         return await cur.fetchall()
+
+
+async def trigger_unlock_tasks(db: Connection):
+    """Function to unlock_old_locked_tasks manually."""
+    active_projects_query = """
+        SELECT DISTINCT project_id
+        FROM task_events
+        WHERE created_at >= NOW() - INTERVAL '7 days'
+    """
+    async with db.cursor() as cur:
+        await cur.execute(active_projects_query)
+        active_projects = await cur.fetchall()
+
+    time_now = datetime.now(timezone.utc)
+    threedaysago = (time_now - timedelta(days=3)).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+    onehourago = (time_now - timedelta(hours=1)).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+
+    for (project_id,) in active_projects:
+        project = await DbProject.one(db, project_id, True)
+
+        recent_project_entities = await get_entities_data(
+            project.odk_credentials, project.odkid, filter_date=threedaysago
+        )
+        # If there are recent entity updates, skip this project
+        if recent_project_entities:
+            continue
+
+        await reset_entities_status(db, project.id, onehourago)
+
+        # Only unlock tasks if there are no recent entity updates
+        await unlock_old_locked_tasks(db, project.id)
+
+
+async def reset_entities_status(db, project_id, filter_date):
+    """Reset status for entities that have been 'open in ODK' for more than 1hr."""
+    project = await DbProject.one(db, project_id, True)
+    recent_opened_entities = await get_entities_data(
+        project.odk_credentials,
+        project.odkid,
+        filter_date=filter_date,
+    )
+    for entity in recent_opened_entities:
+        if entity["status"] != str(EntityState.OPENED_IN_ODK):
+            continue
+        await update_entity_mapping_status(
+            project.odk_credentials,
+            project.odkid,
+            entity["id"],
+            f"Task {entity['task_id']} Feature {entity['osm_id']}",
+            str(EntityState.READY),
+        )
+
+    # Sync ODK entities in our database
+    project_entities = await get_entities_data(project.odk_credentials, project.odkid)
+    await DbOdkEntities.upsert(db, project.id, project_entities)
+
+
+async def unlock_old_locked_tasks(db, project_id):
+    """Unlock tasks locked for more than 3 days."""
+    unlock_query = """
+        WITH svc_user AS (
+            SELECT id AS svc_user_id, username AS svc_username
+            FROM users
+            WHERE username = 'svcfmtm'
+        ),
+        recent_events AS (
+            SELECT DISTINCT ON (t.id, t.project_id)
+                t.id AS task_id,
+                t.project_id,
+                the.created_at AS last_event_time,
+                the.event AS last_event
+            FROM tasks t
+            JOIN task_events the
+                ON t.id = the.task_id
+                AND t.project_id = the.project_id
+                AND the.comment IS NULL
+            WHERE t.project_id = %(project_id)s
+            ORDER BY t.id, t.project_id, the.created_at DESC
+        ),
+        filtered_events AS (
+            SELECT *
+            FROM recent_events
+            WHERE (
+                (last_event = 'ASSIGN' AND last_event_time < NOW() - INTERVAL '3 days')
+            OR
+                (last_event = 'MAP' AND last_event_time < NOW() - INTERVAL '3 hours')
+            )
+        )
+        INSERT INTO task_events (
+            event_id,
+            task_id,
+            project_id,
+            event,
+            user_id,
+            state,
+            created_at,
+            username
+        )
+        SELECT
+            gen_random_uuid(),
+            fe.task_id,
+            fe.project_id,
+            'RESET'::taskevent,
+            svc.svc_user_id,
+            'UNLOCKED_TO_MAP'::mappingstate,
+            NOW(),
+            svc.svc_username
+        FROM filtered_events fe
+        CROSS JOIN svc_user svc;
+    """
+    async with db.cursor() as cur:
+        await cur.execute(unlock_query, {"project_id": project_id})

src/backend/app/tasks/task_routes.py

@@ -115,3 +115,12 @@ async def get_task_event_history(
 ):
     """Get the detailed history for a task."""
     return await DbTaskEvent.all(db, task_id=task_id, days=days, comments=comments)
+
+
+@router.post("/unlock-tasks")
+async def unlock_tasks(db: Annotated[Connection, Depends(db_conn)]):
+    """Endpoint to trigger unlock_old_locked_tasks manually."""
+    log.info("Start processing inactive tasks")
+    await task_crud.trigger_unlock_tasks(db)
+    log.info("Finished processing inactive tasks")
+    return {"message": "Old locked tasks unlocked successfully."}