Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,8 +482,9 @@ unavailable, so that run is Failed rather than a fabricated score.
The home list is clickable: `GET /api/analysis-runs/{id}` fills a
labeled detail (cutoff, requested date, 12-character digest prefixes
with full digests on hover, counts, status history)
without exposing a DSN or raw record. Opening a cutoff title warns
that the live body may have changed after the run. Status history is detail-only
without exposing a DSN or raw record. Opening a cutoff title still
shows the live body and names both clocks when the title was
rewritten after the run. Status history is detail-only
and uses lookup labels plus occurrence times; a failure event keeps
its machine `failure_code` rather than an invented caption. Failed
TEPP list rows add a next-action line (open the run, then connect the
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.d/0.87.1-analysis-run-live-write-clock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# 0.87.1 Analysis-run live write clock

In-cutoff titles now say whether the live row was rewritten after the
run. Open Demo public post as the edited counter-example; Demo private
post still matches the January cutoff. The opened live body names both
clocks. Bodies stay live.
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@ All notable changes to this project are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.87.1] - 2026-08-16

### Added

- Analysis-run detail now compares each in-cutoff title's live
`updated_at` with that run's knowledge cutoff. After `make seed`,
open the Demo Corp lineage run: Demo public post is marked
**Updated after cutoff**; Demo private post is not. Opening a
marked title still shows the live body and names both clocks.
Cutoff body versioning stays a later slice (ADR 0016). The list
stays aggregates-only. No TEPP theta is invented.

## [0.87.0] - 2026-08-16

### Added
Expand Down
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ mention TEPP. A failed period-report row rebuilds the report. A
pending TEPP row does not claim a calibrated measurement. A pending
lineage row says reconstruction has not started yet.
Digest prefixes stay audible; hover a prefix to read the full digest.
Opening a cutoff title shows the live post -- compare it with the
cutoff before treating the body as reconstructed evidence (ADR 0016).
Opening a cutoff title shows the live post. Titles marked updated
after cutoff were rewritten after the run; the opened body names
both clocks. Compare those bodies before treating them as
reconstructed evidence (ADR 0016 / 0021).
`POST /api/analysis-runs` records Pending on an authorized
cutoff capture (ADR 0017) and does not reconstruct lineage.
46 changes: 39 additions & 7 deletions backend/app/analysis_run_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,22 @@ def _iso(value: Any) -> str:
return value.isoformat() if hasattr(value, "isoformat") else str(value)


def _as_utc(value: datetime) -> datetime:
"""Treat a naive clock as UTC so cutoff comparison stays timezone-aware."""
if value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)


def live_write_after_cutoff(updated_at: datetime, knowledge_cutoff: datetime) -> bool:
"""True when the live row was rewritten after the run's analysis clock.

``created_at <= knowledge_cutoff`` admits the title. ``updated_at`` is
the live write clock (ADR 0016). Equal times stay in-cutoff evidence.
"""
return _as_utc(updated_at) > _as_utc(knowledge_cutoff)


async def _counts_by_run(
conn: asyncpg.Connection,
run_ids: list[str],
Expand Down Expand Up @@ -258,15 +274,21 @@ async def fetch_visible_scope_posts(
scope_key: str | None,
affiliated_entity_ids: list[str],
knowledge_cutoff: Any,
) -> list[dict[str, str]]:
) -> list[dict[str, Any]]:
"""ABAC-visible post titles known at the run cutoff -- never a hidden body.

``knowledge_cutoff`` is the analysis clock (W3C Time / ISO 8601-1:2019;
ADR 0013/0016). A later live post must not appear inside an earlier run.
``updated_at`` is compared separately so the operator can see which
in-cutoff titles were rewritten after that clock. The live body is
still not returned.
"""
columns = (
"post_id, post_title, visibility_code, corporate_entity_id, updated_at"
)
if scope_kind_code == "analysis_scope_corporate_entity" and corporate_entity_id:
rows = await conn.fetch(
"select post_id, post_title, visibility_code, corporate_entity_id "
f"select {columns} "
"from source_post where corporate_entity_id = $1 "
"and created_at <= $2 "
"order by created_at, post_title",
Expand All @@ -275,7 +297,7 @@ async def fetch_visible_scope_posts(
)
elif scope_kind_code == "analysis_scope_process_unit" and process_unit_id:
rows = await conn.fetch(
"select post_id, post_title, visibility_code, corporate_entity_id "
f"select {columns} "
"from source_post where process_unit_id = $1 "
"and created_at <= $2 "
"order by created_at, post_title",
Expand All @@ -284,7 +306,7 @@ async def fetch_visible_scope_posts(
)
elif scope_kind_code == "analysis_scope_thread_group" and scope_key:
rows = await conn.fetch(
"select post_id, post_title, visibility_code, corporate_entity_id "
f"select {columns} "
"from source_post where thread_group_key = $1 "
"and created_at <= $2 "
"order by created_at, post_title",
Expand All @@ -293,20 +315,30 @@ async def fetch_visible_scope_posts(
)
elif scope_kind_code == "analysis_scope_all_visible":
rows = await conn.fetch(
"select post_id, post_title, visibility_code, corporate_entity_id "
f"select {columns} "
"from source_post where created_at <= $1 "
"order by created_at, post_title",
knowledge_cutoff,
)
else:
return []
affiliated = {str(entity_id) for entity_id in affiliated_entity_ids}
posts: list[dict[str, str]] = []
posts: list[dict[str, Any]] = []
for row in rows:
visible = row["visibility_code"] == "public" or str(row["corporate_entity_id"]) in affiliated
if not visible:
continue
posts.append({"post_id": str(row["post_id"]), "post_title": row["post_title"]})
updated_at = row["updated_at"]
posts.append(
{
"post_id": str(row["post_id"]),
"post_title": row["post_title"],
"updated_at": _iso(updated_at),
"live_after_cutoff": live_write_after_cutoff(
updated_at, knowledge_cutoff
),
}
)
return posts


Expand Down
73 changes: 69 additions & 4 deletions backend/tests/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
_MIGRATION_PATH = Path(__file__).resolve().parents[2] / "migrations" / "0001_initial_schema.sql"
_REGISTRY_MIGRATION = Path(__file__).resolve().parents[2] / "migrations" / "0018_analysis_run_registry.sql"
_RETENTION_MIGRATION = Path(__file__).resolve().parents[2] / "migrations" / "0020_analysis_run_retention_purge.sql"
_WRITE_CLOCK_MIGRATION = Path(__file__).resolve().parents[2] / "migrations" / "0021_source_post_write_clock.sql"


def _postgres_available() -> bool:
Expand Down Expand Up @@ -117,6 +118,7 @@ def seeded_db(demo_analyst_token):
cur.execute(_MIGRATION_PATH.read_text())
cur.execute(_REGISTRY_MIGRATION.read_text())
cur.execute(_RETENTION_MIGRATION.read_text())
cur.execute(_WRITE_CLOCK_MIGRATION.read_text())
cur.execute(
"insert into common_lookup_value (lookup_category, lookup_code, lookup_label) values "
"('corporate_entity_level', 'group', 'Group'), "
Expand Down Expand Up @@ -305,11 +307,21 @@ def _insert_post(
visibility_code: str,
body: str = "body",
created_at: str = "2026-01-10T12:00:00Z",
updated_at: str | None = None,
) -> str:
written_at = updated_at if updated_at is not None else created_at
cur.execute(
"insert into source_post (author_account_id, corporate_entity_id, post_title, post_body, voc_type_code, visibility_code, created_at) "
"values (%s, %s, %s, %s, 'voc', %s, %s) returning post_id",
(account_id, corporate_entity_id, title, body, visibility_code, created_at),
"insert into source_post (author_account_id, corporate_entity_id, post_title, post_body, voc_type_code, visibility_code, created_at, updated_at) "
"values (%s, %s, %s, %s, 'voc', %s, %s, %s) returning post_id",
(
account_id,
corporate_entity_id,
title,
body,
visibility_code,
created_at,
written_at,
),
)
return str(cur.fetchone()[0])

Expand All @@ -329,6 +341,14 @@ def _insert_post(
"A follow-up written after the January 2026 run cutoff.",
created_at="2026-01-20T12:00:00Z",
)
_insert_post(
"Edited own-corp private post",
own_corp_id,
"private",
"A January post rewritten after the run cutoff.",
created_at="2026-01-10T12:00:00Z",
updated_at="2026-01-13T09:00:00Z",
)

cur.execute(
"insert into cataloged_person (person_name, person_side_code) values "
Expand Down Expand Up @@ -494,11 +514,51 @@ def test_analysis_runs_are_labeled_aggregates_and_hide_other_scopes(
assert all("failure_code" not in event for event in history)
titles = {post["post_title"] for post in body["visible_posts"]}
assert "Own-corp private post" in titles
assert "Edited own-corp private post" in titles
assert "Late own-corp private post" not in titles
assert "Other-corp private post" not in titles
posts_by_title = {post["post_title"]: post for post in body["visible_posts"]}
assert posts_by_title["Own-corp private post"]["live_after_cutoff"] is False
assert posts_by_title["Edited own-corp private post"]["live_after_cutoff"] is True
assert posts_by_title["Edited own-corp private post"]["updated_at"].startswith("2026-01-13")
assert "post_body" not in posts_by_title["Edited own-corp private post"]
assert "postgresql://" not in str(body)
assert "visible_posts" not in visible

admin_conn = psycopg2.connect(seeded_db["dsn"])
admin_conn.autocommit = True
try:
with admin_conn.cursor() as cur:
cur.execute(
"select updated_at from source_post where post_title = %s",
("Own-corp private post",),
)
pinned = cur.fetchone()[0]
cur.execute(
"update source_post set post_body = post_body || ' rewritten' "
"where post_title = %s",
("Own-corp private post",),
)
cur.execute(
"select updated_at from source_post where post_title = %s",
("Own-corp private post",),
)
rewritten = cur.fetchone()[0]
assert rewritten > pinned
cur.execute(
"update source_post set post_body = %s, updated_at = %s "
"where post_title = %s",
("body", pinned, "Own-corp private post"),
)
cur.execute(
"select updated_at from source_post where post_title = %s",
("Own-corp private post",),
)
honoured = cur.fetchone()[0]
assert honoured == pinned
finally:
admin_conn.close()

hidden = client.get(
f"/api/analysis-runs/{seeded_db['hidden_run_id']}",
headers={"Authorization": f"Bearer {demo_analyst_token}"},
Expand Down Expand Up @@ -587,7 +647,12 @@ def test_post_list_includes_public_and_own_corp_but_excludes_other_corp(client,
response = client.get("/api/posts", headers={"Authorization": f"Bearer {demo_analyst_token}"})
assert response.status_code == 200
titles = {post["post_title"] for post in response.json()}
assert titles == {"Public post", "Own-corp private post", "Late own-corp private post"}
assert titles == {
"Public post",
"Own-corp private post",
"Late own-corp private post",
"Edited own-corp private post",
}
public = next(post for post in response.json() if post["post_title"] == "Public post")
assert public["voc_type_label"] == "Voice of Customer"
assert public["visibility_label"] == "Public"
Expand Down
1 change: 1 addition & 0 deletions docker/postgres-init/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ COPY migrations/0017_prov_o_standard_relations.sql /docker-entrypoint-initdb.d/1
COPY migrations/0018_analysis_run_registry.sql /docker-entrypoint-initdb.d/19-analysis-run-registry.sql
COPY migrations/0019_role_catalog_identity.sql /docker-entrypoint-initdb.d/20-role-catalog-identity.sql
COPY migrations/0020_analysis_run_retention_purge.sql /docker-entrypoint-initdb.d/21-analysis-run-retention-purge.sql
COPY migrations/0021_source_post_write_clock.sql /docker-entrypoint-initdb.d/22-source-post-write-clock.sql
# Official image already drops to this account at runtime; declare it so
# the Dockerfile itself satisfies DS-0002 (explicit non-root USER).
USER postgres
19 changes: 13 additions & 6 deletions docs/adr/0016-analysis-run-knowledge-cutoff-posts.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,11 @@ every scope branch (corporate entity, process unit, thread group, and
all-visible). ABAC visibility is applied after that temporal gate.
Click-through still opens the live post body -- post versioning is a
later slice -- but the run list itself must not advertise a post the
run was not allowed to know. The detail must say that next action
plainly: compare the opened body with this cutoff before treating it
as reconstructed evidence.
run was not allowed to know. Detail compares the live `updated_at`
write clock with `knowledge_cutoff` and marks titles rewritten after
the run. The next action is specific: only those marked titles need a
cutoff comparison before treating the live body as reconstructed
evidence.

Reproducibility digests on the same detail use a labeled group whose
accessible name does not replace the visible prefixes (W3C Accessible
Expand All @@ -44,11 +46,16 @@ run.
- After `make seed`, the Demo Corp lineage run lists Demo public post
and other in-cutoff Demo Corp titles. The later fixture account-review
post (2026-02-10) does not appear.
- Open the run, read the live-body warning, then open a listed post
and compare it with the cutoff date.
- Open the run: Demo public post is marked updated after cutoff
(`updated_at` 2026-01-13). Demo private post is not.
- Open a marked title: the live popup names the write clock and the
cutoff. Compare that body with the run before treating it as
reconstructed evidence.
- Hover a digest prefix to read the full code or configuration digest
when you need to match the API payload.
- Post-body versioning at the cutoff remains future work.
- Post-body versioning at the cutoff remains future work. The write
clock is a projection, not a stored cutoff body. Migration 0021
(ADR 0021) keeps `updated_at` honest after a title or body write.
- Thread-group *run list* visibility now uses the same cutoff
(ADR 0018). A later public post cannot surface a previously hidden
thread-group run.
Expand Down
49 changes: 49 additions & 0 deletions docs/adr/0021-source-post-write-clock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# ADR 0021 — Source-post write clock honors explicit pins

**Decision status:** Accepted
**Date:** 2026-08-16

## Context

ADR 0016 compares live `source_post.updated_at` with
`analysis_run.knowledge_cutoff` so the operator can see which in-cutoff
titles were rewritten after the run. The column defaulted to `now()` on
insert and never moved on a later title or body write, so a real rewrite
stayed unmarked and `make seed` could not pin Demo public post to
2026-01-13 unless every statement assigned `updated_at`.

W3C Time Ontology in OWL (Hobbs & Pan, 2017) and ISO 8601-1:2019 keep
the live write clock distinct from the analysis cutoff. A missing write
clock and a confidently-in-cutoff clock are different things.

## Decision

Migration `0021_source_post_write_clock.sql` adds
`touch_source_post_write_clock` on `BEFORE UPDATE` of `source_post`.
The trigger bumps `updated_at` to `clock_timestamp()` only when
`post_title` or `post_body` changes and the statement did not already
assign `updated_at`. Thread-group, visibility, or grouping-key updates
do not pretend to be a rewrite.

`make seed` may still pin Demo public post to 2026-01-13 and Demo
private post to its create clock. A later product body edit after the
January cutoff marks the title.

## Consequences

- After `make seed`, open the Demo Corp lineage run: Demo public post
is marked updated after cutoff; Demo private post is not.
- Open a marked title: the live popup names both clocks. Compare that
body with the run before treating it as reconstructed evidence.
- Cutoff body versioning remains a later slice.
- Roll back `0021` before `0020` / `0018` when emptying a database
that also uses the retention purge.

## References

International Organization for Standardization. (2019). *ISO 8601-1:2019:
Date and time—Representations for information interchange—Part 1: Basic
rules* (confirmed 2024; Amendment 1:2022).

World Wide Web Consortium. (2022). *Time ontology in OWL* (W3C
Recommendation). https://www.w3.org/TR/owl-time/
2 changes: 1 addition & 1 deletion docs/doctoring/ANALYSIS_RUN_REGISTRY_REFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
| Source | Product implication | Implemented evidence |
|---|---|---|
| W3C PROV-DM and PROV-O | Preserve identifiable entities, activities, agents, generation/use, and derivation without flattening provenance into display-only edges. | `analysis_source_snapshot`, `analysis_run`, authenticated requester, append-only status events, immutable digests; later product bindings continue to use the separate `provenance_*` layer from ADR 0011. |
| W3C Time Ontology in OWL | Keep temporal concepts explicit and avoid collapsing distinct clocks. | Evidence availability and snapshot capture remain on `analysis_source_snapshot`; analysis knowledge cutoff and request time remain on `analysis_run`; status occurrence and database record time remain distinct. `GET /api/analysis-runs/{id}` visible posts apply `created_at <= knowledge_cutoff` (ADR 0016). Opening a listed title warns that the live body may have changed after that cutoff. |
| W3C Time Ontology in OWL | Keep temporal concepts explicit and avoid collapsing distinct clocks. | Evidence availability and snapshot capture remain on `analysis_source_snapshot`; analysis knowledge cutoff and request time remain on `analysis_run`; status occurrence and database record time remain distinct. `GET /api/analysis-runs/{id}` visible posts apply `created_at <= knowledge_cutoff` (ADR 0016). Detail compares live `updated_at` with that cutoff and marks titles rewritten after the run. Migration 0021 bumps `updated_at` on a title or body rewrite unless the statement already assigned the clock (ADR 0021). |
| W3C Accessible Name and Description Computation 1.1 | Do not let `aria-label` replace visible text the operator must hear. | Analysis-run digest prefixes live in a labeled group; the prefixes remain the accessible contents and the full digest is on `title` for hover verification. |
| ISO 8601-1:2019 | Use unambiguous timestamp representation and timezone-aware persistence. | PostgreSQL `timestamptz` for availability, capture, cutoff, request, occurrence, and record clocks; tests use explicit `Z` offsets. |
| PostgreSQL 18 constraints and trigger contracts | Put integrity close to durable truth and use constraints for row shape while triggers enforce cross-row state and serialization. | Digest/check constraints, category allowlists, account-scoped uniqueness, shape constraints, immutable-row triggers, shared snapshot-row locking, and serialized status transitions. |
Expand Down
Loading
Loading