Add temporal lineage traversal support - #28426
Conversation
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
Code Review
|
| Compact |
|
Was this helpful? React with 👍 / 👎 | Gitar
There was a problem hiding this comment.
Pull request overview
This PR adds temporal lineage traversal so lineage APIs, search traversal, and the UI can filter lineage edges by historical time windows while preserving lineage viewport state.
Changes:
- Adds
startTime/endTimesupport across lineage request schemas, REST parameters, cache keys, ES/OS traversal, and sample lineage data. - Adds lineage change-event types and emits add/update/delete lineage events.
- Adds a UI lineage time filter, edge audit display fields, URL-backed time filter state, and tests for lineage fetch/centering behavior.
Reviewed changes
Copilot reviewed 134 out of 145 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java |
Adds temporal query params to lineage endpoints. |
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchUtils.java |
Adds time-window filtering for ES lineage edges. |
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java |
Carries lineage temporal fields during search document updates. |
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ESLineageGraphBuilder.java |
Applies temporal filtering during ES lineage traversal/counting. |
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OSLineageGraphBuilder.java |
Applies temporal filtering during OpenSearch lineage traversal/counting. |
openmetadata-service/src/main/java/org/openmetadata/service/search/lineage/LineageCacheKey.java |
Adds time-window fields to lineage cache keys. |
openmetadata-service/src/test/java/org/openmetadata/service/search/lineage/LineageCacheKeyTest.java |
Updates cache key tests for new constructor fields. |
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java |
Preserves lineage temporal metadata and emits lineage change events. |
openmetadata-service/src/main/java/org/openmetadata/service/openlineage/OpenLineageMapper.java |
Maps OpenLineage event time into lineage details. |
openmetadata-service/src/main/java/org/openmetadata/service/util/LineageUtil.java |
Updates search delete lookup for lineage doc IDs. |
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/OpenLineageLineageResolutionIT.java |
Adds OpenLineage temporal metadata integration tests. |
openmetadata-spec/src/main/resources/json/schema/type/changeEventType.json |
Adds lineage add/update/delete event types. |
openmetadata-spec/src/main/resources/json/schema/api/lineage/searchLineageRequest.json |
Adds temporal request fields for lineage search. |
openmetadata-spec/src/main/resources/json/schema/api/lineage/entityCountLineageRequest.json |
Adds temporal request fields for entity-count lineage. |
openmetadata-spec/src/main/resources/elasticsearch/{en,jp,ru,zh}/* |
Adds lineage temporal/audit fields to search mappings. |
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.tsx |
Adds URL-backed time filter state and fetch-key handling. |
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.interface.tsx |
Exposes lineage time filter in context. |
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.test.tsx |
Adds lineage/impact-analysis fetch behavior tests. |
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.component.tsx |
Adds lineage time filter UI. |
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.interface.ts |
Defines time filter props and presets. |
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/CustomControls.component.tsx |
Renders the lineage time filter in controls. |
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityInfoDrawer/EdgeInfoDrawer.component.tsx |
Shows lineage edge created/updated metadata. |
openmetadata-ui/src/main/resources/ui/src/components/Lineage/EntityLineageTab/EntityLineageTab.tsx |
Preserves lineage graph DOM when switching views. |
openmetadata-ui/src/main/resources/ui/src/components/Lineage/Lineage.interface.ts |
Adds edge audit fields to UI edge details. |
openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.ts |
Threads temporal params through lineage API calls. |
openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.test.ts |
Updates lineage API request expectations. |
openmetadata-ui/src/main/resources/ui/src/utils/EntityLineageUtils.tsx |
Fixes node-centering width precedence. |
openmetadata-ui/src/main/resources/ui/src/utils/EntityLineageUtils.test.tsx |
Tests corrected lineage node centering. |
openmetadata-ui/src/main/resources/ui/src/locale/languages/*.json |
Adds translated labels for temporal lineage UI. |
openmetadata-ui/src/main/resources/ui/package.json |
Adds a Rollup native package dev dependency. |
openmetadata-ui/src/main/resources/ui/yarn.lock |
Updates lockfile entries. |
ingestion/src/metadata/ingestion/source/database/sample_data.py |
Supports temporal lineage details in sample ingestion. |
ingestion/examples/sample_data/lineage/lineage.json |
Adds temporal metadata to sample lineage edges. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 138 out of 149 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (2)
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ESLineageGraphBuilder.java:279
- The time filter is applied only after the downstream search hit has already been added to
result.getNodes()and queued inhasToFqnMapForLayer. If a downstream entity matches the structural query because of an inactive edge from the current layer, but all of its overlapping edges are filtered out here, the node is still displayed and traversed as if it were active. Build the filtered matching edge list first and only add/queue the node when at least one edge both overlaps the time window and connects to the current layer.
List<EsLineageData> upstreamEntities =
getUpstreamLineageListIfExist(
entityMap, lineageRequest.getStartTime(), lineageRequest.getEndTime());
for (EsLineageData esLineageData : upstreamEntities) {
if (hasToFqnMap.containsKey(esLineageData.getFromEntity().getFqnHash())) {
result
.getDownstreamEdges()
.putIfAbsent(esLineageData.getDocId(), esLineageData.withToEntity(toEntity));
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OSLineageGraphBuilder.java:279
- The time filter is applied only after the downstream search hit has already been added to
result.getNodes()and queued inhasToFqnMapForLayer. If a downstream entity matches the structural query because of an inactive edge from the current layer, but all of its overlapping edges are filtered out here, the node is still displayed and traversed as if it were active. Build the filtered matching edge list first and only add/queue the node when at least one edge both overlaps the time window and connects to the current layer.
List<EsLineageData> upstreamEntities =
getUpstreamLineageListIfExist(
entityMap, lineageRequest.getStartTime(), lineageRequest.getEndTime());
for (EsLineageData esLineageData : upstreamEntities) {
if (hasToFqnMap.containsKey(esLineageData.getFromEntity().getFqnHash())) {
result
.getDownstreamEdges()
.putIfAbsent(esLineageData.getDocId(), esLineageData.withToEntity(toEntity));
mohityadav766
left a comment
There was a problem hiding this comment.
Temporal lineage — review notes
Overall the plumbing is clean and consistent: the ES and OS builders mirror each other, the cache key is extended with the window, validateTemporalBounds guards startTime > endTime, and there's solid IT/unit coverage. One interaction I want to flag, though.
Preserve-paths does not extend to the time window
A path-preservation user expects that a node which doesn't itself match the filter is still shown when something in its upstream/downstream qualifies. That holds for the node-level query filter, but not for the new startTime/endTime window — and the combination is currently a silent no-op for the temporal dimension.
1. The time window hard-prunes during traversal. In fetchDownstreamNodesRecursively:
// ESLineageGraphBuilder.java:266-274
List<EsLineageData> upstreamEntities =
getMatchingUpstreamLineageData(entityMap, hasToFqnMap.keySet(),
lineageRequest.getStartTime(), lineageRequest.getEndTime());
if (upstreamEntities.isEmpty()) {
continue; // node dropped AND not enqueued for the next layer
}For a path root → A → (edge out of window) → B → (edge in window) → C: at B the connecting edge A→B fails the window, so B is skipped and never added to the next frontier. The BFS never explores B's downstream, so edge B→C and node C are never discovered — even though B→C is inside the window. The same hard-prune pattern exists in the upstream branch and in the depth-count BFS (getDepthWiseEntityCounts, allEntitiesUpToDepth). An out-of-window edge severs discovery of everything behind it, in both directions.
2. Preserve-path runs afterward, on an already-time-pruned graph. Path preservation deliberately strips the node-level query filter from the base traversal so the full topology is available, then filters in-memory:
// ESLineageGraphBuilder.java:325-333
SearchLineageRequest unfilteredRequest =
JsonUtils.deepCopy(lineageRequest, SearchLineageRequest.class)
.withQueryFilter(getStructuralFilterOnly(lineageRequest.getQueryFilter()));
result = searchLineageWithStrategyInternal(unfilteredRequest);
result = applyInMemoryFiltersWithPathPreservation(result, lineageRequest);The deepCopy overrides only queryFilter — startTime/endTime survive into unfilteredRequest. So the "unfiltered" base graph handed to LineagePathPreserver is already time-pruned, and the preserver can only trace paths through edges that still exist (buildParentAdjacencyMap walks existing edges only). It cannot resurrect B or C.
Net: the query filter is excluded from the base traversal and applied in post-processing (preserve-able); the time window is baked into the base traversal as a hard edge prune (not preserve-able). A user passing preservePaths=true + a time window gets no path preservation for the temporal dimension, with no error or signal.
Recommendation
Decide the intended semantic and make it explicit:
- If the window should honor preserve-paths (keep an out-of-window intermediate so an in-window up/downstream node still shows): move time filtering out of the inline BFS into the same post-processing path as the node filter. Concretely — also strip
startTime/endTimefromunfilteredRequest, traverse structurally, then in the in-memory step collect endpoints of in-window edges as "matching nodes" and feed them toLineagePathPreserver.preservePathsWithEdges(...), exactly as the column/node filter does. - If hard pruning is intended (show the graph as of the window, severing through stale edges): that's defensible, but it should be documented on the
startTime/endTimeschema fields, and it should be explicit thatpreservePathsdoes not extend to the temporal filter — otherwise the silent interaction is confusing.
Smaller notes
SearchUtils.edgeMatchesWindow: legacy edges with both timestamps null always match any window. Reasonable for back-compat, but a window query over mixed legacy + temporal data returns all legacy edges — worth calling out in the API docs.- Minor directional asymmetry: downstream drops the candidate node entirely on an out-of-window connecting edge (
continue), while upstream keeps the frontier node and only prunes onward edges. Same path-severing effect, but node retention differs between directions — worth a sanity check.
Addresses review on PR #28426: the startTime/endTime window is applied as a hard prune during graph traversal, not as a preserve-paths post-filter. An out-of-window edge severs discovery of everything reachable only through it, so preservePaths does not extend to the temporal dimension. - Document the hard-prune + preservePaths interaction and the legacy null-timestamp back-compat behavior on startTime/endTime in both searchLineageRequest.json and entityCountLineageRequest.json. - Add code comments at the downstream prune sites (ES + OS builders) capturing the shared invariant: a non-root node appears iff reached by at least one in-window connecting edge. Upstream gates this at enqueue time, downstream via the empty-list continue; net behavior is symmetric.
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
# Conflicts: # openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/UserMetricsResourceIT.java # openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/mcp/McpToolsValidationIT.java
|
|
Code Review ✅ Approved 5 resolved / 5 findingsImplements temporal lineage traversal across the backend, search indexing, and UI components to enable historical edge state viewing. Resolved issues include incorrect export endpoint parameter forwarding, faulty ES script overwrites, duplicate interface definitions, and missing handler documentation. ✅ 5 resolved✅ Bug: startTime/endTime accepted but never forwarded in export endpoints
✅ Bug: ES Painless script unconditionally overwrites createdAt with old value
✅ Quality: Duplicate LineageTimeRange interface definition
✅ Quality: EXTERNAL_HANDLER bare
|
| Compact |
|
Was this helpful? React with 👍 / 👎 | Gitar



I worked on temporal lineage traversal because lineage views need to show edge state across historical time windows without disrupting the graph viewport.
High-level design:
Adds temporal fields to lineage APIs, cache keys, search mappings, ES/OS graph traversal, and lineage change events, then wires the UI time filter so lineage can be queried by range or point in time while preserving the rendered graph when switching between Lineage and Impact Analysis.
Tests:
Use cases covered
Historical lineage traversal returns edges active in the selected time window.
Lineage viewport centers/zooms on first load and does not redraw unnecessarily when switching Impact Analysis back to Lineage.
Sample ingestion data includes older and moved lineage edges for temporal traversal validation.
Unit tests
I added unit tests for the new/changed logic.
Files added/updated: LineageProvider.test.tsx, EntityLineageUtils.test.tsx, lineageAPI.test.ts, LineageCacheKeyTest.java.
Backend integration tests
I added integration tests for lineage temporal traversal.
Files added/updated: LineageResourceIT.java, OpenLineageLineageResolutionIT.java.
Ingestion integration tests
Not applicable; sample ingestion lineage data was updated.
Playwright (UI) tests
Not applicable; focused Jest coverage was added for the UI behavior.
Manual testing performed
Seeded local sample lineage edges through the REST API and verified different time windows return different active edges.
Verified last 7, 30, 90, and 180 day lineage windows for the ecommerce sample lineage graph.
temporal_lineage.mov
Summary by Gitar
applyTemporalFieldsinLineageRepositoryto maintaincreatedAt/createdByand trackupdatedAt/updatedByacross lineage re-emissions.decideEventTypeto triggerENTITY_LINEAGE_UPDATEDevents when lineage metadata or pipeline details change.ESLineageGraphBuilderandOSLineageGraphBuilderto supportstartTimeandendTimefilters in graph traversal logic.LineageResourceendpoints, including export and pagination APIs.SampleDataSourceto support ingestion of temporal fields (createdAt,updatedAt) andTempLineageTablemetadata.AuditLogRepositoryTestto verify that lineage lifecycle changes are correctly persisted as audit events.LineageTimeFilterto UI to allow users to toggle range or point-in-time traversal filters.EdgeInfoDrawerfields to display temporal provenance metadata.This will update automatically on new commits.