Skip to content

Add temporal lineage traversal support - #28426

Merged
harshach merged 28 commits into
mainfrom
temporal_lineage
Jun 5, 2026
Merged

Add temporal lineage traversal support#28426
harshach merged 28 commits into
mainfrom
temporal_lineage

Conversation

@harshach

@harshach harshach commented May 26, 2026

Copy link
Copy Markdown
Collaborator

I worked on temporal lineage traversal because lineage views need to show edge state across historical time windows without disrupting the graph viewport.

High-level design:

Adds temporal fields to lineage APIs, cache keys, search mappings, ES/OS graph traversal, and lineage change events, then wires the UI time filter so lineage can be queried by range or point in time while preserving the rendered graph when switching between Lineage and Impact Analysis.

Tests:

Use cases covered

Historical lineage traversal returns edges active in the selected time window.
Lineage viewport centers/zooms on first load and does not redraw unnecessarily when switching Impact Analysis back to Lineage.
Sample ingestion data includes older and moved lineage edges for temporal traversal validation.
Unit tests

I added unit tests for the new/changed logic.
Files added/updated: LineageProvider.test.tsx, EntityLineageUtils.test.tsx, lineageAPI.test.ts, LineageCacheKeyTest.java.
Backend integration tests

I added integration tests for lineage temporal traversal.
Files added/updated: LineageResourceIT.java, OpenLineageLineageResolutionIT.java.
Ingestion integration tests

Not applicable; sample ingestion lineage data was updated.
Playwright (UI) tests

Not applicable; focused Jest coverage was added for the UI behavior.
Manual testing performed

Seeded local sample lineage edges through the REST API and verified different time windows return different active edges.
Verified last 7, 30, 90, and 180 day lineage windows for the ecommerce sample lineage graph.

temporal_lineage.mov

Summary by Gitar

  • Backend lineage persistence:
    • Integrated applyTemporalFields in LineageRepository to maintain createdAt/createdBy and track updatedAt/updatedBy across lineage re-emissions.
    • Added decideEventType to trigger ENTITY_LINEAGE_UPDATED events when lineage metadata or pipeline details change.
  • Search & API enhancements:
    • Updated ESLineageGraphBuilder and OSLineageGraphBuilder to support startTime and endTime filters in graph traversal logic.
    • Expose temporal filters in LineageResource endpoints, including export and pagination APIs.
  • Temporal ingestion:
    • Updated SampleDataSource to support ingestion of temporal fields (createdAt, updatedAt) and TempLineageTable metadata.
  • Audit logging:
    • Added AuditLogRepositoryTest to verify that lineage lifecycle changes are correctly persisted as audit events.
  • UI components:
    • Added LineageTimeFilter to UI to allow users to toggle range or point-in-time traversal filters.
    • Added EdgeInfoDrawer fields to display temporal provenance metadata.

This will update automatically on new commits.

Copilot AI review requested due to automatic review settings May 26, 2026 06:03
@harshach
harshach requested review from a team as code owners May 26, 2026 06:03
@github-actions github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels May 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

@github-actions

Copy link
Copy Markdown
Contributor

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@gitar-bot

gitar-bot Bot commented May 26, 2026

Copy link
Copy Markdown
Code Review ⚠️ Changes requested 0 resolved / 3 findings

Implements temporal lineage traversal across APIs and UI, but fails to propagate time parameters in export endpoints and contains a bug in the ES update script that incorrectly overwrites createdAt values.

⚠️ Bug: startTime/endTime accepted but never forwarded in export endpoints

📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:491-505 📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:632-646

Both exportLineageAsync (line 508) and exportLineageByEntityCountAsync (line 649) accept startTime/endTime query parameters but never pass them to the underlying DAO methods (dao.exportCsvAsync and dao.exportByEntityCountCsvAsync). Users calling these endpoints with time filters will receive unfiltered exports, which is silently incorrect behavior.

Either forward startTime/endTime to the DAO calls (requires updating the DAO method signatures), or remove the parameters from the endpoint signatures until the backend supports them, to avoid misleading callers.
// For exportLineageAsync — forward the params:
String csvData = dao.exportCsvAsync(
    fqn, upstreamDepth, downstreamDepth, queryFilter, entityType, deleted, startTime, endTime);

// For exportLineageByEntityCountAsync — forward the params:
String csvData = dao.exportByEntityCountCsvAsync(
    fqn, direction, from, size, nodeDepth, maxDepth, queryFilter, deleted, entityType,
    includeSourceFields, startTime, endTime);
⚠️ Bug: ES Painless script unconditionally overwrites createdAt with old value

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java:373-377

In the ES update script (SearchClient.java lines 373-377), when an edge already exists the script always carries forward the old createdAt/createdBy values into the new edgeData, regardless of whether the new edgeData already contains a lower createdAt. The Java-side applyTemporalFields in LineageRepository already applies min/max logic and sets the correct createdAt on the document being indexed. By unconditionally overwriting edgeData.createdAt with the old stored value, the script defeats the min-semantics — e.g. if a late-arriving earlier event sets createdAt to an earlier timestamp, ES will discard it and keep the later one.

This means temporal replay (out-of-order event ingestion) will not correctly minimize createdAt in the search index, even though the relational DB record is correct.

Apply min-semantics for createdAt in the Painless script: only carry forward the old value if it is earlier than what the new edgeData already provides.
def old = ctx._source.upstreamLineage[i];
def carryCreatedAt = old.get('createdAt');
def carryCreatedBy = old.get('createdBy');
def newCreatedAt = edgeData.get('createdAt');
if (carryCreatedAt != null && (newCreatedAt == null || carryCreatedAt < newCreatedAt)) {
  edgeData.put('createdAt', carryCreatedAt);
  if (carryCreatedBy != null) edgeData.put('createdBy', carryCreatedBy);
}
💡 Quality: Duplicate LineageTimeRange interface definition

📄 openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.interface.ts:14-17 📄 openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.interface.tsx:44-47

LineageTimeRange is identically defined in both LineageTimeFilter.interface.ts (line 14-17) and LineageProvider.interface.tsx (line 44-47). This creates a maintenance risk — if the interface evolves, both must be updated in sync. The filter component should import from the provider interface (the canonical location) instead of re-declaring it.

Remove the duplicate from LineageTimeFilter.interface.ts and import it from the provider interface instead.
// In LineageTimeFilter.interface.ts
import { LineageTimeRange } from '../../../context/LineageProvider/LineageProvider.interface';

export interface LineageTimeFilterProps {
  startTime?: number;
  endTime?: number;
  onChange: (range: LineageTimeRange) => void;
}
🤖 Prompt for agents
Code Review: Implements temporal lineage traversal across APIs and UI, but fails to propagate time parameters in export endpoints and contains a bug in the ES update script that incorrectly overwrites createdAt values.

1. ⚠️ Bug: startTime/endTime accepted but never forwarded in export endpoints
   Files: openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:491-505, openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:632-646

   Both `exportLineageAsync` (line 508) and `exportLineageByEntityCountAsync` (line 649) accept `startTime`/`endTime` query parameters but never pass them to the underlying DAO methods (`dao.exportCsvAsync` and `dao.exportByEntityCountCsvAsync`). Users calling these endpoints with time filters will receive unfiltered exports, which is silently incorrect behavior.

   Fix (Either forward startTime/endTime to the DAO calls (requires updating the DAO method signatures), or remove the parameters from the endpoint signatures until the backend supports them, to avoid misleading callers.):
   // For exportLineageAsync — forward the params:
   String csvData = dao.exportCsvAsync(
       fqn, upstreamDepth, downstreamDepth, queryFilter, entityType, deleted, startTime, endTime);
   
   // For exportLineageByEntityCountAsync — forward the params:
   String csvData = dao.exportByEntityCountCsvAsync(
       fqn, direction, from, size, nodeDepth, maxDepth, queryFilter, deleted, entityType,
       includeSourceFields, startTime, endTime);

2. ⚠️ Bug: ES Painless script unconditionally overwrites createdAt with old value
   Files: openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java:373-377

   In the ES update script (SearchClient.java lines 373-377), when an edge already exists the script always carries forward the old `createdAt`/`createdBy` values into the new `edgeData`, regardless of whether the new `edgeData` already contains a *lower* `createdAt`. The Java-side `applyTemporalFields` in `LineageRepository` already applies min/max logic and sets the correct `createdAt` on the document being indexed. By unconditionally overwriting `edgeData.createdAt` with the old stored value, the script defeats the min-semantics — e.g. if a late-arriving earlier event sets `createdAt` to an earlier timestamp, ES will discard it and keep the later one.
   
   This means temporal replay (out-of-order event ingestion) will not correctly minimize `createdAt` in the search index, even though the relational DB record is correct.

   Fix (Apply min-semantics for createdAt in the Painless script: only carry forward the old value if it is earlier than what the new edgeData already provides.):
   def old = ctx._source.upstreamLineage[i];
   def carryCreatedAt = old.get('createdAt');
   def carryCreatedBy = old.get('createdBy');
   def newCreatedAt = edgeData.get('createdAt');
   if (carryCreatedAt != null && (newCreatedAt == null || carryCreatedAt < newCreatedAt)) {
     edgeData.put('createdAt', carryCreatedAt);
     if (carryCreatedBy != null) edgeData.put('createdBy', carryCreatedBy);
   }

3. 💡 Quality: Duplicate LineageTimeRange interface definition
   Files: openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.interface.ts:14-17, openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.interface.tsx:44-47

   `LineageTimeRange` is identically defined in both `LineageTimeFilter.interface.ts` (line 14-17) and `LineageProvider.interface.tsx` (line 44-47). This creates a maintenance risk — if the interface evolves, both must be updated in sync. The filter component should import from the provider interface (the canonical location) instead of re-declaring it.

   Fix (Remove the duplicate from LineageTimeFilter.interface.ts and import it from the provider interface instead.):
   // In LineageTimeFilter.interface.ts
   import { LineageTimeRange } from '../../../context/LineageProvider/LineageProvider.interface';
   
   export interface LineageTimeFilterProps {
     startTime?: number;
     endTime?: number;
     onChange: (range: LineageTimeRange) => void;
   }

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds temporal lineage traversal so lineage APIs, search traversal, and the UI can filter lineage edges by historical time windows while preserving lineage viewport state.

Changes:

  • Adds startTime/endTime support across lineage request schemas, REST parameters, cache keys, ES/OS traversal, and sample lineage data.
  • Adds lineage change-event types and emits add/update/delete lineage events.
  • Adds a UI lineage time filter, edge audit display fields, URL-backed time filter state, and tests for lineage fetch/centering behavior.

Reviewed changes

Copilot reviewed 134 out of 145 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java Adds temporal query params to lineage endpoints.
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchUtils.java Adds time-window filtering for ES lineage edges.
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java Carries lineage temporal fields during search document updates.
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ESLineageGraphBuilder.java Applies temporal filtering during ES lineage traversal/counting.
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OSLineageGraphBuilder.java Applies temporal filtering during OpenSearch lineage traversal/counting.
openmetadata-service/src/main/java/org/openmetadata/service/search/lineage/LineageCacheKey.java Adds time-window fields to lineage cache keys.
openmetadata-service/src/test/java/org/openmetadata/service/search/lineage/LineageCacheKeyTest.java Updates cache key tests for new constructor fields.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java Preserves lineage temporal metadata and emits lineage change events.
openmetadata-service/src/main/java/org/openmetadata/service/openlineage/OpenLineageMapper.java Maps OpenLineage event time into lineage details.
openmetadata-service/src/main/java/org/openmetadata/service/util/LineageUtil.java Updates search delete lookup for lineage doc IDs.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/OpenLineageLineageResolutionIT.java Adds OpenLineage temporal metadata integration tests.
openmetadata-spec/src/main/resources/json/schema/type/changeEventType.json Adds lineage add/update/delete event types.
openmetadata-spec/src/main/resources/json/schema/api/lineage/searchLineageRequest.json Adds temporal request fields for lineage search.
openmetadata-spec/src/main/resources/json/schema/api/lineage/entityCountLineageRequest.json Adds temporal request fields for entity-count lineage.
openmetadata-spec/src/main/resources/elasticsearch/{en,jp,ru,zh}/* Adds lineage temporal/audit fields to search mappings.
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.tsx Adds URL-backed time filter state and fetch-key handling.
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.interface.tsx Exposes lineage time filter in context.
openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.test.tsx Adds lineage/impact-analysis fetch behavior tests.
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.component.tsx Adds lineage time filter UI.
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.interface.ts Defines time filter props and presets.
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/CustomControls.component.tsx Renders the lineage time filter in controls.
openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityInfoDrawer/EdgeInfoDrawer.component.tsx Shows lineage edge created/updated metadata.
openmetadata-ui/src/main/resources/ui/src/components/Lineage/EntityLineageTab/EntityLineageTab.tsx Preserves lineage graph DOM when switching views.
openmetadata-ui/src/main/resources/ui/src/components/Lineage/Lineage.interface.ts Adds edge audit fields to UI edge details.
openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.ts Threads temporal params through lineage API calls.
openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.test.ts Updates lineage API request expectations.
openmetadata-ui/src/main/resources/ui/src/utils/EntityLineageUtils.tsx Fixes node-centering width precedence.
openmetadata-ui/src/main/resources/ui/src/utils/EntityLineageUtils.test.tsx Tests corrected lineage node centering.
openmetadata-ui/src/main/resources/ui/src/locale/languages/*.json Adds translated labels for temporal lineage UI.
openmetadata-ui/src/main/resources/ui/package.json Adds a Rollup native package dev dependency.
openmetadata-ui/src/main/resources/ui/yarn.lock Updates lockfile entries.
ingestion/src/metadata/ingestion/source/database/sample_data.py Supports temporal lineage details in sample ingestion.
ingestion/examples/sample_data/lineage/lineage.json Adds temporal metadata to sample lineage edges.

Comment thread openmetadata-ui/src/main/resources/ui/package.json Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 06:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 138 out of 149 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ESLineageGraphBuilder.java:279

  • The time filter is applied only after the downstream search hit has already been added to result.getNodes() and queued in hasToFqnMapForLayer. If a downstream entity matches the structural query because of an inactive edge from the current layer, but all of its overlapping edges are filtered out here, the node is still displayed and traversed as if it were active. Build the filtered matching edge list first and only add/queue the node when at least one edge both overlaps the time window and connects to the current layer.
          List<EsLineageData> upstreamEntities =
              getUpstreamLineageListIfExist(
                  entityMap, lineageRequest.getStartTime(), lineageRequest.getEndTime());
          for (EsLineageData esLineageData : upstreamEntities) {
            if (hasToFqnMap.containsKey(esLineageData.getFromEntity().getFqnHash())) {
              result
                  .getDownstreamEdges()
                  .putIfAbsent(esLineageData.getDocId(), esLineageData.withToEntity(toEntity));

openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OSLineageGraphBuilder.java:279

  • The time filter is applied only after the downstream search hit has already been added to result.getNodes() and queued in hasToFqnMapForLayer. If a downstream entity matches the structural query because of an inactive edge from the current layer, but all of its overlapping edges are filtered out here, the node is still displayed and traversed as if it were active. Build the filtered matching edge list first and only add/queue the node when at least one edge both overlaps the time window and connects to the current layer.
          List<EsLineageData> upstreamEntities =
              getUpstreamLineageListIfExist(
                  entityMap, lineageRequest.getStartTime(), lineageRequest.getEndTime());
          for (EsLineageData esLineageData : upstreamEntities) {
            if (hasToFqnMap.containsKey(esLineageData.getFromEntity().getFqnHash())) {
              result
                  .getDownstreamEdges()
                  .putIfAbsent(esLineageData.getDocId(), esLineageData.withToEntity(toEntity));

Comment thread openmetadata-ui/src/main/resources/ui/package.json Outdated
Comment thread openmetadata-ui/src/main/resources/ui/playwright/e2e/Flow/LineageSettings.spec.ts Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 06:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 138 out of 149 changed files in this pull request and generated 2 comments.

@mohityadav766 mohityadav766 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporal lineage — review notes

Overall the plumbing is clean and consistent: the ES and OS builders mirror each other, the cache key is extended with the window, validateTemporalBounds guards startTime > endTime, and there's solid IT/unit coverage. One interaction I want to flag, though.

Preserve-paths does not extend to the time window

A path-preservation user expects that a node which doesn't itself match the filter is still shown when something in its upstream/downstream qualifies. That holds for the node-level query filter, but not for the new startTime/endTime window — and the combination is currently a silent no-op for the temporal dimension.

1. The time window hard-prunes during traversal. In fetchDownstreamNodesRecursively:

// ESLineageGraphBuilder.java:266-274
List<EsLineageData> upstreamEntities =
    getMatchingUpstreamLineageData(entityMap, hasToFqnMap.keySet(),
        lineageRequest.getStartTime(), lineageRequest.getEndTime());
if (upstreamEntities.isEmpty()) {
  continue;   // node dropped AND not enqueued for the next layer
}

For a path root → A → (edge out of window) → B → (edge in window) → C: at B the connecting edge A→B fails the window, so B is skipped and never added to the next frontier. The BFS never explores B's downstream, so edge B→C and node C are never discovered — even though B→C is inside the window. The same hard-prune pattern exists in the upstream branch and in the depth-count BFS (getDepthWiseEntityCounts, allEntitiesUpToDepth). An out-of-window edge severs discovery of everything behind it, in both directions.

2. Preserve-path runs afterward, on an already-time-pruned graph. Path preservation deliberately strips the node-level query filter from the base traversal so the full topology is available, then filters in-memory:

// ESLineageGraphBuilder.java:325-333
SearchLineageRequest unfilteredRequest =
    JsonUtils.deepCopy(lineageRequest, SearchLineageRequest.class)
        .withQueryFilter(getStructuralFilterOnly(lineageRequest.getQueryFilter()));
result = searchLineageWithStrategyInternal(unfilteredRequest);
result = applyInMemoryFiltersWithPathPreservation(result, lineageRequest);

The deepCopy overrides only queryFilterstartTime/endTime survive into unfilteredRequest. So the "unfiltered" base graph handed to LineagePathPreserver is already time-pruned, and the preserver can only trace paths through edges that still exist (buildParentAdjacencyMap walks existing edges only). It cannot resurrect B or C.

Net: the query filter is excluded from the base traversal and applied in post-processing (preserve-able); the time window is baked into the base traversal as a hard edge prune (not preserve-able). A user passing preservePaths=true + a time window gets no path preservation for the temporal dimension, with no error or signal.

Recommendation

Decide the intended semantic and make it explicit:

  • If the window should honor preserve-paths (keep an out-of-window intermediate so an in-window up/downstream node still shows): move time filtering out of the inline BFS into the same post-processing path as the node filter. Concretely — also strip startTime/endTime from unfilteredRequest, traverse structurally, then in the in-memory step collect endpoints of in-window edges as "matching nodes" and feed them to LineagePathPreserver.preservePathsWithEdges(...), exactly as the column/node filter does.
  • If hard pruning is intended (show the graph as of the window, severing through stale edges): that's defensible, but it should be documented on the startTime/endTime schema fields, and it should be explicit that preservePaths does not extend to the temporal filter — otherwise the silent interaction is confusing.

Smaller notes

  • SearchUtils.edgeMatchesWindow: legacy edges with both timestamps null always match any window. Reasonable for back-compat, but a window query over mixed legacy + temporal data returns all legacy edges — worth calling out in the API docs.
  • Minor directional asymmetry: downstream drops the candidate node entirely on an out-of-window connecting edge (continue), while upstream keeps the frontier node and only prunes onward edges. Same path-severing effect, but node retention differs between directions — worth a sanity check.

Addresses review on PR #28426: the startTime/endTime window is applied as
a hard prune during graph traversal, not as a preserve-paths post-filter.
An out-of-window edge severs discovery of everything reachable only
through it, so preservePaths does not extend to the temporal dimension.

- Document the hard-prune + preservePaths interaction and the legacy
  null-timestamp back-compat behavior on startTime/endTime in both
  searchLineageRequest.json and entityCountLineageRequest.json.
- Add code comments at the downstream prune sites (ES + OS builders)
  capturing the shared invariant: a non-root node appears iff reached by
  at least one in-window connecting edge. Upstream gates this at enqueue
  time, downstream via the empty-list continue; net behavior is symmetric.
Copilot AI review requested due to automatic review settings June 2, 2026 15:52
mohityadav766
mohityadav766 previously approved these changes Jun 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 149 out of 159 changed files in this pull request and generated 4 comments.

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

✅ TypeScript Types Auto-Updated

The generated TypeScript types have been automatically updated based on JSON schema changes in this PR.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 149 out of 159 changed files in this pull request and generated 1 comment.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 151 out of 161 changed files in this pull request and generated 3 comments.

@sonarqubecloud

sonarqubecloud Bot commented Jun 4, 2026

Copy link
Copy Markdown

@sonarqubecloud

sonarqubecloud Bot commented Jun 4, 2026

Copy link
Copy Markdown

@gitar-bot

gitar-bot Bot commented Jun 5, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 5 resolved / 5 findings

Implements temporal lineage traversal across the backend, search indexing, and UI components to enable historical edge state viewing. Resolved issues include incorrect export endpoint parameter forwarding, faulty ES script overwrites, duplicate interface definitions, and missing handler documentation.

✅ 5 resolved
Bug: startTime/endTime accepted but never forwarded in export endpoints

📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:491-505 📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java:632-646
Both exportLineageAsync (line 508) and exportLineageByEntityCountAsync (line 649) accept startTime/endTime query parameters but never pass them to the underlying DAO methods (dao.exportCsvAsync and dao.exportByEntityCountCsvAsync). Users calling these endpoints with time filters will receive unfiltered exports, which is silently incorrect behavior.

Bug: ES Painless script unconditionally overwrites createdAt with old value

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java:373-377
In the ES update script (SearchClient.java lines 373-377), when an edge already exists the script always carries forward the old createdAt/createdBy values into the new edgeData, regardless of whether the new edgeData already contains a lower createdAt. The Java-side applyTemporalFields in LineageRepository already applies min/max logic and sets the correct createdAt on the document being indexed. By unconditionally overwriting edgeData.createdAt with the old stored value, the script defeats the min-semantics — e.g. if a late-arriving earlier event sets createdAt to an earlier timestamp, ES will discard it and keep the later one.

This means temporal replay (out-of-order event ingestion) will not correctly minimize createdAt in the search index, even though the relational DB record is correct.

Quality: Duplicate LineageTimeRange interface definition

📄 openmetadata-ui/src/main/resources/ui/src/components/Entity/EntityLineage/LineageTimeFilter.interface.ts:14-17 📄 openmetadata-ui/src/main/resources/ui/src/context/LineageProvider/LineageProvider.interface.tsx:44-47
LineageTimeRange is identically defined in both LineageTimeFilter.interface.ts (line 14-17) and LineageProvider.interface.tsx (line 44-47). This creates a maintenance risk — if the interface evolves, both must be updated in sync. The filter component should import from the provider interface (the canonical location) instead of re-declaring it.

Quality: EXTERNAL_HANDLER bare return drops explanatory comment

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:2075-2077 📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:2129-2131 📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:2196-2198
The three case EXTERNAL_HANDLER branches were changed from a documented no-op (// No-op: a dedicated handler ... drives the cascade.) to a bare return;. Functionally this is equivalent — it just skips the trailing script.append(" "), which is harmless since no script is generated for this case. However, removing the comment makes the early return non-obvious: a future reader sees case EXTERNAL_HANDLER -> { return; } with no indication that propagation is intentionally handled elsewhere (e.g. propagateCertificationTags). Per the project's documentation guidance, comments are warranted for non-obvious control flow. Consider keeping a brief explanatory comment alongside the return;.

Bug: expect.poll(..).toBe(0) passes instantly, weak no-fetch assertion

📄 openmetadata-ui/src/main/resources/ui/playwright/e2e/Flow/LineageSettings.spec.ts:310-314
At line 311, expect.poll(() => lineageFetchesAfterViewSwitch).toBe(0) resolves immediately because the counter is already 0 when polling starts. It does not wait to confirm no request fires asynchronously after the tab switch. If the lineage fetch is triggered with a slight delay (e.g., via useEffect or debounce), this assertion would pass as a false-positive.

A more robust pattern is to add a short explicit wait before asserting, or to use toPass with a minimum polling interval to ensure the value stays 0 over a window of time.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants