fix(ingest/datahub): Create Structured property templates in advance and batch processing #13355

treff7es · 2025-04-29T07:47:45Z

Structured properties ingested first to make sure template created in advance.
Add process database data in batches to make sure it won't run out of memory on db side

codecov · 2025-04-29T07:50:21Z

Codecov Report

Attention: Patch coverage is 57.97101% with 29 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ngestion/source/datahub/datahub_database_reader.py	57.57%	28 Missing ⚠️
...datahub/ingestion/source/datahub/datahub_source.py	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (57.97%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.

📢 Thoughts on this report? Let us know!

pedro93 · 2025-04-29T08:23:14Z

metadata-ingestion/src/datahub/ingestion/source/datahub/datahub_database_reader.py

+        )
+
+        time.sleep(
+            self.config.structured_properties_template_cache_invalidation_interval


This time should be based on on how many structured properties exist.

pedro93

LGTM, assuming green CI.

Ideally we could use something like: https://docs.datahub.com/docs/advanced/api-tracing to know when structured properties have been processed.

pedro93 · 2025-04-30T12:20:08Z

metadata-ingestion/src/datahub/ingestion/source/datahub/config.py

+    )
+
+    query_timeout: Optional[int] = Field(
+        hidden_from_docs=True,


This should not be hidden from docs

pedro93 · 2025-04-30T12:53:13Z

metadata-ingestion/src/datahub/ingestion/source/datahub/datahub_database_reader.py

+            logger.debug("Fetching soft-deleted URNs")
+
+            # Use server-side cursor implementation
+            rows = self.execute_server_cursor(self.soft_deleted_urns_query, params)


@treff7es this query should also take into account the execution dates + limits & offsets. We can have ingestion sources emitting a lot of soft deletes so the disk issue on the server is also applicable here.

alwaysmeticulous · 2025-05-06T09:23:11Z

🔴 Meticulous spotted visual differences in 36 of 612 screens tested: view and approve differences detected.

Meticulous evaluated ~10 hours of user flows against your PR.

_{Last updated for commit 8afa715. This comment will update as new commits are pushed.}

asikowitz

Nicely done!

asikowitz · 2025-05-07T08:55:09Z

metadata-ingestion/src/datahub/ingestion/source/datahub/datahub_database_reader.py

-                            duplicate_found = True
-                            break
+                    if first_iteration:
+                        start_date = row.get("createdon", start_date)


… advance

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Apr 29, 2025

github-actions bot deployed to datahub-wheels (Preview) April 29, 2025 07:49 View deployment

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Apr 29, 2025

vercel bot had a problem deploying to Preview April 29, 2025 07:54 Failure

github-actions bot deployed to datahub-project-web-react (Preview) April 29, 2025 07:55 View deployment

pedro93 reviewed Apr 29, 2025

View reviewed changes

pedro93 approved these changes Apr 29, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Apr 29, 2025

github-actions bot deployed to datahub-wheels (Preview) April 29, 2025 15:47 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) April 29, 2025 15:50 View deployment

vercel bot had a problem deploying to Preview April 29, 2025 15:53 Failure

treff7es requested a review from pedro93 April 29, 2025 16:19

treff7es changed the title ~~fix(ingest/datahub): Create Structured property templates in advance~~ fix(ingest/datahub): Create Structured property templates in advance and batch processing Apr 29, 2025

github-actions bot deployed to datahub-wheels (Preview) April 29, 2025 16:21 View deployment

vercel bot had a problem deploying to Preview April 29, 2025 16:27 Failure

github-actions bot deployed to datahub-project-web-react (Preview) April 29, 2025 16:29 View deployment

github-actions bot deployed to datahub-wheels (Preview) April 29, 2025 16:46 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) April 29, 2025 16:49 View deployment

vercel bot had a problem deploying to Preview April 29, 2025 16:52 Failure

github-actions bot deployed to datahub-wheels (Preview) April 30, 2025 11:42 View deployment

vercel bot had a problem deploying to Preview April 30, 2025 11:46 Failure

github-actions bot deployed to datahub-project-web-react (Preview) April 30, 2025 11:59 View deployment

github-actions bot deployed to datahub-wheels (Preview) April 30, 2025 12:15 View deployment

pedro93 reviewed Apr 30, 2025

View reviewed changes

github-actions bot deployed to datahub-project-web-react (Preview) April 30, 2025 12:21 View deployment

vercel bot had a problem deploying to Preview April 30, 2025 12:28 Failure

pedro93 reviewed Apr 30, 2025

View reviewed changes

pedro93 mentioned this pull request Apr 30, 2025

fix(ingest): Pull structured properties before anything else #13342

Closed

github-actions bot deployed to datahub-wheels (Preview) May 6, 2025 09:19 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) May 6, 2025 09:22 View deployment

vercel bot had a problem deploying to Preview May 6, 2025 09:25 Failure

asikowitz approved these changes May 7, 2025

View reviewed changes

treff7es added 7 commits May 7, 2025 10:59

Structured properties ingested first to make sure template created in…

0d4652c

… advance

Add batched read to datahub source database reader

4c2d337

Small tweaks

8e9bf36

Fix mypy issue

462019d

Move to limit date range based scrolling

c69ea70

Small fixes

4806dfb

pr review fixes

3f01d68

treff7es force-pushed the datahub_structured_property_fix branch from 8afa715 to 3f01d68 Compare May 7, 2025 08:59

github-actions bot deployed to datahub-wheels (Preview) May 7, 2025 09:10 View deployment

vercel bot deployed to Preview May 7, 2025 09:23 View deployment

pedro93 added the publish-docker label May 7, 2025

Merge branch 'master' into datahub_structured_property_fix

fc29bbc

vercel bot deployed to Preview May 7, 2025 12:43 View deployment

github-actions bot deployed to datahub-wheels (Preview) May 7, 2025 13:00 View deployment

Merge branch 'master' into datahub_structured_property_fix

c1a386e

github-actions bot deployed to datahub-wheels (Preview) May 7, 2025 15:36 View deployment

vercel bot deployed to Preview May 7, 2025 15:42 View deployment

Merge branch 'master' into datahub_structured_property_fix

ef492dd

github-actions bot deployed to datahub-wheels (Preview) May 9, 2025 13:24 View deployment

vercel bot deployed to Preview May 9, 2025 13:38 View deployment

Merge branch 'master' into datahub_structured_property_fix

c211ad1

github-actions bot deployed to datahub-wheels (Preview) May 15, 2025 11:16 View deployment

vercel bot deployed to Preview May 15, 2025 11:33 View deployment

treff7es merged commit 2ffa84b into master May 26, 2025
74 of 75 checks passed

treff7es deleted the datahub_structured_property_fix branch May 26, 2025 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(ingest/datahub): Create Structured property templates in advance and batch processing #13355

fix(ingest/datahub): Create Structured property templates in advance and batch processing #13355

Uh oh!

treff7es commented Apr 29, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 29, 2025 •

edited

Loading

Uh oh!

pedro93 Apr 29, 2025

Uh oh!

pedro93 left a comment

Uh oh!

pedro93 Apr 30, 2025

Uh oh!

asikowitz May 5, 2025

Uh oh!

pedro93 Apr 30, 2025

Uh oh!

alwaysmeticulous bot commented May 6, 2025 •

edited

Loading

Uh oh!

asikowitz left a comment

Uh oh!

asikowitz May 7, 2025

Uh oh!

Uh oh!

Uh oh!

fix(ingest/datahub): Create Structured property templates in advance and batch processing #13355

fix(ingest/datahub): Create Structured property templates in advance and batch processing #13355

Uh oh!

Conversation

treff7es commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pedro93 Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

pedro93 left a comment

Choose a reason for hiding this comment

Uh oh!

pedro93 Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

asikowitz May 5, 2025

Choose a reason for hiding this comment

Uh oh!

pedro93 Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

alwaysmeticulous bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asikowitz left a comment

Choose a reason for hiding this comment

Uh oh!

asikowitz May 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

treff7es commented Apr 29, 2025 •

edited

Loading

codecov bot commented Apr 29, 2025 •

edited

Loading

alwaysmeticulous bot commented May 6, 2025 •

edited

Loading