feat(ingest/lineage): generate static json lineage file

[anshbansal]

Scope of the PR

To expose what are the aspects of an entity that are considered lineage in a static json file lineage.json

What has been changed

• CI and pre-commit hooks changed so that this lineage.json file remains up-to-date
• Code changed in modeldocgen.py so it generates this lineage file
• Tests added to increase coverage
• gradle task to generate this lineage json file

Why not server endpoint for lineage registry

An endpoint was recently added #13865. An obvious question is why is that not enough. There are few reasons

• It does not return enough granular details. i.e. there is no information on what field path's value being present means it is lineage
  
  vs
  

• I am unable to co-relate the values returned by the endpoint with our pdl files. e.g. upstreamEdges is returned by the endpoint locally but there is nothing with that name in pdl files. So where is this coming from? We don't have any docs here
  

• For anything that is offline analysis where we don't have access to a server e.g. if someone is doing offline analytics on graph exports they need to know what aspect's what field is lineage. This information needs to be hard-coded currently. But with this static json file being present they can do offline analysis by downloading this file from github.

• In case we wanted to remove the responsibility of getting this relationship information from modeldocgen this allows us a gradual path where we can remove the responsibility from there and depend on this json. That speeds up and simplifies modeldocgen code

future work planned

• marking how much lineage was produced in source report using this information. The changes will be mainly done in _populate_aspect_metrics method of metadata-ingestion/src/datahub/ingestion/api/source.py. This is avoid hard-coding what is considered lineage for every different aspect

Notes

• metadata-ingestion/src/datahub/ingestion/autogenerated/lineage_helper.py has been added. It is considered experimental for now and each method has been clearly marked as such. It is in this folder mainly to show this only works with folder's code. Also, it has mostly been auto-generated using AI code. It has mainly been added as a stop-gap measure to test some things. WIll mostly be tweaked when changes in source.py are done in a follow up PR

[codecov]

Codecov Report

Attention: Patch coverage is 98.03922% with 1 line in your changes missing coverage. Please review.

❌ Unsupported file format

Upload processing failed due to unsupported file format. Please review the parser error message:

Error parsing JUnit XML in /home/runner/work/datahub/datahub/metadata-io/build/test-results/test/TEST-com.linkedin.metadata.graph.search.elasticsearch.SearchGraphServiceElasticSearchTest.xml at 117:1058
Caused by:

RuntimeError: Error converting computed name to ValidatedString
Caused by:
    string is too long</code></pre>

For more help, visit our troubleshooting guide.

Files with missing lines	Patch %	Lines
.../datahub/ingestion/autogenerated/lineage_helper.py	98.03%	1 Missing ⚠️

:loudspeaker: Thoughts on this report? Let us know!

[pedro93]

LGTM assuming green CI