[MLMD] Integrate LineageView into KFP

[kwasi]

/area metadata
/area front-end
/priority p0
/assign @kwasi
/cc @avdaredevil
/cc @Bobgy

Notes:

This is tracks the work to integrate the MLMD Lineage View, which is now available in a shared repository into kubeflow/pipelines. I'll add further details in the coming days. I have implemented this in a branch based on a checkpoint from 2020-01-08 in https://github.com/kwasi/pipelines/tree/lineage-in-kfp-wip. This is the branch we used for the bug bash, but it's fallen far behind the latest master.

The branch referenced above makes a number of assumptions about how code should be shared between the projects that haven't been reviewed or discussed. This issue should stay open until the frontend teams for kubeflow/pipelines and kubeflow/metadata meet to discuss how to share code and should wait for approval by Pipelines.

[k8s-ci-robot]

@kwasi: The label(s) area/metadata cannot be applied, because the repository doesn't have them

In response to this:

/area metadata
/area front-end
/priority p0
/assign @kwasi
/cc @avdaredevil
/cc @Bobgy

Notes:

This is tracks the work to integrate the MLMD Lineage View, which is now available in a shared repository into kubeflow/pipelines. I'll add further details in the coming days. I have implemented this in a branch based on a checkpoint from 2020-01-08 in https://github.com/kwasi/pipelines/tree/lineage-in-kfp-wip. This is the branch we used for the bug bash, but it's fallen far behind the latest master.

The branch referenced above makes a number of assumptions about how code should be shared between the projects that haven't been reviewed or discussed. This issue should stay open until the frontend teams for kubeflow/pipelines and kubeflow/metadata meet to discuss how to share code and should wait for approval by Pipelines.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

Great work for the bugbash!

Can you summarize how you did the integration? I am on a vacation for 9 days, so I don't have much time to review.

[Ark-kun]

Please make sure the Lineage viewer is compatible with the oncoming TFX changes - most artifact properties become custom_properties. Executions might follow suite later.
To prepare for this, the service should check both .properties and .custom_properties and use what's available.

[kwasi]

@Bobgy, Thanks! Here's a brief summary of the changes from a development point of view:

Most of the changes for the integration are in the ArtifactList, ArtifactDetails, ExecutionList, and ExecutionDetails pages. The ArtifactDetails page now has tabs and renders the LineageViewer and all pages use gRPC web protos instead of the Swagger API.

Since the LineageViewer uses gRPC web to fetch MLMD information, Pipelines now uses generated gRPC web shared from kubeflow/frontend instead of its own generated definitions. This is to keep the types and versions in sync between the shared MLMD LineageViewer and the MLMD data being shown in other parts of the app.

Other shared utilities for MLMD have been moved to the kubeflow/frontend repository so that kubeflow/metadata can benefit from improvements.

It may make sense to make the Artifact and Execution pages shared components as well. Currently kubeflow/metadata and kubeflow/pipelines are slightly out of sync where Pipelines has features like Artifact list on execution page that were built that haven't been backported, even though the pages are largely the same down to the routing. This is probably something we can tackle in Q2.

[kwasi]

@Ark-kun I backported some changes from Pipelines to the shared MLMD util. Do these changes cover the TFX property changes and are they / do they need to be backwards compatible?

kubeflow/frontend@756476a#diff-c09f8ecd27838a890829a3bb9d58a436

If you have documentation on the changes, I can also look at that to verify