[Metadata] Executions and artifacts produced by KF Pipelines should show in the UX

[Ark-kun]

The KFP UX has the Executions and Artifacts sections, but those sections remain empty when users run their pipelines.

The system should be adding all executions and their artifacts produced when running any KFP pipeline.
This system should use all information available: Input/output names and types, artifact URIs, execution component ID and name, etc.

The component metadata (input/output names and types) as well as the input/output parameters and artifacts is already included in compiled pipeline and is available to Backend, Frontend and other systems.

[rmgogogo]

TFX with MLMD can show it. Alex, is this you are looking for? As for more general MLMD data generation, I think it requires components support it, e.x. Xgboost component can call MLMD API to generate data.

One more step, we can impl Driver and Publisher for KFP components not from TFX. You may have a quick sync with Ajay. +Ajay to this ticket.

https://screenshot.googleplex.com/4kmDiZuc0Vm
https://screenshot.googleplex.com/6Ah1X2DSmLa

[rmgogogo]

changed to "area/sdk/client"

[Ark-kun]

TFX with MLMD can show it.

Yes. This issue is about generating the metadata from non-TFX components and pipelines.

I've created a proof of concept metadata writer that records all KFP executions and artifacts in the MLMD. There will be a design review soon.

[pepnova-9]

@Ark-kun

I was wondering why the Executions and Artifacts sections were blank even if a pipeline completed and produced artifacts successfully.
So, is it correct that for now, I cannot see any artifacts produced by KFP pipeline in the Artifacts sections unless I add those into MLMD by myself?

[Ark-kun]

So, is it correct that for now, I cannot see any artifacts produced by KFP pipeline in the Artifacts sections unless I add those into MLMD by myself?

Yes. But this will be fixed in next release.

[Ark-kun]

Fixed by #2674