Artifact representation in Pipeline

[animeshsingh]

Are there any thoughts on how to define custom artifacts in pipelines which different steps can consume? For e.g. if we need to bring in a model as a typed artifact object, which a pipeline can consume, and it can be defined in DSL

[Ark-kun]

A good way that also follows Argo would be to add the input_artifact_paths and output_artifact_paths parameters to the ContainerOp constructor, so that it's possible to specify the artifact names and file locations where the system should place or collect them.
output_artifact_paths={"out_art1": "/outputs/out_art1/data", ...}
input_artifact_paths={"in_art1": "/inputs/in_art1/data", ...}

See Argo example on how the artifact inputs/outputs and artifact passing is described in Argo: https://github.com/argoproj/argo/blob/master/examples/artifact-passing.yaml

Adding the artifact production support to the DSL compiler is easy.
Adding the artifact consumption support to the DSL compiler is more complicated as the compiler code is quite convoluted.

[hongye-sun]

@animeshsingh would you mind giving us more context of your use cases about typed artifact object?

We are in an effort to support artifact metadata (including location, type and runtime data) tracking in a metadata store. For consuming artifact inside a pipeline, we are still open to advice.

[Ark-kun]

Specifying output artifacts is now possible since
#998
Input artifacts are waiting for #791 to be merged .
Then we'll need to update the compiler to support passing output artifacts to artifact inputs.

[gaoning777]

@animeshsingh same as hongye, more context is welcome. Is this issue about defining custom artifact types or artifact passing, or both? Thanks

[Ark-kun]

Fixed by #791, #998 , #2173