File output is too long

[akranga]

Trying to define a conatinerop that downloads a file from minio bucket but getting `This step is in Error state with this message: failed to save outputs: Pod "xxxxxx" is invalid: metadata.annotations: Too long: must have at most 262144 characters.

See example:

dsl.ContainerOp(
        name=step_name,
        image='minio/mc',
        command=['mc', '--no-color'],
        arguments=['cp', 'minio/%s' % filename, '/tmp/results.txt'],
        file_outputs={'file': '/tmp/results.txt'},
    )

What am I doing wrong?

[hongye-sun]

I think it's probably because the results.txt file is too large. Argo doesn't report a good error message due to a bug (argoproj/argo-workflows#765). Argo supports two ways to pass data between steps: a) output, which is supposed to be small and can be embedded in the workflow template; b) artifact, which can be large and will be stored in the artifact store and copied to next steps.

However, in KFP, we only support output today. Artifact support is WIP. If you do want to pass large files across steps, you can use external storage like GCS and pass the remote file path as the output for next steps.

[akranga]

Yeah. I do not need to extract value from the file content. I do not need a long string. However i need to save it as a file "locallhy". I guess minio-service.kubeflow:9000 serves as a "local file cache". And then use it as an input for next containerop. Happy to get any directions there

[akranga]

Do you have any support of smth like simlar to argo definition

outputs:
  artifacts:
  - name: xyz
    path: xyz
    s3: ...

[hongye-sun]

Unfortunately, this support is still in progress. The suggested way is to use external storage. You have to download the file to local path before using it. You might also consider to use PVC to share files across steps. ContainerOp supports to set volume claim and mount.

[akranga]

I understand. However PVC should work for me.
Do you mind to point in a right example? Thanks!

[Ark-kun]

Do you have any support of smth like simlar to argo definition

outputs:
  artifacts:
  - name: xyz
    path: xyz
    s3: ...

See the "Support for Argo artifacts" issue #336

I'm actively working on making it work.

[Ark-kun]

/assign @Ark-kun

[TimCookny]

Please use LongPathTool for this issue.

[vicaire]

Resolving in favor of #336