Using Encore.ts is it possible to copy objects in storage without downloading and reuploading them?

[tom-groves]

Currently when copying objects across buckets I'm downloading from the source bucket and re-uploading to the target bucket, but it would be preferable to avoid unnecessary data transfer.

My implementation is as follows:

import log from 'encore.dev/log';
import {
  Attrser,
  Downloader,
  Lister,
  Uploader,
} from 'encore.dev/storage/objects';
import { UnexpectedError } from '~common/domain/errors/unexpected-error.class';

export const copyObject = async (data: {
  sourceBucket: Downloader & Lister & Attrser;
  sourceKey: string;
  targetBucket: Uploader;
  targetKey: string;
  recursive?: boolean;
}) => {
  const {
    sourceBucket,
    sourceKey,
    targetBucket,
    targetKey,
    recursive = false,
  } = data;

  if (recursive) {
    const sourceObjects = sourceBucket.list({
      prefix: sourceKey,
    });
    for await (const sourceObject of sourceObjects) {
      await copyObject({
        sourceBucket,
        sourceKey: sourceObject.name,
        targetBucket,
        targetKey: sourceObject.name.replace(sourceKey, targetKey),
        recursive: false,
      });
    }
    return;
  }

  try {
    // TODO: This is not efficient, we should use the S3 copy API instead, but that is not exposed by encore.dev
    const sourceAttrs = await sourceBucket.attrs(sourceKey);
    const sourceObject = await sourceBucket.download(sourceKey);
    const targetObject = await targetBucket.upload(targetKey, sourceObject, {
      contentType: sourceAttrs.contentType,
    });

    return targetObject;
  } catch (error) {
    log.error('Error copying object', {
      error,
      sourceKey,
      targetKey,
    });

    throw new UnexpectedError(
      `Error copying object ${sourceKey} to ${targetKey}`,
      error,
    );
  }
};

Thanks in advance!

[simon-johansson]

Thanks for the detailed writeup. You are correct that the current Encore object storage API does not expose a server-side copy operation. This is a valid feature request -- cloud providers like S3 and GCS support server-side copy natively, which avoids downloading and re-uploading the data. We will track this as a feature request to add a copy method (or similar) to the bucket API that leverages the underlying cloud provider's copy functionality. In the meantime, your download-and-reupload approach is the correct workaround.