FR: Skip re-downloading files from Storage

[surajbarkale]

Feature proposal

• Firebase Component: Storage

When using writeToFile, a file is alway downloaded fresh from the Storage. In my app, I download a lot of files and most of them are unchanged. Which results in a lot of traffic. I was getting files from S3 previously and used following caching strategy to speed things up:

1. When a file is downloaded, get the value of Etag head field
2. Save the Etag values as extended attribute (using setxattr()) with XATTR_FLAG_CONTENT_DEPENDENT flag
3. When the file is downloaded again, get the Etag extended attribute
4. If the attribute is present, set header If-None-Match to the etag value
5. If server returns 304, treat it as success

Since the Storage APIs hide the HTTP headers and response codes, I can not implement this strategy in my code. I am proposing to add a option checkEtgBeforeDownload and implement this algorithm within Firebase SDK.

[schmidt-sebastian]

Thanks for filing this. You can probably do this already if you issue a getMetadata request and look at the generation number:

firebase-ios-sdk/FirebaseStorage/Sources/FIRStorageConstants.m

Line 68 in 2c1f579

NSString *const kFIRStorageMetadataGeneration = @"generation";

@tonyjhuang might be able to provide more guidance.

[surajbarkale]

Using getMetadata means two requests per file. Which adds up when downloading a lot of files. In my case, since the file sizes were small, two requests took effectively double the time.

[surajbarkale]

Here is my code if for reference:

StorageReference Extension

extension StorageReference {
  func download(to localUrl: URL) async throws {
    let remoteEtag: String = try await withCheckedThrowingContinuation { continuation in
      self.getMetadata { metadata, error in
        if let error = error {
          continuation.resume(with: .failure(error))
        } else {
          continuation.resume(with: .success(metadata?.md5Hash ?? ""))
        }
      }
    }
    guard remoteEtag != getXattr(path: localUrl.path, .eTag) else {
      return
    }
    try await self.write(toFile: localUrl)
    do {
      try setXattr(path: localUrl.path, .eTag, to: remoteEtag)
    } catch let error {
      print("Failed to set ETag for \(localUrl): \(error)")
    }
  }

  func write(toFile localUrl: URL) async throws {
    try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
      self.write(toFile: localUrl) { _, error in
        if let error = error {
          continuation.resume(with: .failure(error))
        } else {
          continuation.resume(with: .success(()))
        }
      }
    }
  }
}

Extended Attributes

import System
import Foundation

public struct ExtendedAttribute: Equatable {
  public struct Flags: OptionSet {
    public var rawValue: xattr_flags_t

    public init(rawValue: xattr_flags_t) {
      self.rawValue = rawValue
    }

    public static let noExport = Self(rawValue: XATTR_FLAG_NO_EXPORT)
    public static let syncable = Self(rawValue: XATTR_FLAG_SYNCABLE)
    public static let neverPreserve = Self(rawValue: XATTR_FLAG_NEVER_PRESERVE)
    public static let contentDependant = Self(rawValue: XATTR_FLAG_CONTENT_DEPENDENT)
  }

  public var name: String
  public var flags: Flags

  public init(name: String, flags: ExtendedAttribute.Flags) {
    self.name = name
    self.flags = flags
  }
}

extension ExtendedAttribute {
  public static let eTag = Self(name: "etag", flags: [.contentDependant, .syncable])
}

func getXattr(path: String, _ attr: ExtendedAttribute) -> String? {
  getXattr(path: path, attr)
    .flatMap { String(data: $0, encoding: .utf8) }
}

func getXattr(path: String, _ attr: ExtendedAttribute) -> Data? {
  path.withCString { path in
    let name = xattr_name_with_flags(attr.name, attr.flags.rawValue)
    defer { free(name) }
    let len = getxattr(path, name, nil, 0, 0, 0)
    guard len >= 0 else { return nil }
    let value = [UInt8].init(unsafeUninitializedCapacity: len) { (buf, bufLen) in
        bufLen = len
        getxattr(path, name, .init(buf.baseAddress!), len, 0, 0)
      }
    return Data(value)
  }
}

func setXattr(path: String, _ attr: ExtendedAttribute, to value: String?) throws {
  try setXattr(path: path, attr, to: value?.data(using: .utf8))
}

func setXattr(path: String, _ attr: ExtendedAttribute, to value: Data?) throws {
  let errorNum: Int32 = path.withCString { path in
    if let value = value {
      let name = xattr_name_with_flags(attr.name, attr.flags.rawValue)
      defer { free(name) }
      let res = value.withUnsafeBytes {
        setxattr(path, name, $0.baseAddress, $0.count, 0, 0)
      }
      return res < 0 ? errno : 0
    } else {
      let status = removexattr(path, attr.name, 0)
      return (status < 0 && status != -ENOATTR) ? errno : 0
    }
  }
  if errorNum != 0 {
    throw NSError(
      domain: NSPOSIXErrorDomain, code: Int(errorNum),
      userInfo: [NSLocalizedDescriptionKey: String(cString: strerror(errorNum))]
    )
  }
}

[tonyjhuang]

Hi @surajbarkale thanks for filing this FR. We will consider adding this to our sdks, for now, is it feasible to implement your own client-side cache of images to avoid redundant downloads?