Project Input standardization (#48)

Upload Input Inspection and Standardization 1 (#17)

Add scaffolding

Add initializers

Add internal and external state mapping

Remove duplicate status enum and add inline docs to external status

Add inline API docs to PHAsset-based MuxUpload constructor

Consolidate all `MuxUpload` options into a single struct `UploadOptions`

Declare asynchronous MuxUpload constructor in PHAsset extension

Place extension methods into dedicated directories

Polish inline API documentation

Add new API documentation and note the placeholder implementation

Add option variants as static members: defaults, disabled inputStandardization

Deprecate existing initializer, normally this API should be removed prior to GA, but since it was the only initializer exposed up to this point removing it would break everybody. Instead deprecate and remove at a later date.

Store all MuxUpload-related options in UploadInfo

Use correct starting byte parameter when restarting upload

Hardcode request options when exporting AVAsset from PhotosKit

chunkSize -> chunkSizeInBytes

Input standardization 2 (#41)

Maintain handle to canonical input asset inside UploadInput

If input standardized, standardized input URL is passed to UploadInfo
instead of the original input URL used for initializer

Note: SDK probably needs to re-export a high quality asset anyway so
possibly need a bridging status

Add dedicated internal initializer for MuxUpload error with unknown error code

Disable input standardization when running tests

Add inspection logic

Request local and remote assets

Status -> TransportStatus

Rely on input status in MuxUpload computed property getters

Expose progress from internal upload state if available

Track transport status inside of UploadInput

Transport status start time optional

Add TransportStatus docs

Inspect and check video track frame rate

Migrate tests for MuxUpload input result handler

Input standardization 3 (#46)

Standardize via export session

Back out outputURL construction into MuxUpload

Expose hook for client to cancel upload if standardization failed

Call cancellation hook if inspection fails. We're not sure if the input is standard or not so better to be safe
and confirm

Export based on maximum resolution set by client

Cleaner non standard input handler invocation (#57)

Add CustomStringConvertible conformance to maximum resolution (#56)

Update input state when upload is finished (#63)

Update input state when the upload succeeds or fails

Tweak input state definition, keep an error in the failed case and the success struct in the success case

NFC: remove some if let checks for the handler closure properties in MuxUpload and replace with ? operator. Keeps the existing logic.

Replace video file get with actual implementation (#64)

Reporting updates (#49)

Only mark upload as started if its ready

Add upload failed and input standardization events

Add standardization failure event handler to reporter

Add standardization success event handler to reporter

Rename method for upload success

Rename upload event to upload succeeded

Support dispatching multiple events at the same time

Naive non-thread safe implementation reporter networking

Change event type casing, individual files for events

Fix reporter test

Actually report upload failure

Read off file size

Use Date in events and serialize to ISO8601 string

Finangle duration and start time

Set correct version in correct place

Keep a session UUID

Route every request through one chokepoint

Upload failed test

Shared test encoder

Input standardization failed test

Input standardization succeeded test

Better organize expected json strings

Use a shared instance

Remove dupe file

Thread asset duration through inspection and standardization

Pass export error back

Include non standard input reasons

Workaround duration load pause

Avoid mutating while iterating when UploadManager makes delegate calls

Set custom HTTP request header for reporting (#67)

Remove Equatable and Hashable conformance from MuxUpload (#68)

Safe storage for MuxUpload (#71)

Intended to prevent a crash if MuxUpload is extended by the SDK client to conform to Equatable or Hashable protocols

Author: andrewjl-mux

Example/SwiftUploadSDKExample/SwiftUploadSDKExample/Model/ThumbnailModel.swift

@@ -52,7 +52,7 @@ class ThumbnailModel: ObservableObject {
     private var thumbnailGenerator: AVAssetImageGenerator?
 
     @Published var thumbnail: CGImage?
-    @Published var uploadProgress: MuxUpload.Status?
+    @Published var uploadProgress: MuxUpload.TransportStatus?
 
     init(asset: AVAsset, upload: MuxUpload) {
         self.asset = asset

Example/SwiftUploadSDKExample/SwiftUploadSDKExample/Model/UploadCreationModel.swift

@@ -56,6 +56,10 @@ class UploadCreationModel : ObservableObject {
 
     /// Prepares a Photos Asset for upload by exporting it to a local temp file
     func tryToPrepare(from pickerResult: PHPickerResult) {
+        if case ExportState.preparing = exportState {
+            return
+        }
+
         // Cancel anything that was already happening
         if let assetRequestId = assetRequestId {
             PHImageManager.default().cancelImageRequest(assetRequestId)
@@ -92,7 +96,8 @@ class UploadCreationModel : ObservableObject {
         }
 
         let exportOptions = PHVideoRequestOptions()
-        //exportOptions.deliveryMode = .highQualityFormat
+        exportOptions.isNetworkAccessAllowed = true
+        exportOptions.deliveryMode = .highQualityFormat
         assetRequestId = PHImageManager.default().requestExportSession(forVideo: phAsset, options: exportOptions, exportPreset: AVAssetExportPresetHighestQuality, resultHandler: {(exportSession, info) -> Void in
             DispatchQueue.main.async {
                 guard let exportSession = exportSession else {

Example/SwiftUploadSDKExample/SwiftUploadSDKExample/Model/UploadListModel.swift

@@ -15,17 +15,27 @@ class UploadListModel : ObservableObject {
         UploadManager.shared.addUploadsUpdatedDelegate(
             Delegate(
                 handler: { uploads in
-                var uploadSet = Set(self.lastKnownUploads)
-                uploads.forEach {
-                    uploadSet.insert($0)
-                }
-                self.lastKnownUploads = Array(uploadSet)
-                    .sorted(
-                        by: { lhs, rhs in
-                            lhs.uploadStatus.startTime >= rhs.uploadStatus.startTime
+
+                    var lastKnownUploadsToUpdate = self.lastKnownUploads
+
+                    for updatedUpload in uploads {
+                        if !lastKnownUploadsToUpdate.contains(
+                            where: {
+                                $0.uploadURL == updatedUpload.uploadURL &&
+                                $0.videoFile == updatedUpload.videoFile
+                            }
+                        ) {
+                            lastKnownUploadsToUpdate.append(updatedUpload)
                         }
-                    )
-                }
+                    }
+
+                    self.lastKnownUploads = lastKnownUploadsToUpdate
+                        .sorted(
+                            by: { lhs, rhs in
+                                (lhs.uploadStatus?.startTime ?? 0) >= (rhs.uploadStatus?.startTime ?? 0)
+                            }
+                        )
+                    }
             )
         )
     }

Example/SwiftUploadSDKExample/SwiftUploadSDKExample/Screens/UploadListView.swift

@@ -23,6 +23,12 @@ struct UploadListScreen: View {
     }
 }
 
+extension MuxUpload {
+    var objectIdentifier: ObjectIdentifier {
+        ObjectIdentifier(self)
+    }
+}
+
 fileprivate struct ListContainerView: View {
 
     @EnvironmentObject var viewModel: UploadListModel
@@ -33,7 +39,7 @@ fileprivate struct ListContainerView: View {
         } else {
             ScrollView {
                 LazyVStack {
-                    ForEach(viewModel.lastKnownUploads, id: \.self) { upload in
+                    ForEach(viewModel.lastKnownUploads, id: \.objectIdentifier) { upload in
                         ListItem(upload: upload)
                     }
                 }
@@ -124,12 +130,12 @@ fileprivate struct ListItem: View {
         }
     }
 
-    private func statusLine(status: MuxUpload.Status?) -> String {
-        guard let status = status, let progress = status.progress, status.startTime > 0 else {
+    private func statusLine(status: MuxUpload.TransportStatus?) -> String {
+        guard let status = status, let progress = status.progress, let startTime = status.startTime, startTime > 0 else {
             return "missing status"
         }
 
-        let totalTimeSecs = status.updatedTime - status.startTime
+        let totalTimeSecs = status.updatedTime - (status.startTime ?? 0)
         let totalTimeMs = Int64((totalTimeSecs) * 1000)
         let kbytesPerSec = (progress.completedUnitCount) / totalTimeMs // bytes/milli = kb/sec
         let fourSigs = NumberFormatter()
@@ -146,7 +152,7 @@ fileprivate struct ListItem: View {
         return "\(formattedMBytes) MB in \(formattedTime)s (\(formattedDataRate) KB/s)"
     }
 
-    private func elapsedBytesOfTotal(status: MuxUpload.Status) -> String {
+    private func elapsedBytesOfTotal(status: MuxUpload.TransportStatus) -> String {
         guard let progress = status.progress else {
             return "unknown"
         }
@@ -157,7 +163,7 @@ fileprivate struct ListItem: View {
         self.upload = upload
         _thumbnailModel = StateObject(
             wrappedValue: {
-                ThumbnailModel(asset: AVAsset(url: upload.videoFile), upload: upload)
+                ThumbnailModel(asset: AVAsset(url: upload.videoFile!), upload: upload)
             }()
         )
     }

Sources/MuxUploadSDK/Extensions/NSMutableURLRequest.swift renamed to Sources/MuxUploadSDK/Extensions/NSMutableURLRequest+Reporting.swift

@@ -7,7 +7,8 @@ import Foundation
 extension NSMutableURLRequest {
     static func makeJSONPost(
         url: URL,
-        httpBody: Data
+        httpBody: Data,
+        additionalHTTPHeaders: [String: String]
     ) -> NSMutableURLRequest {
         let request = NSMutableURLRequest(
             url: url,
@@ -18,6 +19,10 @@ extension NSMutableURLRequest {
         request.httpMethod = "POST"
         request.setValue("application/json", forHTTPHeaderField: "Accept")
         request.setValue("application/json", forHTTPHeaderField: "Content-Type")
+        for keypair in additionalHTTPHeaders {
+            request.setValue(keypair.value, forHTTPHeaderField: keypair.key)
+        }
+
         request.httpBody = httpBody
 
         return request

Sources/MuxUploadSDK/InputInspection/UploadInputFormatInspectionResult.swift

@@ -0,0 +1,87 @@
+//
+//  UploadInputFormatInspectionResult.swift
+//
+
+import AVFoundation
+import Foundation
+
+enum UploadInputFormatInspectionResult {
+
+    enum NonstandardInputReason {
+        case videoCodec
+        case audioCodec
+        case videoGOPSize
+        case videoFrameRate
+        case videoResolution
+        case videoBitrate
+        case pixelAspectRatio
+        case videoEditList
+        case audioEditList
+        case unexpectedMediaFileParameters
+        case unsupportedPixelFormat
+    }
+
+    case inspectionFailure(duration: CMTime)
+    case standard(duration: CMTime)
+    case nonstandard(
+        reasons: [NonstandardInputReason],
+        duration: CMTime
+    )
+
+    var isStandard: Bool {
+        if case Self.standard = self {
+            return true
+        } else {
+            return false
+        }
+    }
+
+    var sourceInputDuration: CMTime {
+        switch self {
+        case .inspectionFailure(duration: let duration):
+            return duration
+        case .standard(duration: let duration):
+            return duration
+        case .nonstandard(_, duration: let duration):
+            return duration
+        }
+    }
+
+    var nonstandardInputReasons: [NonstandardInputReason]? {
+        if case Self.nonstandard(let nonstandardInputReasons, _) = self {
+            return nonstandardInputReasons
+        } else {
+            return nil
+        }
+    }
+
+}
+
+extension UploadInputFormatInspectionResult.NonstandardInputReason: CustomStringConvertible {
+    var description: String {
+        switch self {
+        case .audioCodec:
+            return "audio_codec"
+        case .audioEditList:
+            return "audio_edit_list"
+        case .pixelAspectRatio:
+            return "pixel_aspect_ratio"
+        case .videoBitrate:
+            return "video_bitrate"
+        case .videoCodec:
+            return "video_codec"
+        case .videoEditList:
+            return "video_edit_list"
+        case .videoFrameRate:
+            return "video_frame_rate"
+        case .videoGOPSize:
+            return "video_gop_size"
+        case .videoResolution:
+            return "video_resolution"
+        case .unexpectedMediaFileParameters:
+            return "unexpected_media_file_parameters"
+        case .unsupportedPixelFormat:
+            return "unsupported_pixel_format"
+        }
+    }
+}

Sources/MuxUploadSDK/InputInspection/UploadInputInspector.swift

@@ -0,0 +1,165 @@
+//
+//  UploadInputInspector.swift
+//  
+
+import AVFoundation
+import CoreMedia
+import Foundation
+
+protocol UploadInputInspector {
+    func performInspection(
+        sourceInput: AVAsset,
+        completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
+    )
+}
+
+class AVFoundationUploadInputInspector: UploadInputInspector {
+
+    static let shared = AVFoundationUploadInputInspector()
+
+    func performInspection(
+        sourceInput: AVAsset,
+        completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
+    ) {
+        sourceInput.loadValuesAsynchronously(
+            forKeys: [
+                "duration"
+            ]
+        ) {
+            // FIXME: Trying to avoid the callback pyramid of doom
+            // here, newer AVAsset APIs use Concurrency
+            // but Concurrency itself has very primitive
+            // task sequencing. Replace with async AVAsset
+            // methods.
+            let sourceInputDuration = sourceInput.duration
+            self.performInspection(
+                sourceInput: sourceInput,
+                sourceInputDuration: sourceInputDuration,
+                completionHandler: completionHandler
+            )
+        }
+    }
+
+    func performInspection(
+        sourceInput: AVAsset,
+        sourceInputDuration: CMTime,
+        completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
+    ) {
+        // TODO: Eventually load audio tracks too
+        if #available(iOS 15, *) {
+            sourceInput.loadTracks(
+                withMediaType: .video
+            ) { tracks, error in
+                if error != nil {
+                    completionHandler(
+                        .inspectionFailure(duration: sourceInputDuration)
+                    )
+                    return
+                }
+
+                if let tracks {
+                    self.inspect(
+                        sourceInputDuration: sourceInputDuration,
+                        tracks: tracks,
+                        completionHandler: completionHandler
+                    )
+                }
+            }
+        } else {
+            sourceInput.loadValuesAsynchronously(
+                forKeys: [
+                    "tracks"
+                ]
+            ) {
+                // Non-blocking if "tracks" is already loaded
+                let tracks = sourceInput.tracks(
+                    withMediaType: .video
+                )
+
+                self.inspect(
+                    sourceInputDuration: sourceInputDuration,
+                    tracks: tracks,
+                    completionHandler: completionHandler
+                )
+            }
+        }
+    }
+
+    func inspect(
+        sourceInputDuration: CMTime,
+        tracks: [AVAssetTrack],
+        completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
+    ) {
+        switch tracks.count {
+        case 0:
+            // Nothing to inspect, therefore nothing to standardize
+            // declare as already standard
+            completionHandler(
+                .standard(duration: sourceInputDuration)
+            )
+        case 1:
+            if let track = tracks.first {
+                track.loadValuesAsynchronously(
+                    forKeys: [
+                        "formatDescriptions",
+                        "nominalFrameRate"
+                    ]
+                ) {
+                    guard let formatDescriptions = track.formatDescriptions as? [CMFormatDescription] else {
+                        completionHandler(
+                            .inspectionFailure(
+                                duration: sourceInputDuration
+                            )
+                        )
+                        return
+                    }
+
+                    guard let formatDescription = formatDescriptions.first else {
+                        completionHandler(
+                            .inspectionFailure(duration: sourceInputDuration)
+                        )
+                        return
+                    }
+
+                    var nonStandardReasons: [UploadInputFormatInspectionResult.NonstandardInputReason] = []
+
+                    let videoDimensions = CMVideoFormatDescriptionGetDimensions(
+                        formatDescription
+                    )
+
+                    if max(videoDimensions.width, videoDimensions.height) > 1920 {
+                        nonStandardReasons.append(.videoResolution)
+                    }
+
+                    let videoCodecType = formatDescription.mediaSubType
+
+                    let standard = CMFormatDescription.MediaSubType.h264
+
+                    if videoCodecType != standard {
+                        nonStandardReasons.append(.videoCodec)
+                    }
+
+                    let frameRate = track.nominalFrameRate
+                    if frameRate > 120.0 {
+                        nonStandardReasons.append(.videoFrameRate)
+                    }
+
+                    if nonStandardReasons.isEmpty {
+                        completionHandler(
+                            .standard(duration: sourceInputDuration)
+                        )
+                    } else {
+                        completionHandler(.nonstandard(reasons: nonStandardReasons, duration: sourceInputDuration))
+                    }
+
+                }
+            }
+        default:
+            // Inspection fails for multi-video track inputs
+            // for the time being
+            completionHandler(
+                .inspectionFailure(duration: sourceInputDuration)
+            )
+        }
+    }
+}