Skip to content

Commit 0428d60

Browse files
andrewjl-muxandrewjl-mux
authored
Project Input standardization (#48)
Upload Input Inspection and Standardization 1 (#17) Add scaffolding Add initializers Add internal and external state mapping Remove duplicate status enum and add inline docs to external status Add inline API docs to PHAsset-based MuxUpload constructor Consolidate all `MuxUpload` options into a single struct `UploadOptions` Declare asynchronous MuxUpload constructor in PHAsset extension Place extension methods into dedicated directories Polish inline API documentation Add new API documentation and note the placeholder implementation Add option variants as static members: defaults, disabled inputStandardization Deprecate existing initializer, normally this API should be removed prior to GA, but since it was the only initializer exposed up to this point removing it would break everybody. Instead deprecate and remove at a later date. Store all MuxUpload-related options in UploadInfo Use correct starting byte parameter when restarting upload Hardcode request options when exporting AVAsset from PhotosKit chunkSize -> chunkSizeInBytes Input standardization 2 (#41) Maintain handle to canonical input asset inside UploadInput If input standardized, standardized input URL is passed to UploadInfo instead of the original input URL used for initializer Note: SDK probably needs to re-export a high quality asset anyway so possibly need a bridging status Add dedicated internal initializer for MuxUpload error with unknown error code Disable input standardization when running tests Add inspection logic Request local and remote assets Status -> TransportStatus Rely on input status in MuxUpload computed property getters Expose progress from internal upload state if available Track transport status inside of UploadInput Transport status start time optional Add TransportStatus docs Inspect and check video track frame rate Migrate tests for MuxUpload input result handler Input standardization 3 (#46) Standardize via export session Back out outputURL construction into MuxUpload Expose hook for client to cancel upload if standardization failed Call cancellation hook if inspection fails. We're not sure if the input is standard or not so better to be safe and confirm Export based on maximum resolution set by client Cleaner non standard input handler invocation (#57) Add CustomStringConvertible conformance to maximum resolution (#56) Update input state when upload is finished (#63) Update input state when the upload succeeds or fails Tweak input state definition, keep an error in the failed case and the success struct in the success case NFC: remove some if let checks for the handler closure properties in MuxUpload and replace with ? operator. Keeps the existing logic. Replace video file get with actual implementation (#64) Reporting updates (#49) Only mark upload as started if its ready Add upload failed and input standardization events Add standardization failure event handler to reporter Add standardization success event handler to reporter Rename method for upload success Rename upload event to upload succeeded Support dispatching multiple events at the same time Naive non-thread safe implementation reporter networking Change event type casing, individual files for events Fix reporter test Actually report upload failure Read off file size Use Date in events and serialize to ISO8601 string Finangle duration and start time Set correct version in correct place Keep a session UUID Route every request through one chokepoint Upload failed test Shared test encoder Input standardization failed test Input standardization succeeded test Better organize expected json strings Use a shared instance Remove dupe file Thread asset duration through inspection and standardization Pass export error back Include non standard input reasons Workaround duration load pause Avoid mutating while iterating when UploadManager makes delegate calls Set custom HTTP request header for reporting (#67) Remove Equatable and Hashable conformance from MuxUpload (#68) Safe storage for MuxUpload (#71) Intended to prevent a crash if MuxUpload is extended by the SDK client to conform to Equatable or Hashable protocols
1 parent 2b23dc1 commit 0428d60

33 files changed

+2561
-367
lines changed

‎Sources/MuxUploadSDK/Extensions/NSMutableURLRequest.swift renamed to ‎Sources/MuxUploadSDK/Extensions/NSMutableURLRequest+Reporting.swift

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ import Foundation
77
extension NSMutableURLRequest {
88
static func makeJSONPost(
99
url: URL,
10-
httpBody: Data
10+
httpBody: Data,
11+
additionalHTTPHeaders: [String: String]
1112
) -> NSMutableURLRequest {
1213
let request = NSMutableURLRequest(
1314
url: url,
@@ -18,6 +19,10 @@ extension NSMutableURLRequest {
1819
request.httpMethod = "POST"
1920
request.setValue("application/json", forHTTPHeaderField: "Accept")
2021
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
22+
for keypair in additionalHTTPHeaders {
23+
request.setValue(keypair.value, forHTTPHeaderField: keypair.key)
24+
}
25+
2126
request.httpBody = httpBody
2227

2328
return request

‎Sources/MuxUploadSDK/InputInspection/UploadInputFormatInspectionResult.swift

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
//
2+
// UploadInputFormatInspectionResult.swift
3+
//
4+
5+
import AVFoundation
6+
import Foundation
7+
8+
enum UploadInputFormatInspectionResult {
9+
10+
enum NonstandardInputReason {
11+
case videoCodec
12+
case audioCodec
13+
case videoGOPSize
14+
case videoFrameRate
15+
case videoResolution
16+
case videoBitrate
17+
case pixelAspectRatio
18+
case videoEditList
19+
case audioEditList
20+
case unexpectedMediaFileParameters
21+
case unsupportedPixelFormat
22+
}
23+
24+
case inspectionFailure(duration: CMTime)
25+
case standard(duration: CMTime)
26+
case nonstandard(
27+
reasons: [NonstandardInputReason],
28+
duration: CMTime
29+
)
30+
31+
var isStandard: Bool {
32+
if case Self.standard = self {
33+
return true
34+
} else {
35+
return false
36+
}
37+
}
38+
39+
var sourceInputDuration: CMTime {
40+
switch self {
41+
case .inspectionFailure(duration: let duration):
42+
return duration
43+
case .standard(duration: let duration):
44+
return duration
45+
case .nonstandard(_, duration: let duration):
46+
return duration
47+
}
48+
}
49+
50+
var nonstandardInputReasons: [NonstandardInputReason]? {
51+
if case Self.nonstandard(let nonstandardInputReasons, _) = self {
52+
return nonstandardInputReasons
53+
} else {
54+
return nil
55+
}
56+
}
57+
58+
}
59+
60+
extension UploadInputFormatInspectionResult.NonstandardInputReason: CustomStringConvertible {
61+
var description: String {
62+
switch self {
63+
case .audioCodec:
64+
return "audio_codec"
65+
case .audioEditList:
66+
return "audio_edit_list"
67+
case .pixelAspectRatio:
68+
return "pixel_aspect_ratio"
69+
case .videoBitrate:
70+
return "video_bitrate"
71+
case .videoCodec:
72+
return "video_codec"
73+
case .videoEditList:
74+
return "video_edit_list"
75+
case .videoFrameRate:
76+
return "video_frame_rate"
77+
case .videoGOPSize:
78+
return "video_gop_size"
79+
case .videoResolution:
80+
return "video_resolution"
81+
case .unexpectedMediaFileParameters:
82+
return "unexpected_media_file_parameters"
83+
case .unsupportedPixelFormat:
84+
return "unsupported_pixel_format"
85+
}
86+
}
87+
}

‎Sources/MuxUploadSDK/InputInspection/UploadInputInspector.swift

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
//
2+
// UploadInputInspector.swift
3+
//
4+
5+
import AVFoundation
6+
import CoreMedia
7+
import Foundation
8+
9+
protocol UploadInputInspector {
10+
func performInspection(
11+
sourceInput: AVAsset,
12+
completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
13+
)
14+
}
15+
16+
class AVFoundationUploadInputInspector: UploadInputInspector {
17+
18+
static let shared = AVFoundationUploadInputInspector()
19+
20+
func performInspection(
21+
sourceInput: AVAsset,
22+
completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
23+
) {
24+
sourceInput.loadValuesAsynchronously(
25+
forKeys: [
26+
"duration"
27+
]
28+
) {
29+
// FIXME: Trying to avoid the callback pyramid of doom
30+
// here, newer AVAsset APIs use Concurrency
31+
// but Concurrency itself has very primitive
32+
// task sequencing. Replace with async AVAsset
33+
// methods.
34+
let sourceInputDuration = sourceInput.duration
35+
self.performInspection(
36+
sourceInput: sourceInput,
37+
sourceInputDuration: sourceInputDuration,
38+
completionHandler: completionHandler
39+
)
40+
}
41+
}
42+
43+
func performInspection(
44+
sourceInput: AVAsset,
45+
sourceInputDuration: CMTime,
46+
completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
47+
) {
48+
// TODO: Eventually load audio tracks too
49+
if #available(iOS 15, *) {
50+
sourceInput.loadTracks(
51+
withMediaType: .video
52+
) { tracks, error in
53+
if error != nil {
54+
completionHandler(
55+
.inspectionFailure(duration: sourceInputDuration)
56+
)
57+
return
58+
}
59+
60+
if let tracks {
61+
self.inspect(
62+
sourceInputDuration: sourceInputDuration,
63+
tracks: tracks,
64+
completionHandler: completionHandler
65+
)
66+
}
67+
}
68+
} else {
69+
sourceInput.loadValuesAsynchronously(
70+
forKeys: [
71+
"tracks"
72+
]
73+
) {
74+
// Non-blocking if "tracks" is already loaded
75+
let tracks = sourceInput.tracks(
76+
withMediaType: .video
77+
)
78+
79+
self.inspect(
80+
sourceInputDuration: sourceInputDuration,
81+
tracks: tracks,
82+
completionHandler: completionHandler
83+
)
84+
}
85+
}
86+
}
87+
88+
func inspect(
89+
sourceInputDuration: CMTime,
90+
tracks: [AVAssetTrack],
91+
completionHandler: @escaping (UploadInputFormatInspectionResult) -> ()
92+
) {
93+
switch tracks.count {
94+
case 0:
95+
// Nothing to inspect, therefore nothing to standardize
96+
// declare as already standard
97+
completionHandler(
98+
.standard(duration: sourceInputDuration)
99+
)
100+
case 1:
101+
if let track = tracks.first {
102+
track.loadValuesAsynchronously(
103+
forKeys: [
104+
"formatDescriptions",
105+
"nominalFrameRate"
106+
]
107+
) {
108+
guard let formatDescriptions = track.formatDescriptions as? [CMFormatDescription] else {
109+
completionHandler(
110+
.inspectionFailure(
111+
duration: sourceInputDuration
112+
)
113+
)
114+
return
115+
}
116+
117+
guard let formatDescription = formatDescriptions.first else {
118+
completionHandler(
119+
.inspectionFailure(duration: sourceInputDuration)
120+
)
121+
return
122+
}
123+
124+
var nonStandardReasons: [UploadInputFormatInspectionResult.NonstandardInputReason] = []
125+
126+
let videoDimensions = CMVideoFormatDescriptionGetDimensions(
127+
formatDescription
128+
)
129+
130+
if max(videoDimensions.width, videoDimensions.height) > 1920 {
131+
nonStandardReasons.append(.videoResolution)
132+
}
133+
134+
let videoCodecType = formatDescription.mediaSubType
135+
136+
let standard = CMFormatDescription.MediaSubType.h264
137+
138+
if videoCodecType != standard {
139+
nonStandardReasons.append(.videoCodec)
140+
}
141+
142+
let frameRate = track.nominalFrameRate
143+
if frameRate > 120.0 {
144+
nonStandardReasons.append(.videoFrameRate)
145+
}
146+
147+
if nonStandardReasons.isEmpty {
148+
completionHandler(
149+
.standard(duration: sourceInputDuration)
150+
)
151+
} else {
152+
completionHandler(.nonstandard(reasons: nonStandardReasons, duration: sourceInputDuration))
153+
}
154+
155+
}
156+
}
157+
default:
158+
// Inspection fails for multi-video track inputs
159+
// for the time being
160+
completionHandler(
161+
.inspectionFailure(duration: sourceInputDuration)
162+
)
163+
}
164+
}
165+
}

‎Sources/MuxUploadSDK/InputStandardization/UploadInputStandardizationWorker.swift

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
//
2+
// UploadInputStandardizationWorker.swift
3+
//
4+
5+
import AVFoundation
6+
import Foundation
7+
8+
protocol Standardizable { }
9+
10+
extension AVAsset: Standardizable { }
11+
12+
enum StandardizationResult {
13+
case success(standardizedAsset: AVAsset)
14+
case failure(error: Error)
15+
}
16+
17+
enum StandardizationStrategy {
18+
// Prefer using export session whenever possible
19+
case exportSession
20+
}
21+
22+
struct StandardizationError: Error {
23+
var localizedDescription: String
24+
25+
static var missingExportPreset = StandardizationError(
26+
localizedDescription: "Missing export session preset"
27+
)
28+
29+
static var exportSessionInitializationFailure = StandardizationError(
30+
localizedDescription: "Export session failed to initialize"
31+
)
32+
33+
static var standardizedAssetExportFailure = StandardizationError(
34+
localizedDescription: "Failed to export standardized asset"
35+
)
36+
}
37+
38+
class UploadInputStandardizationWorker {
39+
40+
var sourceInput: AVAsset?
41+
42+
var standardizedInput: AVAsset?
43+
44+
func standardize(
45+
sourceAsset: AVAsset,
46+
maximumResolution: UploadOptions.InputStandardization.MaximumResolution,
47+
outputURL: URL,
48+
completion: @escaping (AVAsset, AVAsset?, Error?) -> ()
49+
) {
50+
51+
let availableExportPresets = AVAssetExportSession.allExportPresets()
52+
53+
let exportPreset: String
54+
if maximumResolution == .preset1280x720 {
55+
exportPreset = AVAssetExportPreset1280x720
56+
} else {
57+
exportPreset = AVAssetExportPreset1920x1080
58+
}
59+
60+
guard availableExportPresets.contains(where: {
61+
$0 == exportPreset
62+
}) else {
63+
// TODO: Use VideoToolbox if export preset unavailable
64+
completion(sourceAsset, nil, StandardizationError.missingExportPreset)
65+
return
66+
}
67+
68+
guard let exportSession = AVAssetExportSession(
69+
asset: sourceAsset,
70+
presetName: exportPreset
71+
) else {
72+
// TODO: Use VideoToolbox if export session fails to initialize
73+
completion(sourceAsset, nil, StandardizationError.exportSessionInitializationFailure)
74+
return
75+
}
76+
77+
exportSession.outputFileType = .mp4
78+
exportSession.outputURL = outputURL
79+
80+
// TODO: Use Swift Concurrency
81+
exportSession.exportAsynchronously {
82+
if let exportError = exportSession.error {
83+
completion(sourceAsset, nil, exportError)
84+
} else if let standardizedAssetURL = exportSession.outputURL {
85+
let standardizedAsset = AVAsset(url: standardizedAssetURL)
86+
completion(sourceAsset, standardizedAsset, nil)
87+
} else {
88+
completion(sourceAsset, nil, StandardizationError.standardizedAssetExportFailure)
89+
}
90+
}
91+
}
92+
}

0 commit comments

Comments
 (0)