Skip to content

Commit

Permalink
Refactor ios-build-test workflow to support binary release (pytorch#1…
Browse files Browse the repository at this point in the history
…08322)

This refactors the logic from CircleCI iOS [build](https://github.com/pytorch/pytorch/blob/main/.circleci/config.yml#L1323-L1344) and [upload](https://github.com/pytorch/pytorch/blob/main/.circleci/config.yml#L1369-L1377) jobs to GHA.

* Nightly artifacts will be available again on `ossci-ios-build` S3 bucket, for example `libtorch_lite_ios_nightly_2.1.0.20230517.zip`.  The last one there was s3://ossci-ios-build/libtorch_lite_ios_nightly_2.1.0.20230517.zip from May 17th
  * [LibTorch-Lite-Nightly](https://github.com/CocoaPods/Specs/blob/master/Specs/c/3/1/LibTorch-Lite-Nightly/1.14.0.20221109/LibTorch-Lite-Nightly.podspec.json) on cocoapods
* Release artifacts will be on `ossci-ios` S3 bucket, for example `s3://ossci-ios/libtorch_lite_ios_1.13.0.zip` from Nov 3rd 2022
  * [LibTorch-Lite](https://github.com/CocoaPods/Specs/blob/master/Specs/c/c/3/LibTorch-Lite/1.13.0.1/LibTorch-Lite.podspec.json) on cocoapods
  * [LibTorch](https://github.com/CocoaPods/Specs/blob/master/Specs/1/3/c/LibTorch/1.13.0.1/LibTorch.podspec.json) on cocoapods

I will clean up Circle CI code in another PR.

### Testing

Generate new release artifacts for testing from main branch.  Simulator testing have all passed.

* With lite interpreter https://github.com/pytorch/pytorch/actions/runs/6093860118
  * https://ossci-ios.s3.amazonaws.com/libtorch_lite_ios_2.1.0.zip
  * https://ossci-ios.s3.amazonaws.com/LibTorch-Lite-2.1.0.podspec

* LibTorch binary can be built without lite interpreter https://github.com/pytorch/pytorch/actions/runs/6103616035 and uses TorchScript, but it has been long dead from my understanding.  The binary can still be built and tested though.
  * https://ossci-ios.s3.amazonaws.com/libtorch_ios_2.1.0.zip
  * https://ossci-ios.s3.amazonaws.com/LibTorch-2.1.0.podspec

### Next step for release

* Once the PR is committed.  I plan to use the workflow dispatch to build the binaries manually on `release/2.1` branch.  Once they looks good, we can publish them on cocoapods.
Pull Request resolved: pytorch#108322
Approved by: https://github.com/atalman
Rate limit · GitHub

Access has been restricted

You have triggered a rate limit.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

huydhn authored and pytorchmergebot committed Sep 10, 2023
1 parent 63ae105 commit 4a98c89
Showing 8 changed files with 431 additions and 79 deletions.
367 changes: 320 additions & 47 deletions .github/workflows/_ios-build-test.yml

Large diffs are not rendered by default.

70 changes: 70 additions & 0 deletions .github/workflows/build-ios-binaries.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: Build iOS binaries

on:
push:
branches:
- nightly
tags:
# NOTE: Binary build pipelines should only get triggered on release candidate builds
# Release candidate tags look like: v1.11.0-rc1
- v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+
paths:
- .github/workflows/build-ios-binaries.yml
- .github/workflows/_ios-build-test.yml
pull_request:
paths:
- .github/workflows/build-ios-binaries.yml
- .github/workflows/_ios-build-test.yml
# NB: We can use this workflow dispatch to test and build iOS binaries manually
workflow_dispatch:
inputs:
use_lite_interpreter:
description: "Use PyTorch lite interpreter?"
type: string
default: 1
use_coreml:
description: "Use Apple Core ML?"
type: string
default: 1
use_custom_op_list:
description: "Specify the custom ops list to include in the binaries"
type: string
default: ""

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true

jobs:
# TODO: Figure out how to migrate this job to M1 runner
ios-build-test:
name: ios-build-test
uses: ./.github/workflows/_ios-build-test.yml
with:
build-environment: ios-build-test
sync-tag: ios-build-test
test-matrix: |
{ include: [
{ config: "default",
shard: 1,
num_shards: 1,
runner: "macos-12",
ios_platform: "SIMULATOR",
ios_arch: "x86_64",
use_lite_interpreter: ${{ inputs.use_lite_interpreter || 1 }},
use_metal: 0,
use_coreml: ${{ inputs.use_coreml || 1 }},
use_custom_op_list: ${{ inputs.use_custom_op_list || '' }}
},
{ config: "default",
shard: 1,
num_shards: 1,
runner: "macos-12",
ios_platform: "OS",
ios_arch: "arm64",
use_lite_interpreter: ${{ inputs.use_lite_interpreter || 1 }},
use_metal: 1,
use_coreml: ${{ inputs.use_coreml || 1 }},
use_custom_op_list: ${{ inputs.use_custom_op_list || '' }}
}
]}
46 changes: 27 additions & 19 deletions .github/workflows/periodic.yml
Original file line number Diff line number Diff line change
@@ -112,30 +112,38 @@ jobs:
cuda-version: "11.8"
test-matrix: ${{ needs.win-vs2019-cuda11_8-py3-build.outputs.test-matrix }}

ios-12-5-1-x86-64-coreml:
name: ios-12-5-1-x86-64-coreml
# TODO: Figure out how to migrate this job to M1 runner
ios-build-test:
name: ios-build-test
if: github.event_name != 'schedule' || github.event.schedule == '45 0,8,16 * * 1-5' || github.event.schedule == '45 4 * * 0,6'
uses: ./.github/workflows/_ios-build-test.yml
with:
build-environment: ios-12-5-1-x86-64-coreml
ios-platform: SIMULATOR
ios-arch: x86_64
build-environment: ios-build-test
sync-tag: ios-build-test
test-matrix: |
{ include: [
{ config: "default", shard: 1, num_shards: 1, runner: "macos-12" },
]}
ios-12-5-1-arm64-custom-ops:
name: ios-12-5-1-arm64-custom-ops
if: github.event_name != 'schedule' || github.event.schedule == '45 0,8,16 * * 1-5' || github.event.schedule == '45 4 * * 0,6'
uses: ./.github/workflows/_ios-build-test.yml
with:
build-environment: ios-12-5-1-arm64-custom-ops
ios-platform: OS
ios-arch: arm64
test-matrix: |
{ include: [
{ config: "default", shard: 1, num_shards: 1, runner: "macos-12" },
{ config: "default",
shard: 1,
num_shards: 1,
runner: "macos-12",
ios_platform: "SIMULATOR",
ios_arch: "x86_64",
use_lite_interpreter: 1,
use_metal: 0,
use_coreml: 1,
use_custom_op_list: ""
},
{ config: "default",
shard: 1,
num_shards: 1,
runner: "macos-12",
ios_platform: "OS",
ios_arch: "arm64",
use_lite_interpreter: 1,
use_metal: 1,
use_coreml: 1,
use_custom_op_list: "mobilenetv2.yaml"
}
]}
buck-build-test:
16 changes: 8 additions & 8 deletions cmake/iOS.cmake
Original file line number Diff line number Diff line change
@@ -53,9 +53,9 @@ set(CMAKE_RANLIB ranlib CACHE FILEPATH "" FORCE)
set(PKG_CONFIG_EXECUTABLE pkg-config CACHE FILEPATH "" FORCE)

# Setup iOS platform unless specified manually with IOS_PLATFORM
if(NOT DEFINED IOS_PLATFORM)
if(NOT IOS_PLATFORM)
set(IOS_PLATFORM "OS")
endif(NOT DEFINED IOS_PLATFORM)
endif(NOT IOS_PLATFORM)
set(IOS_PLATFORM ${IOS_PLATFORM} CACHE STRING "Type of iOS Platform")

# Check the platform selection and setup for developer root
@@ -118,9 +118,9 @@ set(CMAKE_FIND_LIBRARY_SUFFIXES ".dylib" ".so" ".a")
# (where install_name_tool was hardcoded) and where CMAKE_INSTALL_NAME_TOOL isn't in the cache
# and still cmake didn't fail in CMakeFindBinUtils.cmake (because it isn't rerun)
# hardcode CMAKE_INSTALL_NAME_TOOL here to install_name_tool, so it behaves as it did before, Alex
if(NOT DEFINED CMAKE_INSTALL_NAME_TOOL)
if(NOT CMAKE_INSTALL_NAME_TOOL)
find_program(CMAKE_INSTALL_NAME_TOOL install_name_tool)
endif(NOT DEFINED CMAKE_INSTALL_NAME_TOOL)
endif(NOT CMAKE_INSTALL_NAME_TOOL)

# Setup iOS deployment target
set(IOS_DEPLOYMENT_TARGET ${IOS_DEPLOYMENT_TARGET} CACHE STRING "Minimum iOS version")
@@ -130,17 +130,17 @@ set(IOS_DEPLOYMENT_TARGET ${IOS_DEPLOYMENT_TARGET} CACHE STRING "Minimum iOS ver
exec_program(/usr/bin/xcode-select ARGS -print-path OUTPUT_VARIABLE CMAKE_XCODE_DEVELOPER_DIR)
set(XCODE_POST_43_ROOT "${CMAKE_XCODE_DEVELOPER_DIR}/Platforms/${IOS_PLATFORM_LOCATION}/Developer")
set(XCODE_PRE_43_ROOT "/Developer/Platforms/${IOS_PLATFORM_LOCATION}/Developer")
if(NOT DEFINED CMAKE_IOS_DEVELOPER_ROOT)
if(NOT CMAKE_IOS_DEVELOPER_ROOT)
if(EXISTS ${XCODE_POST_43_ROOT})
set(CMAKE_IOS_DEVELOPER_ROOT ${XCODE_POST_43_ROOT})
elseif(EXISTS ${XCODE_PRE_43_ROOT})
set(CMAKE_IOS_DEVELOPER_ROOT ${XCODE_PRE_43_ROOT})
endif(EXISTS ${XCODE_POST_43_ROOT})
endif(NOT DEFINED CMAKE_IOS_DEVELOPER_ROOT)
endif(NOT CMAKE_IOS_DEVELOPER_ROOT)
set(CMAKE_IOS_DEVELOPER_ROOT ${CMAKE_IOS_DEVELOPER_ROOT} CACHE PATH "Location of iOS Platform")

# Find and use the most recent iOS sdk unless specified manually with CMAKE_IOS_SDK_ROOT
if(NOT DEFINED CMAKE_IOS_SDK_ROOT)
if(NOT CMAKE_IOS_SDK_ROOT)
file(GLOB _CMAKE_IOS_SDKS "${CMAKE_IOS_DEVELOPER_ROOT}/SDKs/*")
if(_CMAKE_IOS_SDKS)
list(SORT _CMAKE_IOS_SDKS)
@@ -150,7 +150,7 @@ if(NOT DEFINED CMAKE_IOS_SDK_ROOT)
message(FATAL_ERROR "No iOS SDK's found in default search path ${CMAKE_IOS_DEVELOPER_ROOT}. Manually set CMAKE_IOS_SDK_ROOT or install the iOS SDK.")
endif(_CMAKE_IOS_SDKS)
message(STATUS "Toolchain using default iOS SDK: ${CMAKE_IOS_SDK_ROOT}")
endif(NOT DEFINED CMAKE_IOS_SDK_ROOT)
endif(NOT CMAKE_IOS_SDK_ROOT)
set(CMAKE_IOS_SDK_ROOT ${CMAKE_IOS_SDK_ROOT} CACHE PATH "Location of the selected iOS SDK")

# Set the sysroot default to the most recent SDK
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Pod::Spec.new do |s|
s.name = 'LibTorch-Lite'
s.version = '1.13.0'
s.version = 'IOS_BUILD_VERSION'
s.authors = 'PyTorch Team'
s.license = { :type => 'BSD' }
s.homepage = 'https://github.com/pytorch/pytorch'
@@ -33,5 +33,5 @@ Pod::Spec.new do |s|
'VALID_ARCHS' => 'x86_64 arm64'
}
s.library = ['c++', 'stdc++']
s.frameworks = 'Accelerate'
s.frameworks = 'Accelerate', 'MetalPerformanceShaders', 'CoreML'
end
4 changes: 2 additions & 2 deletions ios/LibTorch.podspec → ios/LibTorch.podspec.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Pod::Spec.new do |s|
s.name = 'LibTorch'
s.version = '1.13.0'
s.version = 'IOS_BUILD_VERSION'
s.authors = 'PyTorch Team'
s.license = { :type => 'BSD' }
s.homepage = 'https://github.com/pytorch/pytorch'
@@ -33,5 +33,5 @@ Pod::Spec.new do |s|
'VALID_ARCHS' => 'x86_64 arm64'
}
s.library = ['c++', 'stdc++']
s.frameworks = 'Accelerate'
s.frameworks = 'Accelerate', 'MetalPerformanceShaders', 'CoreML'
end
1 change: 1 addition & 0 deletions ios/TestApp/benchmark/coreml_backend.py
Original file line number Diff line number Diff line change
@@ -32,6 +32,7 @@ def main():
mlmodel = torch._C._jit_to_backend("coreml", model, compile_spec)
print(mlmodel._c._get_method("forward").graph)
mlmodel._save_for_lite_interpreter("../models/model_coreml.ptl")
torch.jit.save(mlmodel, "../models/model_coreml.pt")


if __name__ == "__main__":
2 changes: 1 addition & 1 deletion ios/TestApp/benchmark/trace_model.py
Original file line number Diff line number Diff line change
@@ -9,5 +9,5 @@
optimized_scripted_module = optimize_for_mobile(traced_script_module)
torch.jit.save(optimized_scripted_module, "../models/model.pt")
exported_optimized_scripted_module = (
optimized_scripted_module._save_for_lite_interpreter("../models/model_lite.ptl")
optimized_scripted_module._save_for_lite_interpreter("../models/model.ptl")
)

0 comments on commit 4a98c89

Please sign in to comment.