Cherry picking for Rel-1.6 (#6006)

* Update onnx (#5720)

* update onnx

* update docker image for testing
(cherry picked from commit 705d093)

* cherry pick PR 5720

* C#: Add CreateFromMemory to FixedBufferOnnxValue to allow bind user buffers and pass custom binary compatible types (#5886)

Add CreateFromMemory to FixedBufferOnnxValue so users can bind their own custom binary compatible buffers to feed/fetch data.
(cherry picked from commit c2d6100)

* [Java] Initial Apple Silicon support (#5891)

* Rearranging checks in onnxruntime_mlas.cmake to pickup Apple Silicon.

On an M1 Macbook Pro clang reports:

$ clang -dumpmachine
arm64-apple-darwin20.1.0

So the regex check needs to look for "arm64" first, as otherwise it
matches 32-bit ARM and you get NEON compilation failures.

* Adding Java side library loading support for Apple Silicon (and other aarch64 architectures).

* Adding Qgemm fix from @tracysh

* Fixes the java packaging on Windows.

* Missed a check in the java platform detector.
(cherry picked from commit 8b83c51)

* Add OpenVINO EP shared lib to Py Wheel (#5920)

* Add OpenVINO EP shared lib to Py Wheel

Include the libonnxruntime_providers_openvino.so/.dll to the wheel

* Follow libs.extend pattern as other EPs
(cherry picked from commit 4092686)

* Make NNAPI EP reject nodes with no-shape inputs (#5927)

(cherry picked from commit 8736865)

* Sahar/fix documentation shared lib (#5926)

* Update OpenVINO-ExecutionProvider.Md

update openvino-executionprovider.md for shared library

* Update Build.md

updated --build_shared_lib flag for building openvino shared provider lib

* Update Dockerfile.openvino

building for shared library with the new changes for openvino shared lib

* Revert "Update Build.md"

This reverts commit c9cf5fe.

* Revert "Update Dockerfile.openvino "

This reverts commit e1624e4.

* Update OpenVINO-ExecutionProvider.md

fix documentation to the shared library

Co-authored-by: sfatimar <sahar.fatima@intel/com>
(cherry picked from commit 8168c91)

* Update dockerfiles (#5929)

1. Remove conda from the images. Because conda contains a file named /opt/miniconda/lib/libcrypto.so.1.0.0 which can't pass our security scan. Also, it will be easier for us to manage the third party usage registrations.
2. Remove openssh from the images. Because the official openssh package provided by Ubuntu can't pass our security scan.
3. Reduce the image size to 1/3 by using stages. Also, because it contains less packages, it will be less often needed to update.
4. Put the LICENSE-IMAGE.txt file in right place. It is missed in current images. You can see it was added to a temp folder "/code" but it got deleted afterwards.
5. Update the CPU docker image's base image to Ubuntu 18.04. The GPU one is already 18.04. It's better to keep them the same.
6. Remove the build arg ONNXRUNTIME_REPO/ONNXRUNTIME_BRANCH. Instead, the new one always uses the local source. I feel it can reduce confusion.
(cherry picked from commit 1dbabb2)

* Add Longformer Attention Cuda Op(#5932)

Limitation: Global tokens must be at the beginning of sequence.
(cherry picked from commit 31a6be3)

* Bug fix for MaskRCNN and FasterRCNN (#5935)

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>
(cherry picked from commit e39e82b)

* Fix publishing pipelines. (#5942)

Fix publishing pipelines.
(cherry picked from commit c4b55d2)

* Fix Python Linux GPU package name (#5943)

Fix Python Linux GPU package name. I accidentally added "noopenmp" to it.

(cherry picked from commit 5fdd9f0)

* Update BUILD.md with shared provider information (#5944)

* Update build instructions to include information about shared providers

(cherry picked from commit 27513d1)

* [OpenVINO]Fix memory leak in `IsDebugEnabled()` under Windows (#5948)

* w

* w

Co-authored-by: modav <modav@microsoft.com>
(cherry picked from commit e207589)

* Add support for Python 3.8+ on Windows when CUDA is enabled (#5956)

(cherry picked from commit 015fbb3)

* Support the cross compiling for Apple Silicon (#5974)

* support macos_arm64 cross compiling

* update the build docs

* update as commented.

* Update BUILD.md
(cherry picked from commit 2ec211e)

* Update docker files to put 'unattended-upgrades' in a right place(#5983)

(cherry picked from commit 3323fb6)

* Enable the xcode build for Apple Silicon (arm64 MacOS) (#5924)

* fix the build script for macos/xcode

* add the version check

* correct the osx-arch configuration

* typo
(cherry picked from commit 1852ade)

* Add python 3.9 support (#5874)

1. Add python 3.9 support(except Linux ARM)
2. Add Windows GPU python 3.8 to our packaging pipeline.

* Revert some pipeline changes in #5874

Co-authored-by: Ashwini Khade <askhade@microsoft.com>
Co-authored-by: Du Li <duli@OrtTrainingDev0.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com>
Co-authored-by: Adam Pocock <craigacp@gmail.com>
Co-authored-by: S. Manohar Karlapalem <manohar.karlapalem@intel.com>
Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com>
Co-authored-by: sfatimar <64512376+sfatimar@users.noreply.github.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Maajid khan <n.maajidkhan@gmail.com>
Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com>
Co-authored-by: Moshe David <mosdav165@gmail.com>
Co-authored-by: Ivan Stojiljkovic <17503404+ivanst0@users.noreply.github.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>

Author: 15 people

.dockerignore

@@ -1,5 +1,4 @@
 .git
 .gitignore
 .gitattributes
-docs
 

.gitignore

@@ -14,6 +14,7 @@ cmake-build-debug
 gen
 *~
 .vs
+.DS_Store
 TestResults/
 .idea/
 onnxruntime.egg-info

BUILD.md

@@ -11,6 +11,7 @@
   * [Reduced Operator Kernel Build](#Reduced-Operator-Kernel-Build)
   * [ONNX Runtime for Mobile Platforms](#ONNX-Runtime-for-Mobile-Platforms)
   * [ONNX Runtime Server (Linux)](#Build-ONNX-Runtime-Server-on-Linux)
+  * [Execution Provider Shared Libraries](#Execution-Provider-Shared-Libraries)
   * Execution Providers
     * [NVIDIA CUDA](#CUDA)
     * [NVIDIA TensorRT](#TensorRT)
@@ -59,24 +60,23 @@ Open Developer Command Prompt for Visual Studio version you are going to use. Th
 The default Windows CMake Generator is Visual Studio 2017, but you can also use the newer Visual Studio 2019 by passing `--cmake_generator "Visual Studio 16 2019"` to `.\build.bat`
 
 
-#### Linux/macOS
+#### Linux
 ```
 ./build.sh --config RelWithDebInfo --build_shared_lib --parallel
 ```
+
+
 ##### macOS
 By default, ORT is configured to be built for a minimum target macOS version of 10.12.
 The shared library in the release Nuget(s) and the Python wheel may be installed on macOS versions of 10.12+.
 
-If you would like to use [Xcode](https://developer.apple.com/xcode/) to build the onnxruntime for x86_64 macOS, use
-* With Xcode 11
+If you would like to use [Xcode](https://developer.apple.com/xcode/) to build the onnxruntime for x86_64 macOS, please add the --user_xcode argument in the command line
    ```
    ./build.sh --config RelWithDebInfo --build_shared_lib --parallel --use_xcode
    ```
-* With Xcode 12
-   ```
-   ./build.sh --config RelWithDebInfo --build_shared_lib --parallel --use_xcode \
-              --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=x86_64
-   ```
+While without this flag, the cmake build generator will be Unix makefile by default.
+Also, if you want to try cross compiling for Apple Silicon in an Intel-based MacOS machine, please add the argument --osx_arch arm64 with a cmake > 3.19, however the unit tests will be skipped due to the incompatible CPU instruction set.
+
 
 #### Notes
 
@@ -177,6 +177,24 @@ Nuget packages are created under <native_build_dir>\nuget-artifacts
 
 ---
 
+## Execution Provider Shared Libraries
+The DNNL, TensorRT, and OpenVINO providers are built as shared libraries vs being statically linked into the main onnxruntime. This enables them to be loaded only when needed, and if the dependent libraries of the provider are not installed onnxruntime will still run fine, it just will not be able to use that provider. For non shared library providers, all dependencies of the provider must exist to load onnxruntime.
+
+### Built files
+On Windows, shared provider libraries will be named 'onnxruntime_providers_\*.dll' (for example onnxruntime_providers_openvino.dll).
+On Unix, they will be named 'libonnxruntime_providers_\*.so'
+On Mac, they will be named 'libonnxruntime_providers_\*.dylib'.
+
+There is also a shared library that shared providers depend on called onnxruntime_providers_shared (with the same naming convension applied as above).
+
+Note: It is not recommended to put these libraries in a system location or added to a library search path (like LD_LIBRARY_PATH on Unix). If multiple versions of onnxruntime are installed on the system this can make them find the wrong libraries and lead to undefined behavior.
+
+### Loading the shared providers
+Shared provider libraries are loaded by the onnxruntime code (do not load or depend on them in your client code). The API for registering shared or non shared providers is identical, the difference is that shared ones will be loaded at runtime when the provider is added to the session options (through a call like OrtSessionOptionsAppendExecutionProvider_OpenVINO or SessionOptionsAppendExecutionProvider_OpenVINO in the C API).
+If a shared provider library cannot be loaded (if the file doesn't exist, or its dependencies don't exist or not in the path) then an error will be returned.
+
+The onnxruntime code will look for the provider shared libraries in the same location as the onnxruntime shared library is (or the executable statically linked to the static library version).
+
 ## Execution Providers
 
 ### CUDA
@@ -235,6 +253,8 @@ See more information on the TensorRT Execution Provider [here](./docs/execution_
    * The path to TensorRT installation must be provided via the `--tensorrt_home` parameter.
 
 #### Build Instructions
+Note that TensorRT is built as a [shared provider library](#Execution-Provider-Shared-Libraries)
+
 ##### Windows
 ```
 .\build.bat --cudnn_home <path to cuDNN home> --cuda_home <path to CUDA home> --use_tensorrt --tensorrt_home <path to TensorRT home>
@@ -309,6 +329,8 @@ See more information on DNNL and MKL-ML [here](./docs/execution_providers/DNNL-E
 #### Build Instructions
 The DNNL execution provider can be built for Intel CPU or GPU. To build for Intel GPU, install [Intel SDK for OpenCL Applications](https://software.intel.com/content/www/us/en/develop/tools/opencl-sdk.html). Install the latest GPU driver - [Windows graphics driver](https://downloadcenter.intel.com/product/80939/Graphics), [Linux graphics compute runtime and OpenCL driver](https://github.com/intel/compute-runtime/releases).
 
+Note that DNNL is built as a [shared provider library](#Execution-Provider-Shared-Libraries)
+
 ##### Windows
 `.\build.bat --use_dnnl`
 
@@ -365,6 +387,8 @@ See more information on the OpenVINO Execution Provider [here](./docs/execution_
 
 
 #### Build Instructions
+Note that OpenVINO is built as a [shared provider library](#Execution-Provider-Shared-Libraries)
+
 ##### Windows
 ```
 .\build.bat --config RelWithDebInfo --use_openvino <hardware_option>

cgmanifests/cgmanifest.json

@@ -343,7 +343,7 @@
          "component": {
             "type": "git",
             "git": {
-               "commitHash": "f97fd8664fa824e5d081d776cf4a811dbf09bca4",
+               "commitHash": "bc8ce45b35ade9b8f08d88d4d9fdfd12b4f6f310",
                "repositoryUrl": "https://github.com/pypa/manylinux"
             },
             "comments": "For building our CI build docker image"
@@ -363,20 +363,20 @@
          "component": {
             "type": "git",
             "git": {
-               "commitHash": "439c93d51f45c50541fc755b597725168ecd939a",
+               "commitHash": "9cf6752276e6fcfd0c23fdb064ad27f448aaaf75",
                "repositoryUrl": "https://github.com/python/cpython"
             },
-            "comments": "Python 3.9.0rc1"
+            "comments": "Python 3.9.0"
          }
       },
       {
          "component": {
             "type": "git",
             "git": {
-               "commitHash": "580fbb018fd0844806119614d752b41fc69660f9",
+               "commitHash": "db455296be5f792b8c12b7cd7f3962b52e4f44ee",
                "repositoryUrl": "https://github.com/python/cpython"
             },
-            "comments": "Python 3.8.5"
+            "comments": "Python 3.8.6"
          }
       },
       {
@@ -403,10 +403,10 @@
          "component": {
             "type": "git",
             "git": {
-               "commitHash": "e5f6aba872e66bfd86eb592214696a519cded197",
+               "commitHash": "426b022776672fdf3d71ddd98d89af341c88080f",
                "repositoryUrl": "https://github.com/python/cpython"
             },
-            "comments": "Python 3.5.9"
+            "comments": "Python 3.5.10"
          }
       },
       {

cgmanifests/submodules/cgmanifest.json

@@ -1,6 +1,50 @@
 {
   "Version": 1,
   "Registrations": [
+    {
+      "Component": {
+        "Type": "other",
+        "other": {
+          "Name": "automake",
+          "Version": "1.16.2",
+          "DownloadUrl": "http://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.gz"
+        },
+        "comments": "manylinux dependency"
+      }
+    },
+    {
+      "Component": {
+        "Type": "other",
+        "other": {
+          "Name": "libtool",
+          "Version": "2.4.6",
+          "DownloadUrl": "http://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.gz"
+        },
+        "comments": "manylinux dependency"
+      }
+    },
+    {
+      "Component": {
+        "Type": "other",
+        "other": {
+          "Name": "sqlite",
+          "Version": "3330000",
+          "DownloadUrl": "https://www.sqlite.org/2020/sqlite-autoconf-3330000.tar.gz"
+        },
+        "comments": "manylinux dependency"
+      }
+    },
+    {
+      "Component": {
+        "Type": "other",
+        "other": {
+          "Name": "git",
+          "Version": "2.29.1",
+          "DownloadUrl": "https://www.kernel.org/pub/software/scm/git/git-2.29.1.tar.gz"
+        },
+        "comments": "manylinux dependency"
+      }
+    },
     {
       "component": {
         "type": "git",
@@ -345,7 +389,7 @@
       "component": {
         "type": "git",
         "git": {
-          "commitHash": "63a586914d51fdfd6d8668e81de043e3d9b0b382",
+          "commitHash": "994c6181247d7b419b28889fc57d5817e2089419",
           "repositoryUrl": "https://github.com/onnx/onnx"
         },
         "comments": "git submodule at cmake/external/onnx"

cgmanifests/submodules/generate_submodule_cgmanifest.py

@@ -4,10 +4,54 @@
 import os
 import subprocess
 import sys
+import re
 
 SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
 REPO_DIR = os.path.normpath(os.path.join(SCRIPT_DIR, "..", ".."))
 
+package_name = None
+package_filename = None
+package_url = None
+
+registrations = []
+
+with open(os.path.join(REPO_DIR,'tools','ci_build','github','linux','docker','manylinux2014_build_scripts','build_env.sh'), "r") as f:
+  for line in f:
+    if not line.strip():      
+      package_name = None
+      package_filename = None
+      package_url = None
+    if package_filename is None:
+      m = re.match("(.+?)_ROOT=(.*)$",line)
+      if m != None:
+        package_name = m.group(1)
+        package_filename = m.group(2)
+      else:
+        m = re.match("(.+?)_AUTOCONF_VERSION=(.*)$",line)
+        if m != None:
+          package_name = m.group(1)
+          package_filename = m.group(2)        
+    elif package_url is None:
+      m = re.match("(.+?)_DOWNLOAD_URL=(.+)$",line)
+      if m != None:
+        package_url = m.group(2) + "/" + package_filename + ".tar.gz"
+        registration = {
+          "Component": {
+            "Type": "other",
+            "other": {
+                "Name": package_name.lower(),
+                "Version": package_filename.split("-")[-1],
+                "DownloadUrl": package_url,
+            },
+            "comments": "manylinux dependency"
+          }
+        }
+        registrations.append(registration)
+        package_name = None
+        package_filename = None
+        package_url = None
+
+
 proc = subprocess.run(
     ["git", "submodule", "foreach", "--quiet", "--recursive", "{} {} $toplevel/$sm_path".format(
         sys.executable, os.path.join(SCRIPT_DIR, "print_submodule_info.py"))],
@@ -17,7 +61,7 @@
     stderr=subprocess.PIPE,
     universal_newlines=True)
 
-registrations = []
+
 submodule_lines = proc.stdout.splitlines()
 for submodule_line in submodule_lines:
     (absolute_path, url, commit) = submodule_line.split(" ")

cmake/external/pybind11.cmake

@@ -8,7 +8,7 @@ if(NOT TARGET pybind11::module)
 
   set(pybind11_INCLUDE_DIRS ${CMAKE_CURRENT_BINARY_DIR}/pybind11/src/pybind11/include)
   set(pybind11_URL https://github.com/pybind/pybind11.git)
-  set(pybind11_TAG v2.4.0)
+  set(pybind11_TAG v2.6.1)
 
   ExternalProject_Add(pybind11
         PREFIX pybind11
@@ -21,4 +21,4 @@ if(NOT TARGET pybind11::module)
   set(pybind11_dep pybind11)
 else()
   set(pybind11_lib pybind11::module)
-endif()
+endif()

cmake/onnxruntime_java.cmake

@@ -100,13 +100,31 @@ if (CMAKE_SYSTEM_NAME STREQUAL "Android")
   file(MAKE_DIRECTORY ${ANDROID_PACKAGE_OUTPUT_DIR})
 endif()
 
-# Set platform and ach for packaging
+# Set platform and arch for packaging
+# Checks the names set by MLAS on non-Windows platforms first
 if (CMAKE_SYSTEM_NAME STREQUAL "Android")
   set(JNI_ARCH ${ANDROID_ABI})
-elseif (CMAKE_SIZEOF_VOID_P EQUAL "8")
+elseif (ARM64)
+  set(JNI_ARCH aarch64)
+elseif (X86_64)
   set(JNI_ARCH x64)
+elseif (POWER)
+  set(JNI_ARCH ppc64)
 else()
-  message(FATAL_ERROR "Java is currently not supported for x86 architecture")
+  # Now mirror the checks used with MSVC
+  if(MSVC)
+    if(onnxruntime_target_platform STREQUAL "ARM64")
+      set(JNI_ARCH aarch64)
+    elseif(onnxruntime_target_platform STREQUAL "x64")
+      set(JNI_ARCH x64)
+    else()
+      # if everything else failed then we're on a 32-bit arch and Java isn't supported
+      message(FATAL_ERROR "Java is currently not supported on 32-bit x86 architecture")
+    endif()
+  else()
+    # if everything else failed then we're on a 32-bit arch and Java isn't supported
+    message(FATAL_ERROR "Java is currently not supported on 32-bit x86 architecture")
+  endif()
 endif()
 
 if (WIN32)

cmake/onnxruntime_mlas.cmake

@@ -124,6 +124,15 @@ if(MSVC)
     )
   endif()
 else()
+  if (CMAKE_OSX_ARCHITECTURES STREQUAL "arm64")
+    set(ARM64 TRUE)
+  elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "arm")
+    set(ARM TRUE)
+  elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "x86_64")
+    set(X86_64 TRUE)
+  elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "i386")
+    set(X86 TRUE)
+  endif()
   if (CMAKE_SYSTEM_NAME STREQUAL "Android")
     if (CMAKE_ANDROID_ARCH_ABI STREQUAL "armeabi-v7a")
       set(ARM TRUE)
@@ -136,22 +145,15 @@ else()
     endif()
   elseif(CMAKE_SYSTEM_NAME STREQUAL "iOS" OR CMAKE_SYSTEM_NAME STREQUAL "iOSCross")
     set(IOS TRUE)
-    if (CMAKE_OSX_ARCHITECTURES STREQUAL "arm64")
-      set(ARM64 TRUE)
-    elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "arm")
-      set(ARM TRUE)
-    elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "x86_64")
-      set(X86_64 TRUE)
-    elseif (CMAKE_OSX_ARCHITECTURES STREQUAL "i386")
-      set(X86 TRUE)
-    endif()
   else()
     execute_process(
       COMMAND ${CMAKE_C_COMPILER} -dumpmachine
       OUTPUT_VARIABLE dumpmachine_output
       ERROR_QUIET
     )
-    if(dumpmachine_output MATCHES "^arm.*")
+    if(dumpmachine_output MATCHES "^arm64.*")
+      set(ARM64 TRUE)
+    elseif(dumpmachine_output MATCHES "^arm.*")
       set(ARM TRUE)
     elseif(dumpmachine_output MATCHES "^aarch64.*")
       set(ARM64 TRUE)