|
| area-Diagnostics-coreclr | @tommcdon | @tommcdon | |
| area-Diagnostics-mono | @steveisok | @lateralusX @mdh1418 | |
-| area-EnC-mono | @marek-safar | @lambdageek | Hot Reload on WebAssembly, Android, iOS, etc |
+| area-EnC-mono | @SamMonoRT | @lambdageek | Hot Reload on WebAssembly, Android, iOS, etc |
| area-ExceptionHandling-coreclr | @mangod9 | @janvorli | |
| area-Extensions-Caching | @ericstj | @dotnet/area-extensions-caching | |
| area-Extensions-Configuration | @ericstj | @dotnet/area-extensions-configuration | |
diff --git a/docs/design/coreclr/botr/readytorun-format.md b/docs/design/coreclr/botr/readytorun-format.md
index 2c63997332e487..d68099747d8f66 100644
--- a/docs/design/coreclr/botr/readytorun-format.md
+++ b/docs/design/coreclr/botr/readytorun-format.md
@@ -179,6 +179,7 @@ The following section types are defined and described later in this document:
| PgoInstrumentationData | 117 | Image (added in V5.2)
| ManifestAssemblyMvids | 118 | Image (added in V5.3)
| CrossModuleInlineInfo | 119 | Image (added in V6.3)
+| HotColdMap | 120 | Image (added in V8.0)
## ReadyToRunSectionType.CompilerIdentifier
@@ -644,6 +645,15 @@ The entry of the hashtable is a counted sequence of compressed unsigned integers
This section may be included in addition to a InliningInfo2 section.
+## ReadyToRunSectionType.HotColdMap (v8.0+)
+In ReadyToRun 8.0+, the format supports splitting a method into hot and cold parts so that they are not located together. This hot-cold map section captures the information about how methods are split so that the runtime can locate them for various services.
+
+For every method that is split, there is a single entry in the section. Each entry has two unsigned 32-bit integers. The first integer is the runtime function index of the cold part and the second integer is the runtime function index of the hot part.
+
+The methods in this table are sorted by their hot part runtime function indices, which are also sorted by their cold part runtime function indices because we always emit the cold part in the same order as the hot parts, or by their RVAs because the runtime function table itself is sorted by the RVAs.
+
+This section may not exist if no method is split - this happens when the `--hot-cold-splitting` flag is not specified during compilation, or the compiler decides it should not split any methods.
+
# Native Format
Native format is set of encoding patterns that allow persisting type system data in a binary format that is
diff --git a/docs/design/features/hybrid-globalization.md b/docs/design/features/globalization-hybrid-mode.md
similarity index 89%
rename from docs/design/features/hybrid-globalization.md
rename to docs/design/features/globalization-hybrid-mode.md
index 5d08676ea642e4..0e6ed5d7c1e7f2 100644
--- a/docs/design/features/hybrid-globalization.md
+++ b/docs/design/features/globalization-hybrid-mode.md
@@ -1,6 +1,11 @@
# Hybrid Globalization
-Description, purpose and instruction how to use.
+Originally, internalization data is loaded from ICU data files. In `HybridGlobalization` mode we are leveraging the platform-native internationalization APIs, where it is possible, to allow for loading smaller ICU data files. We still need to rely on ICU files because for a bunch of globalization data no API equivalent is available. For some existing equivalents, the behavior does not fully match the original. The differences you can expect after switching on the mode are listed in this document. Expected size savings can be found under each platform section below.
+
+Hybrid has lower priority than Invariant. To switch on the mode set the property in the build file:
+```
+true
+```
## Behavioral differences
@@ -8,7 +13,9 @@ Hybrid mode does not use ICU data for some functions connected with globalizatio
### WASM
-For WebAssembly in Browser we are using Web API instead of some ICU data. Ideally, we would use `System.Runtime.InteropServices.JavaScript` to call JS code from inside of C# but we cannot reference any assemblies from inside of `System.Private.CoreLib`. That is why we are using iCalls instead.
+For WebAssembly in Browser we are using Web API instead of some ICU data. Ideally, we would use `System.Runtime.InteropServices.JavaScript` to call JS code from inside of C# but we cannot reference any assemblies from inside of `System.Private.CoreLib`. That is why we are using iCalls instead. The host support depends on used Web API functions support - see **dependencies** in each section.
+
+Hybrid has higher priority than sharding or custom modes, described in globalization-icu-wasm.md.
**SortKey**
@@ -33,6 +40,12 @@ Hybrid case change, same as ICU-based, does not support code points expansion e.
ICU-based case change does not respect final-sigma rule, but hybrid does, so "ΒΌΛΟΣ" -> "βόλος", not "βόλοσ".
+Dependencies:
+- [String.prototype.toUpperCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toUpperCase)
+- [String.prototype.toLoweCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLowerCase)
+- [String.prototype.toLocaleUpperCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleUpperCase)
+- [String.prototype.toLocaleLoweCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleLowerCase)
+
**String comparison**
Affected public APIs:
@@ -40,8 +53,12 @@ Affected public APIs:
- String.Compare,
- String.Equals.
+Dependencies:
+- [String.prototype.localeCompare()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare)
+
The number of `CompareOptions` and `StringComparison` combinations is limited. Originally supported combinations can be found [here for CompareOptions](https://learn.microsoft.com/dotnet/api/system.globalization.compareoptions) and [here for StringComparison](https://learn.microsoft.com/dotnet/api/system.stringcomparison).
+
- `IgnoreWidth` is not supported because there is no equivalent in Web API. Throws `PlatformNotSupportedException`.
``` JS
let high = String.fromCharCode(65281) // %uff83 = ﾃ
@@ -196,6 +213,10 @@ Affected public APIs:
- String.StartsWith
- String.EndsWith
+Dependencies:
+- [String.prototype.normalize()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize)
+- [String.prototype.localeCompare()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare)
+
Web API does not expose locale-sensitive endsWith/startsWith function. As a workaround, both strings get normalized and weightless characters are removed. Resulting strings are cut to the same length and comparison is performed. This approach, beyond having the same compare option limitations as described under **String comparison**, has additional limitations connected with the workaround used. Because we are normalizing strings to be able to cut them, we cannot calculate the match length on the original strings. Methods that calculate this information throw PlatformNotSupported exception:
- [CompareInfo.IsPrefix](https://learn.microsoft.com/en-us/dotnet/api/system.globalization.compareinfo.isprefix?view=net-8.0#system-globalization-compareinfo-isprefix(system-readonlyspan((system-char))-system-readonlyspan((system-char))-system-globalization-compareoptions-system-int32@))
diff --git a/docs/project/list-of-diagnostics.md b/docs/project/list-of-diagnostics.md
index 7c814cdf68ae87..d6783d80ecf90a 100644
--- a/docs/project/list-of-diagnostics.md
+++ b/docs/project/list-of-diagnostics.md
@@ -219,8 +219,8 @@ The diagnostic id values reserved for .NET Libraries analyzer warnings are `SYSL
| __`SYSLIB1100`__ | Configuration binding generator: type is not supported. |
| __`SYSLIB1101`__ | Configuration binding generator: property on type is not supported. |
| __`SYSLIB1102`__ | Configuration binding generator: project's language version must be at least C# 11.|
-| __`SYSLIB1103`__ | *_`SYSLIB1100`-`SYSLIB1118` reserved for Microsoft.Extensions.Configuration.Binder.SourceGeneration.* |
-| __`SYSLIB1104`__ | *_`SYSLIB1100`-`SYSLIB1118` reserved for Microsoft.Extensions.Configuration.Binder.SourceGeneration.* |
+| __`SYSLIB1103`__ | Configuration binding generator: value types are invalid inputs to configuration 'Bind' methods.* |
+| __`SYSLIB1104`__ | Configuration binding generator: Generator cannot determine the target configuration type.* |
| __`SYSLIB1105`__ | *_`SYSLIB1100`-`SYSLIB1118` reserved for Microsoft.Extensions.Configuration.Binder.SourceGeneration.* |
| __`SYSLIB1106`__ | *_`SYSLIB1100`-`SYSLIB1118` reserved for Microsoft.Extensions.Configuration.Binder.SourceGeneration.* |
| __`SYSLIB1107`__ | *_`SYSLIB1100`-`SYSLIB1118` reserved for Microsoft.Extensions.Configuration.Binder.SourceGeneration.* |
diff --git a/eng/CodeAnalysis.src.globalconfig b/eng/CodeAnalysis.src.globalconfig
index d83299c431cce5..0fdc74157b6c6a 100644
--- a/eng/CodeAnalysis.src.globalconfig
+++ b/eng/CodeAnalysis.src.globalconfig
@@ -471,6 +471,9 @@ dotnet_diagnostic.CA1859.severity = warning
# CA1860: Avoid using 'Enumerable.Any()' extension method
dotnet_diagnostic.CA1860.severity = warning
+# CA1861: Avoid constant arrays as arguments
+dotnet_diagnostic.CA1861.severity = warning
+
# CA2000: Dispose objects before losing scope
dotnet_diagnostic.CA2000.severity = none
diff --git a/eng/CodeAnalysis.test.globalconfig b/eng/CodeAnalysis.test.globalconfig
index 84c7837a7e536a..ae1dc7eb9872fc 100644
--- a/eng/CodeAnalysis.test.globalconfig
+++ b/eng/CodeAnalysis.test.globalconfig
@@ -468,6 +468,9 @@ dotnet_diagnostic.CA1859.severity = none
# CA1860: Avoid using 'Enumerable.Any()' extension method
dotnet_diagnostic.CA1860.severity = none
+# CA1861: Avoid constant arrays as arguments
+dotnet_diagnostic.CA1861.severity = none
+
# CA2000: Dispose objects before losing scope
dotnet_diagnostic.CA2000.severity = none
diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml
index b2d2ecbd58066b..b55a2045949060 100644
--- a/eng/Version.Details.xml
+++ b/eng/Version.Details.xml
@@ -1,8 +1,8 @@
-
+ https://github.com/dotnet/icu
- 389d19d09d3cf16ec0143dba065fcd704ab8e48c
+ d77dd854ec4d9a3c4a0c584421ee393642d631efhttps://github.com/dotnet/msquic
@@ -12,29 +12,29 @@
https://github.com/dotnet/wcf7f504aabb1988e9a093c1e74d8040bd52feb2f01
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312https://github.com/dotnet/llvm-project
@@ -44,29 +44,29 @@
https://github.com/dotnet/llvm-project76f334f354eb653a7b409a5319b591ea09df5a43
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312https://github.com/dotnet/llvm-project
@@ -85,14 +85,14 @@
8374d5fca634a93458c84414b1604c12f765d1ab
-
+ https://github.com/dotnet/cecil
- f283c6229347f2dc3a25b04f6611014d8871d0b0
+ ea84c8236c45ebace0073dbc01e0e75ffba65a65
-
+ https://github.com/dotnet/emsdk
- fb9e15863c4a8b4fa209096f29406cc6dbb8ef32
+ 63af043d58bd8d471da0e236e6255611e061cc4e
@@ -107,9 +107,9 @@
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
@@ -117,166 +117,166 @@
54eb3b811c57f5e94617d31a102fc9cb664ccdd5
-
+ https://github.com/dotnet/xliff-tasks
- 9e7fbcab4e5275f63c0cd37553ba426de9194309
+ 8fd12314c3e648c03a4189ba0bd1c3b09f5d6a01
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312
-
+ https://github.com/dotnet/llvm-project
- c01ca3bc8a420b3796d816c43bdd06e337c72ea6
+ 43fe12a7a5f6917dee47ca4de17477ed1ac2c312https://github.com/dotnet/runtime
@@ -303,58 +303,58 @@
https://github.com/dotnet/runtime93da0c966f9bc537dd77fbe2cbeb0bec4ff6a48b
-
+ https://github.com/dotnet/xharness
- 5d3b745c64e8092c2ce3ce1feca85d87d814f292
+ 230501dc45d8c96e995ceed1189852eb0ee08693
-
+ https://github.com/dotnet/xharness
- 5d3b745c64e8092c2ce3ce1feca85d87d814f292
+ 230501dc45d8c96e995ceed1189852eb0ee08693
-
+ https://github.com/dotnet/xharness
- 5d3b745c64e8092c2ce3ce1feca85d87d814f292
+ 230501dc45d8c96e995ceed1189852eb0ee08693
-
+ https://github.com/dotnet/arcade
- f4269875971171c278c2cd695e9ef94b6516e4be
+ 234e0726c7384ee84bf08550f2d16a1ff2d5c543
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
-
+ https://github.com/dotnet/hotreload-utils
- 38e5743db0070214e73fa04ed1524a254be95c1a
+ 1eec66c6cff38ddf124c6c740b9ccbe2a1144352
-
+ https://github.com/dotnet/runtime-assets
- 886b608e3f566153c047ba0fd31357404365ae4c
+ 32bcf870359408a0cd6f325b558d744f5bf56c93
-
+ https://github.com/dotnet/roslyn
- 0cb0d5796b336c6a746e6d14c129f2e2b44f4f5e
+ fb8455f2f9046b821b826aab81bf8faa00711065
-
+ https://github.com/dotnet/roslyn
- 0cb0d5796b336c6a746e6d14c129f2e2b44f4f5e
+ fb8455f2f9046b821b826aab81bf8faa00711065
-
+ https://github.com/dotnet/roslyn
- 0cb0d5796b336c6a746e6d14c129f2e2b44f4f5e
+ fb8455f2f9046b821b826aab81bf8faa00711065https://github.com/dotnet/roslyn-analyzers
@@ -364,18 +364,18 @@
https://github.com/dotnet/roslyn-analyzers2b6ab8d727ce73a78bcbf026ac75ea8a7c804daf
-
+ https://github.com/dotnet/sdk
- 2fd62c3936f5336b836f6b12df170aa0e90da767
+ dfdec2edb38ee24c5551125e1cdf12cf6117b6a7
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
-
+ https://dev.azure.com/dnceng/internal/_git/dotnet-optimization
- c85736cdab833e3cefa202f5e7e28f1fb6543604
+ b9af49c3ae71595e30d893960d9b2cecf83c9743
diff --git a/eng/Versions.props b/eng/Versions.props
index 38116216008637..5e154a36b58b3b 100644
--- a/eng/Versions.props
+++ b/eng/Versions.props
@@ -40,9 +40,9 @@
Any tools that contribute to the design-time experience should use the MicrosoftCodeAnalysisVersion_LatestVS property above to ensure
they do not break the local dev experience.
-->
- 4.7.0-1.23269.2
- 4.7.0-1.23269.2
- 4.7.0-1.23269.2
+ 4.7.0-2.23301.9
+ 4.7.0-2.23301.9
+ 4.7.0-2.23301.9
- 8.0.100-preview.2.23107.1
+ 8.0.100-preview.6.23303.2
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 2.5.1-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
- 8.0.0-beta.23268.1
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 2.5.1-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.3
+ 8.0.0-beta.23302.36.0.0-preview.1.102
@@ -105,12 +105,12 @@
6.0.08.0.0-preview.5.23279.2
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.21.0.0-alpha.1.23106.11.0.0-alpha.1.23106.1
@@ -139,27 +139,27 @@
4.5.08.0.0-preview.5.23279.2
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
- 8.0.0-beta.23273.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
+ 8.0.0-beta.23280.1
- 1.0.0-prerelease.23272.4
- 1.0.0-prerelease.23272.4
- 1.0.0-prerelease.23272.4
- 1.0.0-prerelease.23272.4
- 1.0.0-prerelease.23272.4
- 1.0.0-prerelease.23272.4
+ 1.0.0-prerelease.23303.6
+ 1.0.0-prerelease.23303.6
+ 1.0.0-prerelease.23303.6
+ 1.0.0-prerelease.23303.6
+ 1.0.0-prerelease.23303.6
+ 1.0.0-prerelease.23303.616.11.23-beta1.23063.12.0.0-beta4.22564.1
@@ -179,10 +179,10 @@
1.1.017.4.0-preview-20220707-01
- 8.0.0-prerelease.23272.1
- 8.0.0-prerelease.23272.1
- 8.0.0-prerelease.23272.1
- 8.0.0-alpha.0.23272.2
+ 8.0.0-prerelease.23279.1
+ 8.0.0-prerelease.23279.1
+ 8.0.0-prerelease.23279.1
+ 8.0.0-alpha.0.23280.22.4.21.0.02.4.5
@@ -211,39 +211,39 @@
8.0.0-preview.5.23279.2
- 0.11.4-alpha.23265.2
+ 0.11.4-alpha.23301.1
- 8.0.0-preview.4.23177.3
+ 8.0.0-preview.6.23280.12.1.78.0.0-alpha.1.23180.2
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
- 8.0.0-preview.6.23277.2
+ 8.0.0-preview.6.23301.1$(MicrosoftNETWorkloadEmscriptenCurrentManifest80100TransportVersion)1.1.87-gba258badda1.0.0-v3.14.0.5722
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
- 14.0.0-alpha.1.23273.3
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.2
+ 14.0.0-alpha.1.23301.21.0.0-alpha.1.23106.11.0.0-alpha.1.23106.1
diff --git a/eng/common/sdk-task.ps1 b/eng/common/sdk-task.ps1
index e10a5968797463..6c4ac6fec1a99a 100644
--- a/eng/common/sdk-task.ps1
+++ b/eng/common/sdk-task.ps1
@@ -64,7 +64,7 @@ try {
$GlobalJson.tools | Add-Member -Name "vs" -Value (ConvertFrom-Json "{ `"version`": `"16.5`" }") -MemberType NoteProperty
}
if( -not ($GlobalJson.tools.PSObject.Properties.Name -match "xcopy-msbuild" )) {
- $GlobalJson.tools | Add-Member -Name "xcopy-msbuild" -Value "17.4.1" -MemberType NoteProperty
+ $GlobalJson.tools | Add-Member -Name "xcopy-msbuild" -Value "17.6.0-2" -MemberType NoteProperty
}
if ($GlobalJson.tools."xcopy-msbuild".Trim() -ine "none") {
$xcopyMSBuildToolsFolder = InitializeXCopyMSBuild $GlobalJson.tools."xcopy-msbuild" -install $true
diff --git a/eng/common/templates/job/job.yml b/eng/common/templates/job/job.yml
index 44ad26abf54ba2..e20ee3a983cb05 100644
--- a/eng/common/templates/job/job.yml
+++ b/eng/common/templates/job/job.yml
@@ -75,6 +75,10 @@ jobs:
- ${{ if eq(parameters.enableRichCodeNavigation, 'true') }}:
- name: EnableRichCodeNavigation
value: 'true'
+ # Retry signature validation up to three times, waiting 2 seconds between attempts.
+ # See https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#retry-untrusted-root-failures
+ - name: NUGET_EXPERIMENTAL_CHAIN_BUILD_RETRY_POLICY
+ value: 3,2000
- ${{ each variable in parameters.variables }}:
# handle name-value variable syntax
# example:
@@ -83,7 +87,7 @@ jobs:
- ${{ if ne(variable.name, '') }}:
- name: ${{ variable.name }}
value: ${{ variable.value }}
-
+
# handle variable groups
- ${{ if ne(variable.group, '') }}:
- group: ${{ variable.group }}
@@ -169,7 +173,7 @@ jobs:
- ${{ if eq(parameters.enableMicrobuild, 'true') }}:
- ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:
- task: MicroBuildCleanup@1
- displayName: Execute Microbuild cleanup tasks
+ displayName: Execute Microbuild cleanup tasks
condition: and(always(), in(variables['_SignType'], 'real', 'test'), eq(variables['Agent.Os'], 'Windows_NT'))
continueOnError: ${{ parameters.continueOnError }}
env:
@@ -219,7 +223,7 @@ jobs:
displayName: Publish XUnit Test Results
inputs:
testResultsFormat: 'xUnit'
- testResultsFiles: '*.xml'
+ testResultsFiles: '*.xml'
searchFolder: '$(Build.SourcesDirectory)/artifacts/TestResults/$(_BuildConfig)'
testRunTitle: ${{ coalesce(parameters.testRunTitle, parameters.name, '$(System.JobName)') }}-xunit
mergeTestResults: ${{ parameters.mergeTestResults }}
@@ -230,7 +234,7 @@ jobs:
displayName: Publish TRX Test Results
inputs:
testResultsFormat: 'VSTest'
- testResultsFiles: '*.trx'
+ testResultsFiles: '*.trx'
searchFolder: '$(Build.SourcesDirectory)/artifacts/TestResults/$(_BuildConfig)'
testRunTitle: ${{ coalesce(parameters.testRunTitle, parameters.name, '$(System.JobName)') }}-trx
mergeTestResults: ${{ parameters.mergeTestResults }}
diff --git a/eng/common/templates/steps/source-build.yml b/eng/common/templates/steps/source-build.yml
index a97a185a367a92..1100521834a943 100644
--- a/eng/common/templates/steps/source-build.yml
+++ b/eng/common/templates/steps/source-build.yml
@@ -68,6 +68,11 @@ steps:
runtimeOsArgs='/p:RuntimeOS=${{ parameters.platform.runtimeOS }}'
fi
+ baseOsArgs=
+ if [ '${{ parameters.platform.baseOS }}' != '' ]; then
+ baseOsArgs='/p:BaseOS=${{ parameters.platform.baseOS }}'
+ fi
+
publishArgs=
if [ '${{ parameters.platform.skipPublishValidation }}' != 'true' ]; then
publishArgs='--publish'
@@ -86,6 +91,7 @@ steps:
$internalRestoreArgs \
$targetRidArgs \
$runtimeOsArgs \
+ $baseOsArgs \
/p:SourceBuildNonPortable=${{ parameters.platform.nonPortable }} \
/p:ArcadeBuildFromSource=true \
/p:AssetManifestFileName=$assetManifestFileName
diff --git a/eng/common/tools.ps1 b/eng/common/tools.ps1
index 38cf94ff88c6f3..ffe0b4e2dfc4d7 100644
--- a/eng/common/tools.ps1
+++ b/eng/common/tools.ps1
@@ -384,8 +384,8 @@ function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements =
# If the version of msbuild is going to be xcopied,
# use this version. Version matches a package here:
- # https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=dotnet-eng&package=RoslynTools.MSBuild&protocolType=NuGet&version=17.4.1&view=overview
- $defaultXCopyMSBuildVersion = '17.4.1'
+ # https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet-eng/NuGet/RoslynTools.MSBuild/versions/17.6.0-2
+ $defaultXCopyMSBuildVersion = '17.6.0-2'
if (!$vsRequirements) {
if (Get-Member -InputObject $GlobalJson.tools -Name 'vs') {
diff --git a/eng/packaging.targets b/eng/packaging.targets
index 29291f1c6e6816..3e5f6a9e582865 100644
--- a/eng/packaging.targets
+++ b/eng/packaging.targets
@@ -16,6 +16,8 @@
falsetrue
+
+ true$(MSBuildThisFileDirectory)useSharedDesignerContext.txtPACKAGE.md
diff --git a/eng/pipelines/common/evaluate-default-paths.yml b/eng/pipelines/common/evaluate-default-paths.yml
index 70be16e61bad4b..8527d6ae77aaa8 100644
--- a/eng/pipelines/common/evaluate-default-paths.yml
+++ b/eng/pipelines/common/evaluate-default-paths.yml
@@ -17,7 +17,9 @@ parameters:
src/libraries/sendtohelix-wasi.targets
src/mono/mono/**/*wasm*
src/mono/nuget/Microsoft.NET.Runtime.WebAssembly.Sdk/*
+ src/mono/nuget/Microsoft.NET.Runtime.WebAssembly.Wasi.Sdk/*
src/mono/nuget/Microsoft.NET.Runtime.wasm.Sample.Mono/*
+ src/mono/nuget/Microsoft.NET.Sdk.WebAssembly.Pack/*
src/mono/nuget/Microsoft.NETCore.BrowserDebugHost.Transport/*
src/mono/nuget/Microsoft.NET.Workload*
src/mono/sample/wasm/*
@@ -27,6 +29,7 @@ parameters:
src/tasks/WasmAppBuilder/*
src/tasks/WasmBuildTasks/*
src/tasks/WorkloadBuildTasks/*
+ src/tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks/*
src/tests/Common/wasm-test-runner/*
]
_wasm_pipelines: [
@@ -185,6 +188,8 @@ jobs:
- src/mono/nuget/Microsoft.NET.Runtime.MonoAOTCompiler.Task/*
- src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/*
- src/mono/nuget/Microsoft.NET.Runtime.WebAssembly.Sdk/*
+ - src/mono/nuget/Microsoft.NET.Runtime.WebAssembly.Wasi.Sdk/*
+ - src/mono/nuget/Microsoft.NET.Sdk.WebAssembly.Pack/*
- src/mono/nuget/Microsoft.NET.Runtime.wasm.Sample.Mono/*
- src/mono/nuget/Microsoft.NET.Workload*
- src/mono/nuget/Microsoft.NETCore.BrowserDebugHost.Transport/*
@@ -248,6 +253,7 @@ jobs:
- src/mono/wasm/debugger/*
- src/mono/wasm/Wasm.Build.Tests/*
- src/mono/nuget/Microsoft.NET.Runtime*
+ src/mono/nuget/Microsoft.NET.Sdk.WebAssembly.Pack/*
- src/mono/nuget/Microsoft.NET.Workload*
- src/mono/nuget/Microsoft.NETCore.BrowserDebugHost.Transport/*
- ${{ parameters._const_paths._always_exclude }}
diff --git a/eng/pipelines/common/global-build-job.yml b/eng/pipelines/common/global-build-job.yml
index 12f72b0d9e265f..cf93d34062b7da 100644
--- a/eng/pipelines/common/global-build-job.yml
+++ b/eng/pipelines/common/global-build-job.yml
@@ -162,6 +162,7 @@ jobs:
- template: /eng/common/templates/steps/source-build.yml
parameters:
platform:
+ baseOS: ${{ parameters.baseOS }}
buildScript: $(Build.SourcesDirectory)$(dir)build$(scriptExt)
nonPortable: ${{ parameters.isNonPortableSourceBuild }}
targetRID: ${{ parameters.targetRid }}
diff --git a/eng/pipelines/common/platform-matrix.yml b/eng/pipelines/common/platform-matrix.yml
index 4f0d6bfd5aeb3c..bae79645c0c6dc 100644
--- a/eng/pipelines/common/platform-matrix.yml
+++ b/eng/pipelines/common/platform-matrix.yml
@@ -307,6 +307,7 @@ jobs:
container: SourceBuild_centos_x64 # Run the unknown-rid build on a platform with a known RID so our RID graph tooling can automatically add it to the RID graph.
jobParameters:
runtimeFlavor: ${{ parameters.runtimeFlavor }}
+ baseOS: linux
buildConfig: ${{ parameters.buildConfig }}
helixQueueGroup: ${{ parameters.helixQueueGroup }}
isSourceBuild: true
@@ -579,6 +580,26 @@ jobs:
helixQueueGroup: ${{ parameters.helixQueueGroup }}
${{ insert }}: ${{ parameters.jobParameters }}
+# Android x64 with Docker-in-Docker
+
+- ${{ if containsValue(parameters.platforms, 'android_x64_docker') }}:
+ - template: xplat-setup.yml
+ parameters:
+ jobTemplate: ${{ parameters.jobTemplate }}
+ helixQueuesTemplate: ${{ parameters.helixQueuesTemplate }}
+ variables: ${{ parameters.variables }}
+ osGroup: android
+ archType: x64
+ targetRid: android-x64
+ platform: android_x64
+ shouldContinueOnError: ${{ parameters.shouldContinueOnError }}
+ container: android_docker
+ jobParameters:
+ runtimeFlavor: mono
+ buildConfig: ${{ parameters.buildConfig }}
+ helixQueueGroup: ${{ parameters.helixQueueGroup }}
+ ${{ insert }}: ${{ parameters.jobParameters }}
+
# Android x86
- ${{ if containsValue(parameters.platforms, 'android_x86') }}:
diff --git a/eng/pipelines/common/templates/pipeline-with-resources.yml b/eng/pipelines/common/templates/pipeline-with-resources.yml
index 9d3815ec3ded11..e135269d9e571a 100644
--- a/eng/pipelines/common/templates/pipeline-with-resources.yml
+++ b/eng/pipelines/common/templates/pipeline-with-resources.yml
@@ -33,10 +33,14 @@ resources:
image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64-alpine
env:
ROOTFS_DIR: /crossrootfs/arm64
- # This container contains all required toolsets to build for Android and for Linux with bionic libc.
+ # This container contains all required toolsets to build for Android and for Linux with bionic libc.
- container: linux_bionic
- image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-android
+ image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-android
+
+ # This container contains all required toolsets to build for Android as well as tooling to build docker images.
+ - container: android_docker
+ image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-android-docker
- container: linux_x64
image: mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-amd64
diff --git a/eng/pipelines/coreclr/perf-non-wasm-jobs.yml b/eng/pipelines/coreclr/perf-non-wasm-jobs.yml
index cd8f01bf1652c2..3d943fce71b44f 100644
--- a/eng/pipelines/coreclr/perf-non-wasm-jobs.yml
+++ b/eng/pipelines/coreclr/perf-non-wasm-jobs.yml
@@ -102,6 +102,27 @@ jobs:
archiveType: tar
tarCompression: gz
+ # build NativeAOT iOS scenarios
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/common/global-build-job.yml
+ buildConfig: release
+ runtimeFlavor: mono
+ platforms:
+ - ios_arm64
+ jobParameters:
+ nameSuffix: iOSNativeAOT
+ isOfficialBuild: false
+ extraStepsTemplate: /eng/pipelines/coreclr/templates/build-perf-sample-apps.yml
+ extraStepsParameters:
+ rootFolder: '$(Build.SourcesDirectory)/artifacts/'
+ includeRootFolder: true
+ displayName: iOS NativeAOT Artifacts
+ artifactName: iOSNativeAOTarm64
+ archiveExtension: '.tar.gz'
+ archiveType: tar
+ tarCompression: gz
+
# build mono
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
@@ -143,6 +164,24 @@ jobs:
runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
logicalmachine: 'perfiphone12mini'
iOSLlvmBuild: False
+ iOSStripSymbols: False
+
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
+ buildConfig: release
+ runtimeFlavor: mono
+ platforms:
+ - osx_x64
+ jobParameters:
+ testGroup: perf
+ runtimeType: iOSMono
+ projectFile: ios_scenarios.proj
+ runKind: ios_scenarios
+ runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
+ logicalmachine: 'perfiphone12mini'
+ iOSLlvmBuild: False
+ iOSStripSymbols: True
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
@@ -159,6 +198,57 @@ jobs:
runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
logicalmachine: 'perfiphone12mini'
iOSLlvmBuild: True
+ iOSStripSymbols: False
+
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
+ buildConfig: release
+ runtimeFlavor: mono
+ platforms:
+ - osx_x64
+ jobParameters:
+ testGroup: perf
+ runtimeType: iOSMono
+ projectFile: ios_scenarios.proj
+ runKind: ios_scenarios
+ runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
+ logicalmachine: 'perfiphone12mini'
+ iOSLlvmBuild: True
+ iOSStripSymbols: True
+
+ # run NativeAOT iOS scenarios
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
+ buildConfig: release
+ runtimeFlavor: mono
+ platforms:
+ - osx_x64
+ jobParameters:
+ testGroup: perf
+ runtimeType: iOSNativeAOT
+ projectFile: ios_scenarios.proj
+ runKind: ios_scenarios
+ runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
+ logicalmachine: 'perfiphone12mini'
+ iOSStripSymbols: False
+
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
+ buildConfig: release
+ runtimeFlavor: mono
+ platforms:
+ - osx_x64
+ jobParameters:
+ testGroup: perf
+ runtimeType: iOSNativeAOT
+ projectFile: ios_scenarios.proj
+ runKind: ios_scenarios
+ runJobTemplate: /eng/pipelines/coreclr/templates/run-scenarios-job.yml
+ logicalmachine: 'perfiphone12mini'
+ iOSStripSymbols: True
# run mono microbenchmarks perf job
- template: /eng/pipelines/common/platform-matrix.yml
@@ -232,7 +322,7 @@ jobs:
runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
logicalmachine: 'perftiger'
- # run coreclr perftiger microbenchmarks pgo perf jobs
+ # run coreclr perftiger microbenchmarks no dynamic pgo perf jobs
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
@@ -247,23 +337,7 @@ jobs:
runKind: micro
runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
logicalmachine: 'perftiger'
- pgoRunType: -NoPgo
-
- - template: /eng/pipelines/common/platform-matrix.yml
- parameters:
- jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
- buildConfig: release
- runtimeFlavor: coreclr
- platforms:
- - windows_x64
- jobParameters:
- testGroup: perf
- liveLibrariesBuildConfig: Release
- projectFile: microbenchmarks.proj
- runKind: micro
- runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
- logicalmachine: 'perftiger'
- pgoRunType: -DynamicPgo
+ pgoRunType: -NoDynamicPGO
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
@@ -279,23 +353,7 @@ jobs:
runKind: micro
runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
logicalmachine: 'perftiger'
- pgoRunType: --dynamicpgo
-
- - template: /eng/pipelines/common/platform-matrix.yml
- parameters:
- jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
- buildConfig: release
- runtimeFlavor: coreclr
- platforms:
- - windows_x64
- jobParameters:
- testGroup: perf
- liveLibrariesBuildConfig: Release
- projectFile: microbenchmarks.proj
- runKind: micro
- runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
- logicalmachine: 'perftiger'
- pgoRunType: -FullPgo
+ pgoRunType: --nodynamicpgo
# run coreclr perfowl microbenchmarks perf job
- template: /eng/pipelines/common/platform-matrix.yml
diff --git a/eng/pipelines/coreclr/perf-wasm-prepare-artifacts-steps.yml b/eng/pipelines/coreclr/perf-wasm-prepare-artifacts-steps.yml
index 0502a534d36cef..3015f0d6cb1413 100644
--- a/eng/pipelines/coreclr/perf-wasm-prepare-artifacts-steps.yml
+++ b/eng/pipelines/coreclr/perf-wasm-prepare-artifacts-steps.yml
@@ -12,10 +12,11 @@ steps:
displayName: "Install workload using artifacts"
- script: >-
- mkdir -p $(Build.SourcesDirectory)/artifacts/staging &&
+ mkdir -p $(Build.SourcesDirectory)/artifacts/staging/built-nugets &&
cp -r $(Build.SourcesDirectory)/artifacts/bin/dotnet-latest $(Build.SourcesDirectory)/artifacts/staging &&
cp -r $(Build.SourcesDirectory)/artifacts/bin/microsoft.netcore.app.runtime.browser-wasm $(Build.SourcesDirectory)/artifacts/staging &&
- cp -r $(Build.SourcesDirectory)/artifacts/bin/microsoft.netcore.app.ref $(Build.SourcesDirectory)/artifacts/staging
+ cp -r $(Build.SourcesDirectory)/artifacts/bin/microsoft.netcore.app.ref $(Build.SourcesDirectory)/artifacts/staging &&
+ cp -r $(Build.SourcesDirectory)/artifacts/packages/${{ parameters.configForBuild }}/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack* $(Build.SourcesDirectory)/artifacts/staging/built-nugets
displayName: "Prepare artifacts staging directory"
- template: /eng/pipelines/common/upload-artifact-step.yml
diff --git a/eng/pipelines/coreclr/perf_slow.yml b/eng/pipelines/coreclr/perf_slow.yml
index 1a12f56abeaa2a..ceb84962217ecb 100644
--- a/eng/pipelines/coreclr/perf_slow.yml
+++ b/eng/pipelines/coreclr/perf_slow.yml
@@ -189,7 +189,7 @@ extends:
logicalmachine: 'perfampere'
timeoutInMinutes: 780
- # run coreclr Linux arm64 ampere microbenchmarks perf job
+ # run coreclr Linux arm64 ampere no dynamic pgo microbenchmarks perf job
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
@@ -205,7 +205,7 @@ extends:
runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
logicalmachine: 'perfampere'
timeoutInMinutes: 780
- pgoRunType: --dynamicpgo
+ pgoRunType: --nodynamicpgo
# run coreclr Windows arm64 microbenchmarks perf job
- template: /eng/pipelines/common/platform-matrix.yml
@@ -240,7 +240,7 @@ extends:
logicalmachine: 'perfampere'
timeoutInMinutes: 500
- # run coreclr Windows arm64 ampere dynamic Pgo microbenchmarks perf job
+ # run coreclr Windows arm64 ampere no dynamic pgo microbenchmarks perf job
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml
@@ -255,7 +255,7 @@ extends:
runKind: micro
runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
logicalmachine: 'perfampere'
- pgoRunType: -DynamicPgo
+ pgoRunType: -NoDynamicPGO
timeoutInMinutes: 500
# run coreclr cloudvm microbenchmarks perf job
diff --git a/eng/pipelines/coreclr/superpmi-collect.yml b/eng/pipelines/coreclr/superpmi-collect.yml
index 453a4a2e1a8db3..4c66b20aecbc76 100644
--- a/eng/pipelines/coreclr/superpmi-collect.yml
+++ b/eng/pipelines/coreclr/superpmi-collect.yml
@@ -133,6 +133,26 @@ extends:
collectionType: crossgen2
collectionName: libraries
+ - template: /eng/pipelines/common/platform-matrix.yml
+ parameters:
+ jobTemplate: /eng/pipelines/coreclr/templates/superpmi-collect-job.yml
+ buildConfig: checked
+ platforms:
+ - osx_arm64
+ - linux_arm
+ - linux_arm64
+ - linux_x64
+ - windows_x64
+ - windows_x86
+ - windows_arm64
+ helixQueueGroup: ci
+ helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
+ jobParameters:
+ testGroup: outerloop
+ liveLibrariesBuildConfig: Release
+ collectionType: run
+ collectionName: realworld
+
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/coreclr/templates/superpmi-collect-job.yml
diff --git a/eng/pipelines/coreclr/templates/build-perf-sample-apps.yml b/eng/pipelines/coreclr/templates/build-perf-sample-apps.yml
index bdc0475fb5da84..89604f865c4ae8 100644
--- a/eng/pipelines/coreclr/templates/build-perf-sample-apps.yml
+++ b/eng/pipelines/coreclr/templates/build-perf-sample-apps.yml
@@ -35,13 +35,13 @@ steps:
- script: rm -r -f $(Build.SourcesDirectory)/artifacts/bin/AndroidSampleApp
workingDirectory: $(Build.SourcesDirectory)/artifacts/bin
displayName: clean bindir
-
- - ${{ if eq(parameters.osGroup, 'ios') }}:
- - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=False DEPLOY_AND_RUN=false
+
+ - ${{ if and(eq(parameters.osGroup, 'ios'), eq(parameters.nameSuffix, 'iOSMono')) }}:
+ - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=False DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=false
env:
DevTeamProvisioning: '-'
workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
- displayName: Build HelloiOS AOT sample app LLVM=False
+ displayName: Build HelloiOS AOT sample app LLVM=False STRIP_SYMBOLS=False
- task: PublishBuildArtifacts@1
condition: succeededOrFailed()
displayName: 'Publish binlog'
@@ -53,17 +53,39 @@ steps:
rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS/bin/ios-arm64/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
includeRootFolder: true
displayName: iOS Sample App NoLLVM
- artifactName: iOSSampleAppNoLLVM
+ artifactName: iOSSampleAppNoLLVMSymbols
archiveExtension: '.zip'
archiveType: zip
- script: rm -r -f $(Build.SourcesDirectory)/src/mono/sample/iOS/bin
workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
displayName: Clean bindir
- - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=True DEPLOY_AND_RUN=false
+ - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=False DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=true
env:
DevTeamProvisioning: '-'
workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
- displayName: Build HelloiOS AOT sample app LLVM=True
+ displayName: Build HelloiOS AOT sample app LLVM=False STRIP_SYMBOLS=True
+ - task: PublishBuildArtifacts@1
+ condition: succeededOrFailed()
+ displayName: 'Publish binlog'
+ inputs:
+ pathtoPublish: $(Build.SourcesDirectory)/src/mono/sample/iOS/msbuild.binlog
+ artifactName: ${{ parameters.artifactName }}
+ - template: /eng/pipelines/common/upload-artifact-step.yml
+ parameters:
+ rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS/bin/ios-arm64/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
+ includeRootFolder: true
+ displayName: iOS Sample App NoLLVM NoSymbols
+ artifactName: iOSSampleAppNoLLVMNoSymbols
+ archiveExtension: '.zip'
+ archiveType: zip
+ - script: rm -r -f $(Build.SourcesDirectory)/src/mono/sample/iOS/bin
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
+ displayName: Clean bindir
+ - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=True DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=false
+ env:
+ DevTeamProvisioning: '-'
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
+ displayName: Build HelloiOS AOT sample app LLVM=True STRIP_SYMBOLS=False
- task: PublishBuildArtifacts@1
condition: succeededOrFailed()
displayName: 'Publish binlog'
@@ -75,6 +97,71 @@ steps:
rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS/bin/ios-arm64/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
includeRootFolder: true
displayName: iOS Sample App LLVM
- artifactName: iOSSampleAppLLVM
+ artifactName: iOSSampleAppLLVMSymbols
+ archiveExtension: '.zip'
+ archiveType: zip
+ - script: rm -r -f $(Build.SourcesDirectory)/src/mono/sample/iOS/bin
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
+ displayName: Clean bindir
+ - script: make build-appbundle TARGET=ios MONO_ARCH=arm64 MONO_CONFIG=Release AOT=True USE_LLVM=True DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=true
+ env:
+ DevTeamProvisioning: '-'
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS
+ displayName: Build HelloiOS AOT sample app LLVM=True STRIP_SYMBOLS=True
+ - task: PublishBuildArtifacts@1
+ condition: succeededOrFailed()
+ displayName: 'Publish binlog'
+ inputs:
+ pathtoPublish: $(Build.SourcesDirectory)/src/mono/sample/iOS/msbuild.binlog
+ artifactName: ${{ parameters.artifactName }}
+ - template: /eng/pipelines/common/upload-artifact-step.yml
+ parameters:
+ rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS/bin/ios-arm64/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
+ includeRootFolder: true
+ displayName: iOS Sample App LLVM NoSymbols
+ artifactName: iOSSampleAppLLVMNoSymbols
archiveExtension: '.zip'
archiveType: zip
+
+ - ${{ if and(eq(parameters.osGroup, 'ios'), eq(parameters.nameSuffix, 'iOSNativeAOT')) }}:
+ - script: make world TARGET_OS=ios TARGET_ARCH=arm64 BUILD_CONFIG=Release DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=false
+ env:
+ DevTeamProvisioning: '-'
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT
+ displayName: Build HelloiOS Native AOT sample app STRIP_SYMBOLS=False
+ - task: PublishBuildArtifacts@1
+ condition: succeededOrFailed()
+ displayName: 'Publish binlog'
+ inputs:
+ pathtoPublish: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT/msbuild.binlog
+ artifactName: ${{ parameters.artifactName }}
+ - template: /eng/pipelines/common/upload-artifact-step.yml
+ parameters:
+ rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT/bin/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
+ includeRootFolder: true
+ displayName: iOS Sample App Symbols
+ artifactName: iOSSampleAppSymbols
+ archiveExtension: '.zip'
+ archiveType: zip
+ - script: rm -r -f $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT/bin
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT
+ displayName: Clean bindir
+ - script: make world TARGET_OS=ios TARGET_ARCH=arm64 BUILD_CONFIG=Release DEPLOY_AND_RUN=false STRIP_DEBUG_SYMBOLS=true
+ env:
+ DevTeamProvisioning: '-'
+ workingDirectory: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT
+ displayName: Build HelloiOS Native AOT sample app STRIP_SYMBOLS=True
+ - task: PublishBuildArtifacts@1
+ condition: succeededOrFailed()
+ displayName: 'Publish binlog'
+ inputs:
+ pathtoPublish: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT/msbuild.binlog
+ artifactName: ${{ parameters.artifactName }}
+ - template: /eng/pipelines/common/upload-artifact-step.yml
+ parameters:
+ rootFolder: $(Build.SourcesDirectory)/src/mono/sample/iOS-NativeAOT/bin/publish/app/HelloiOS/Release-iphoneos/HelloiOS.app
+ includeRootFolder: true
+ displayName: iOS Sample App NoSymbols
+ artifactName: iOSSampleAppNoSymbols
+ archiveExtension: '.zip'
+ archiveType: zip
\ No newline at end of file
diff --git a/eng/pipelines/coreclr/templates/perf-job.yml b/eng/pipelines/coreclr/templates/perf-job.yml
index e61e6761a89d7f..56766c72187365 100644
--- a/eng/pipelines/coreclr/templates/perf-job.yml
+++ b/eng/pipelines/coreclr/templates/perf-job.yml
@@ -19,6 +19,7 @@ parameters:
pgoRunType: ''
javascriptEngine: 'NoJS'
iOSLlvmBuild: 'False'
+ iOSStripSymbols: 'False'
skipLiveLibrariesDownload: false
collectHelixLogsScript: ''
timeoutInMinutes: 320
@@ -37,8 +38,8 @@ jobs:
- template: ${{ parameters.runJobTemplate }}
parameters:
# Compute job name from template parameters
- jobName: ${{ format('perfbuild_{0}{1}_{2}_{3}_{4}_{5}_{6}_{7}_{8}_{9}_{10}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind, parameters.logicalMachine, parameters.javascriptEngine, parameters.pgoRunType, parameters.iosLlvmBuild) }}
- displayName: ${{ format('Performance {0}{1} {2} {3} {4} {5} {6} {7} {8} {9} {10}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind, parameters.logicalMachine, parameters.javascriptEngine, parameters.pgoRunType, parameters.iosLlvmBuild) }}
+ jobName: ${{ format('perfbuild_{0}{1}_{2}_{3}_{4}_{5}_{6}_{7}_{8}_{9}_{10}_{11}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind, parameters.logicalMachine, parameters.javascriptEngine, parameters.pgoRunType, parameters.iosLlvmBuild, parameters.iosStripSymbols) }}
+ displayName: ${{ format('Performance {0}{1} {2} {3} {4} {5} {6} {7} {8} {9} {10} {11}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind, parameters.logicalMachine, parameters.javascriptEngine, parameters.pgoRunType, parameters.iosLlvmBuild, parameters.iosStripSymbols) }}
pool: ${{ parameters.pool }}
buildConfig: ${{ parameters.buildConfig }}
archType: ${{ parameters.archType }}
@@ -56,6 +57,7 @@ jobs:
pgoRunType: ${{ parameters.pgoRunType }}
javascriptEngine: ${{ parameters.javascriptEngine }}
iosLlvmBuild: ${{ parameters.iosLlvmBuild }}
+ iosStripSymbols: ${{ parameters.iosStripSymbols }}
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
${{ if and(eq(parameters.runtimeType, 'wasm'), eq(parameters.codeGenType, 'aot')) }}:
@@ -68,7 +70,7 @@ jobs:
# Test job depends on the corresponding build job
${{ if eq(parameters.downloadSpecificBuild.buildId, '') }}:
dependsOn:
- - ${{ if not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'wasm')) }}:
+ - ${{ if not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'iOSNativeAOT', 'wasm')) }}:
- ${{ format('coreclr_{0}_product_build_{1}{2}_{3}_{4}', parameters.runtimeVariant, parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }}
- ${{ if and(ne(parameters.liveLibrariesBuildConfig, ''), eq(parameters.skipLiveLibrariesDownload, 'false')) }}:
- ${{ format('libraries_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }}
@@ -83,13 +85,15 @@ jobs:
- ${{ 'Build_ios_arm64_release_PerfBDNApp' }}
- ${{ if eq(parameters.runtimeType, 'iOSMono')}}:
- ${{ 'build_ios_arm64_release_iOSMono' }}
+ - ${{ if eq(parameters.runtimeType, 'iOSNativeAOT')}}:
+ - ${{ 'build_ios_arm64_release_iOSNativeAOT' }}
- ${{ if and(eq(parameters.osGroup, 'windows'), not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono'))) }}:
+ ${{ if and(eq(parameters.osGroup, 'windows'), not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'iOSNativeAOT'))) }}:
${{ if eq(parameters.runtimeType, 'mono') }}:
extraSetupParameters: -Architecture ${{ parameters.archType }} -MonoDotnet $(Build.SourcesDirectory)\.dotnet-mono
${{ if eq(parameters.runtimeType, 'coreclr') }}:
extraSetupParameters: -CoreRootDirectory $(Build.SourcesDirectory)\artifacts\tests\coreclr\${{ parameters.osGroup }}.${{ parameters.archType }}.Release\Tests\Core_Root -Architecture ${{ parameters.archType }}
- ${{ if and(ne(parameters.osGroup, 'windows'), not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono'))) }}:
+ ${{ if and(ne(parameters.osGroup, 'windows'), not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'iOSNativeAOT'))) }}:
${{ if and(eq(parameters.runtimeType, 'mono'), ne(parameters.codeGenType, 'AOT')) }}:
extraSetupParameters: --architecture ${{ parameters.archType }} --monodotnet $(Build.SourcesDirectory)/.dotnet-mono
${{ if and(eq(parameters.runtimeType, 'wasm'), ne(parameters.codeGenType, 'AOT')) }}:
@@ -104,8 +108,10 @@ jobs:
extraSetupParameters: --corerootdirectory $(Build.SourcesDirectory)/artifacts/tests/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.Release/Tests/Core_Root --architecture ${{ parameters.archType }} --alpine
${{ if in(parameters.runtimeType, 'AndroidMono') }}:
extraSetupParameters: -Architecture ${{ parameters.archType }} -AndroidMono
- ${{ if in(parameters.runtimeType, 'iosMono') }}:
- extraSetupParameters: --architecture ${{ parameters.archType }} --iosmono --iosllvmbuild ${{ parameters.iOSLlvmBuild }}
+ ${{ if in(parameters.runtimeType, 'iOSMono') }}:
+ extraSetupParameters: --architecture ${{ parameters.archType }} --iosmono --iosllvmbuild ${{ parameters.iOSLlvmBuild }} --iosstripsymbols ${{ parameters.iOSStripSymbols }}
+ ${{ if in(parameters.runtimeType, 'iOSNativeAOT') }}:
+ extraSetupParameters: --architecture ${{ parameters.archType }} --iosnativeaot --iosllvmbuild ${{ parameters.iOSLlvmBuild }} --iosstripsymbols ${{ parameters.iOSStripSymbols }}
variables:
- ${{ each variable in parameters.variables }}:
@@ -147,7 +153,7 @@ jobs:
displayName: 'live-built libraries'
# Download coreclr
- - ${{ if not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'wasm')) }}:
+ - ${{ if not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono', 'iOSNativeAOT', 'wasm')) }}:
- template: /eng/pipelines/common/download-artifact-step.yml
parameters:
unpackFolder: $(buildProductRootFolderPath)
@@ -188,6 +194,7 @@ jobs:
mkdir -p $(librariesDownloadDir)/bin/wasm/wasm-data &&
mkdir -p $(librariesDownloadDir)/bin/wasm/dotnet &&
cp -r $(librariesDownloadDir)/BrowserWasm/staging/dotnet-latest/* $(librariesDownloadDir)/bin/wasm/dotnet &&
+ cp -r $(librariesDownloadDir)/BrowserWasm/staging/built-nugets $(librariesDownloadDir)/bin/wasm &&
cp src/mono/wasm/Wasm.Build.Tests/data/test-main-7.0.js $(librariesDownloadDir)/bin/wasm/wasm-data/test-main.js &&
find $(librariesDownloadDir)/bin/wasm -type d &&
find $(librariesDownloadDir)/bin/wasm -type f -exec chmod 664 {} \;
@@ -222,39 +229,50 @@ jobs:
artifactName: 'AndroidBDNApk'
displayName: 'Mono Android BDN Apk'
- # Download iOSMono tests
- - ${{ if eq(parameters.runtimeType, 'iOSMono') }}:
+ # Download iOSMono and Native AOT tests
+ - ${{ if or(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.runtimeType, 'iOSNativeAOT')) }}:
- template: /eng/pipelines/common/download-artifact-step.yml
parameters:
- unpackFolder: $(Build.SourcesDirectory)/iosHelloWorld/nollvm
+ unpackFolder: $(Build.SourcesDirectory)/iosHelloWorld
cleanUnpackFolder: false
- artifactFileName: 'iOSSampleAppNoLLVM.zip'
- artifactName: 'iOSSampleAppNoLLVM'
- displayName: 'iOS Sample App NoLLVM'
- - template: /eng/pipelines/common/download-artifact-step.yml
- parameters:
- unpackFolder: $(Build.SourcesDirectory)/iosHelloWorld/llvm
- cleanUnpackFolder: false
- artifactFileName: 'iOSSampleAppLLVM.zip'
- artifactName: 'iOSSampleAppLLVM'
- displayName: 'iOS Sample App LLVM'
- # same artifact as above but don't extract .zip
- - task: DownloadBuildArtifacts@0
- displayName: 'Download iOS Sample App NoLLVM'
- inputs:
- buildType: current
- downloadType: single
- downloadPath: '$(Build.SourcesDirectory)/iosHelloWorldZip/nollvmzip'
- artifactName: 'iOSSampleAppNoLLVM'
- checkDownloadedFiles: true
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'False'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppNoLLVMSymbols'
+ artifactFileName: 'iOSSampleAppNoLLVMSymbols.zip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'False'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppNoLLVMNoSymbols'
+ artifactFileName: 'iOSSampleAppNoLLVMNoSymbols.zip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'True'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppLLVMSymbols'
+ artifactFileName: 'iOSSampleAppLLVMSymbols.zip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'True'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppLLVMNoSymbols'
+ artifactFileName: 'iOSSampleAppLLVMNoSymbols.zip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSNativeAOT'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppSymbols'
+ artifactFileName: 'iOSSampleAppSymbols.zip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSNativeAOT'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppNoSymbols'
+ artifactFileName: 'iOSSampleAppNoSymbols.zip'
+ displayName: 'iOS Sample App'
# same artifact as above but don't extract .zip
- task: DownloadBuildArtifacts@0
- displayName: 'Download iOS Sample App LLVM'
+ displayName: 'Download iOS Sample App'
inputs:
buildType: current
downloadType: single
- downloadPath: '$(Build.SourcesDirectory)/iosHelloWorldZip/llvmzip'
- artifactName: 'iOSSampleAppLLVM'
+ downloadPath: '$(Build.SourcesDirectory)/iosHelloWorldZip'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'False'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppNoLLVMSymbols'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'False'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppNoLLVMNoSymbols'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'True'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppLLVMSymbols'
+ ${{ if and(eq(parameters.runtimeType, 'iOSMono'), eq(parameters.iOSLlvmBuild, 'True'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppLLVMNoSymbols'
+ ${{ if and(eq(parameters.runtimeType, 'iOSNativeAOT'), eq(parameters.iOSStripSymbols, 'False')) }}:
+ artifactName: 'iOSSampleAppSymbols'
+ ${{ if and(eq(parameters.runtimeType, 'iOSNativeAOT'), eq(parameters.iOSStripSymbols, 'True')) }}:
+ artifactName: 'iOSSampleAppNoSymbols'
checkDownloadedFiles: true
# Create Core_Root
@@ -265,8 +283,8 @@ jobs:
# Copy the runtime directory into the testhost folder to include OOBs.
- script: "build.cmd -subset libs.pretest -configuration release -ci -arch $(archType) -testscope innerloop /p:RuntimeArtifactsPath=$(librariesDownloadDir)\\bin\\mono\\$(osGroup).$(archType).$(buildConfigUpper) /p:RuntimeFlavor=mono;xcopy $(Build.SourcesDirectory)\\artifacts\\bin\\runtime\\net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)\\* $(Build.SourcesDirectory)\\artifacts\\bin\\testhost\\net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)\\shared\\Microsoft.NETCore.App\\$(productVersion) /E /I /Y;xcopy $(Build.SourcesDirectory)\\artifacts\\bin\\testhost\\net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)\\* $(Build.SourcesDirectory)\\.dotnet-mono /E /I /Y;copy $(Build.SourcesDirectory)\\artifacts\\bin\\coreclr\\$(osGroup).$(archType).$(buildConfigUpper)\\corerun.exe $(Build.SourcesDirectory)\\.dotnet-mono\\shared\\Microsoft.NETCore.App\\$(productVersion)\\corerun.exe"
displayName: "Create mono dotnet (Windows)"
- condition: and(and(succeeded(), eq(variables.runtimeFlavorName, 'Mono')), eq(variables.osGroup, 'windows'), not(in('${{ parameters.runtimeType }}', 'AndroidMono', 'iOSMono')))
+ condition: and(and(succeeded(), eq(variables.runtimeFlavorName, 'Mono')), eq(variables.osGroup, 'windows'), not(in('${{ parameters.runtimeType }}', 'AndroidMono', 'iOSMono', 'iOSNativeAOT')))
- script: "mkdir $(Build.SourcesDirectory)/.dotnet-mono;./build.sh -subset libs.pretest -configuration release -ci -arch $(archType) -testscope innerloop /p:RuntimeArtifactsPath=$(librariesDownloadDir)/bin/mono/$(osGroup).$(archType).$(buildConfigUpper) /p:RuntimeFlavor=mono;cp $(Build.SourcesDirectory)/artifacts/bin/runtime/net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)/* $(Build.SourcesDirectory)/artifacts/bin/testhost/net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)/shared/Microsoft.NETCore.App/$(productVersion) -rf;cp $(Build.SourcesDirectory)/artifacts/bin/testhost/net8.0-$(osGroup)-$(buildConfigUpper)-$(archType)/* $(Build.SourcesDirectory)/.dotnet-mono -r;cp $(Build.SourcesDirectory)/artifacts/bin/coreclr/$(osGroup).$(archType).$(buildConfigUpper)/corerun $(Build.SourcesDirectory)/.dotnet-mono/shared/Microsoft.NETCore.App/$(productVersion)/corerun"
displayName: "Create mono dotnet (Linux)"
- condition: and(and(succeeded(), eq(variables.runtimeFlavorName, 'Mono')), ne(variables.osGroup, 'windows'), not(in('${{ parameters.runtimeType }}', 'AndroidMono', 'iOSMono')))
+ condition: and(and(succeeded(), eq(variables.runtimeFlavorName, 'Mono')), ne(variables.osGroup, 'windows'), not(in('${{ parameters.runtimeType }}', 'AndroidMono', 'iOSMono', 'iOSNativeAOT')))
diff --git a/eng/pipelines/coreclr/templates/run-scenarios-job.yml b/eng/pipelines/coreclr/templates/run-scenarios-job.yml
index c8267446e516d8..ba6875b6a7fec3 100644
--- a/eng/pipelines/coreclr/templates/run-scenarios-job.yml
+++ b/eng/pipelines/coreclr/templates/run-scenarios-job.yml
@@ -142,6 +142,7 @@ jobs:
- script: >-
mkdir -p $(librariesDownloadDir)/bin/wasm/data &&
cp -r $(librariesDownloadDir)/BrowserWasm/staging/dotnet-latest $(librariesDownloadDir)/bin/wasm &&
+ cp -r $(librariesDownloadDir)/BrowserWasm/staging/built-nugets $(librariesDownloadDir)/bin/wasm &&
cp src/mono/wasm/Wasm.Build.Tests/data/test-main-7.0.js $(librariesDownloadDir)/bin/wasm/data/test-main.js &&
find $(librariesDownloadDir)/bin/wasm -type f -exec chmod 664 {} \;
displayName: "Create wasm directory (Linux)"
@@ -215,6 +216,6 @@ jobs:
displayName: Publish Logs
inputs:
targetPath: $(Build.SourcesDirectory)/artifacts/log
- artifactName: 'Performance_Run_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)_${{ parameters.runtimeType }}_${{ parameters.codeGenType }}_${{ parameters.runKind }}_$(iOSLlvmBuild)'
+ artifactName: 'Performance_Run_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)_${{ parameters.runtimeType }}_${{ parameters.codeGenType }}_${{ parameters.runKind }}_$(iOSLlvmBuild)_$(iOSStripSymbols)'
continueOnError: true
condition: always()
diff --git a/eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml b/eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml
index ef55089a230467..ab401a83f6959f 100644
--- a/eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml
+++ b/eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml
@@ -91,6 +91,9 @@ jobs:
- ${{ if eq(parameters.collectionName, 'benchmarks') }}:
- name: InputDirectory
value: '$(Core_Root_Dir)'
+ - ${{ if eq(parameters.collectionName, 'realworld') }}:
+ - name: InputDirectory
+ value: '$(Core_Root_Dir)'
- ${{ if eq(parameters.collectionName, 'coreclr_tests') }}:
- name: InputDirectory
value: '$(managedTestArtifactRootFolderPath)'
diff --git a/eng/pipelines/runtime-android-grpc-client-tests.yml b/eng/pipelines/runtime-android-grpc-client-tests.yml
index 2c52bbf136c78d..00e51f766d95ce 100644
--- a/eng/pipelines/runtime-android-grpc-client-tests.yml
+++ b/eng/pipelines/runtime-android-grpc-client-tests.yml
@@ -36,7 +36,7 @@ extends:
buildConfig: Release
runtimeFlavor: mono
platforms:
- - android_x64
+ - android_x64_docker
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono_gRPC
diff --git a/eng/pipelines/runtime-codeql.yml b/eng/pipelines/runtime-codeql.yml
index 62c002926b70da..34fa555d180fe6 100644
--- a/eng/pipelines/runtime-codeql.yml
+++ b/eng/pipelines/runtime-codeql.yml
@@ -2,8 +2,13 @@ trigger:
none
schedules:
- - cron: 0 12 * * 1
- displayName: Weekly Monday CodeQL/Semmle run
+ - cron: 0 12 * * 1,4,6
+ # CodeQL auto-injects into other pipelines and will do a scan if one hasn't
+ # been done in 72 hours. These can cause timeouts, so we ensure that one has
+ # already been performed. We can increase this to 4x/week or be more
+ # sophisticated if the 72 hour period ending Thursday ("4" in the cron line)
+ # causes a problem.
+ displayName: 3x/week CodeQL/Semmle run
branches:
include:
- main
diff --git a/eng/pipelines/runtime.yml b/eng/pipelines/runtime.yml
index 23fbbff0770c5c..4b182fb0e7be2a 100644
--- a/eng/pipelines/runtime.yml
+++ b/eng/pipelines/runtime.yml
@@ -248,7 +248,7 @@ extends:
extraStepsTemplate: /eng/pipelines/coreclr/nativeaot-post-build-steps.yml
extraStepsParameters:
creator: dotnet-bot
- testBuildArgs: 'nativeaot tree ";nativeaot;Loader;Interop;tracing/eventpipe/config;tracing/eventpipe/simpleprovidervalidation;" test tracing/eventcounter/runtimecounters.csproj /p:BuildNativeAotFrameworkObjects=true'
+ testBuildArgs: 'nativeaot tree ";nativeaot;Loader;Interop;tracing/eventpipe/config;tracing/eventpipe/diagnosticport;tracing/eventpipe/reverse;" test tracing/eventcounter/runtimecounters.csproj /p:BuildNativeAotFrameworkObjects=true'
liveLibrariesBuildConfig: Release
testRunNamePrefixSuffix: NativeAOT_$(_BuildConfig)
extraVariablesTemplates:
@@ -290,7 +290,7 @@ extends:
extraStepsTemplate: /eng/pipelines/coreclr/nativeaot-post-build-steps.yml
extraStepsParameters:
creator: dotnet-bot
- testBuildArgs: nativeaot tree nativeaot
+ testBuildArgs: 'nativeaot tree ";nativeaot;tracing/eventpipe/simpleprovidervalidation;"'
liveLibrariesBuildConfig: Release
testRunNamePrefixSuffix: NativeAOT_$(_BuildConfig)
extraVariablesTemplates:
diff --git a/eng/resolveContract.targets b/eng/resolveContract.targets
index 2dc517aaa96171..6d414f46f93e6b 100644
--- a/eng/resolveContract.targets
+++ b/eng/resolveContract.targets
@@ -1,4 +1,5 @@
+
+
+ true
+ true
+
+
false
diff --git a/eng/targetingpacks.targets b/eng/targetingpacks.targets
index ebdebf46c8b678..14ceef19b704da 100644
--- a/eng/targetingpacks.targets
+++ b/eng/targetingpacks.targets
@@ -35,7 +35,7 @@
LatestRuntimeFrameworkVersion="$(ProductVersion)"
RuntimeFrameworkName="$(LocalFrameworkOverrideName)"
RuntimePackNamePatterns="$(LocalFrameworkOverrideName).Runtime.**RID**"
- RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
+ RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
TargetFramework="$(NetCoreAppCurrent)"
TargetingPackName="$(LocalFrameworkOverrideName).Ref"
TargetingPackVersion="$(ProductVersion)"
@@ -45,20 +45,20 @@
RuntimeFrameworkName="$(LocalFrameworkOverrideName)"
LatestRuntimeFrameworkVersion="$(ProductVersion)"
RuntimePackNamePatterns="$(LocalFrameworkOverrideName).Runtime.Mono.**RID**"
- RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86;browser-wasm;wasi-wasm;ios-arm64;iossimulator-arm64;iossimulator-x64;tvos-arm64;tvossimulator-arm64;tvossimulator-x64;android-arm64;android-arm;android-x64;android-x86"
+ RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86;browser-wasm;wasi-wasm;ios-arm64;iossimulator-arm64;iossimulator-x64;tvos-arm64;tvossimulator-arm64;tvossimulator-x64;android-arm64;android-arm;android-x64;android-x86"
RuntimePackLabels="Mono"
Condition="'@(KnownRuntimePack)' == '' or !@(KnownRuntimePack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))"/>
@@ -67,7 +67,7 @@
ExcludedRuntimeIdentifiers="android"
AppHostPackNamePattern="$(LocalFrameworkOverrideName).Host.**RID**"
AppHostPackVersion="$(ProductVersion)"
- AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
+ AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
TargetFramework="$(NetCoreAppCurrent)"
Condition="'@(KnownAppHostPack)' == '' or !@(KnownAppHostPack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))" />
diff --git a/eng/testing/ProvisioningVersions.props b/eng/testing/ProvisioningVersions.props
index 4f2df8f136f7d4..7f555c06db3bca 100644
--- a/eng/testing/ProvisioningVersions.props
+++ b/eng/testing/ProvisioningVersions.props
@@ -44,7 +44,7 @@
- true
+ false
diff --git a/eng/testing/performance/blazor_perf.proj b/eng/testing/performance/blazor_perf.proj
index 4c6881dfbb05e4..a526a8cc3b5de3 100644
--- a/eng/testing/performance/blazor_perf.proj
+++ b/eng/testing/performance/blazor_perf.proj
@@ -3,11 +3,14 @@
%HELIX_WORKITEM_UPLOAD_ROOT%\%24{HELIX_WORKITEM_UPLOAD_ROOT}/
+ set RestoreAdditionalProjectSources=%HELIX_CORRELATION_PAYLOAD%\built-nugets &
+ export RestoreAdditionalProjectSources=$HELIX_CORRELATION_PAYLOAD/built-nugets &&
+
python3$(HelixPreCommands);chmod +x $HELIX_WORKITEM_PAYLOAD/SOD/SizeOnDisk--has-workload --readonly-dotnet --msbuild "/p:_TrimmerDumpDependencies=true" --msbuild /warnaserror:NU1602,NU1604 --msbuild-static AdditionalMonoLinkerOptions=%27"%24(AdditionalMonoLinkerOptions) --dump-dependencies"%27 --binlog $(LogDirectory)blazor_publish.binlog
- $(Python) pre.py publish $(PublishArgs)
+ $(EnvVars) $(Python) pre.py publish $(PublishArgs)
diff --git a/eng/testing/performance/ios_scenarios.proj b/eng/testing/performance/ios_scenarios.proj
index 6b30417fdff479..3801fd935d96f8 100644
--- a/eng/testing/performance/ios_scenarios.proj
+++ b/eng/testing/performance/ios_scenarios.proj
@@ -17,6 +17,8 @@
nollvmllvm
+ symbols
+ nosymbols
@@ -28,19 +30,19 @@
-
+ $(WorkItemDirectory)
- cd $(ScenarioDirectory)helloios;cp -rf $HELIX_CORRELATION_PAYLOAD/iosHelloWorld/$(LlvmPath) ./app;$(Python) pre.py --name app
+ cd $(ScenarioDirectory)helloios;cp -rf $HELIX_CORRELATION_PAYLOAD/iosHelloWorld ./app;$(Python) pre.py --name app$(Python) test.py sod --scenario-name "%(Identity)"$(Python) post.py
-
+ $(WorkItemDirectory)
- cd $(ScenarioDirectory)helloios;cp -v $HELIX_CORRELATION_PAYLOAD/iosHelloWorldZip/$(LlvmPath)zip/iOSSampleApp$(LlvmPath)/iOSSampleApp$(LlvmPath).zip .;$(Python) pre.py --name iOSSampleApp$(LlvmPath).zip
+ cd $(ScenarioDirectory)helloios;cp -v $HELIX_CORRELATION_PAYLOAD/iosHelloWorldZip/iOSSampleApp.zip .;$(Python) pre.py --name iOSSampleApp.zip$(Python) test.py sod --scenario-name "%(Identity)"$(Python) post.py
-
+ $(WorkItemDirectory).zip00:15:00ios-device
@@ -49,7 +51,7 @@
# PreCommands
export XHARNESSPATH=$XHARNESS_CLI_PATH
- cp -r $HELIX_CORRELATION_PAYLOAD/iosHelloWorld/$(LlvmPath)/HelloiOS.app $(ScenarioDirectory)helloios/HelloiOS.app
+ cp -r $HELIX_CORRELATION_PAYLOAD/iosHelloWorld/iosHelloWorld/HelloiOS.app $(ScenarioDirectory)helloios/HelloiOS.app
cp -f embedded.mobileprovision $(ScenarioDirectory)helloios/HelloiOS.app
cd $(ScenarioDirectory)helloios
sign HelloiOS.app
diff --git a/eng/testing/performance/performance-setup.ps1 b/eng/testing/performance/performance-setup.ps1
index 29e3d6199cb788..a826f39f045e17 100644
--- a/eng/testing/performance/performance-setup.ps1
+++ b/eng/testing/performance/performance-setup.ps1
@@ -22,10 +22,10 @@ Param(
[string] $LogicalMachine="",
[switch] $AndroidMono,
[switch] $iOSMono,
- [switch] $NoPGO,
- [switch] $DynamicPGO,
- [switch] $FullPGO,
+ [switch] $iOSNativeAOT,
+ [switch] $NoDynamicPGO,
[switch] $iOSLlvmBuild,
+ [switch] $iOSStripSymbols,
[string] $MauiVersion,
[switch] $UseLocalCommitTime
)
@@ -83,21 +83,18 @@ if($MonoDotnet -ne "")
}
}
-if($NoPGO)
+if($NoDynamicPGO)
{
- $Configurations += " PGOType=nopgo"
-}
-elseif($DynamicPGO)
-{
- $Configurations += " PGOType=dynamicpgo"
-}
-elseif($FullPGO)
-{
- $Configurations += " PGOType=fullpgo"
+ $Configurations += " PGOType=nodynamicpgo"
}
if ($iOSMono) {
$Configurations += " iOSLlvmBuild=$iOSLlvmBuild"
+ $Configurations += " iOSStripSymbols=$iOSStripSymbols"
+}
+
+if ($iOSNativeAOT) {
+ $Configurations += " iOSStripSymbols=$iOSStripSymbols"
}
# FIX ME: This is a workaround until we get this from the actual pipeline
@@ -109,17 +106,9 @@ if($Branch.Contains("refs/heads/release"))
$CommonSetupArguments="--channel $CleanedBranchName --queue $Queue --build-number $BuildNumber --build-configs $Configurations --architecture $Architecture"
$SetupArguments = "--repository https://github.com/$Repository --branch $Branch --get-perf-hash --commit-sha $CommitSha $CommonSetupArguments"
-if($NoPGO)
-{
- $SetupArguments = "$SetupArguments --no-pgo"
-}
-elseif($DynamicPGO)
-{
- $SetupArguments = "$SetupArguments --dynamic-pgo"
-}
-elseif($FullPGO)
+if($NoDynamicPGO)
{
- $SetupArguments = "$SetupArguments --full-pgo"
+ $SetupArguments = "$SetupArguments --no-dynamic-pgo"
}
if($UseLocalCommitTime)
@@ -196,6 +185,7 @@ Write-PipelineSetVariable -Name 'RunFromPerfRepo' -Value "$RunFromPerformanceRep
Write-PipelineSetVariable -Name 'Compare' -Value "$Compare" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'MonoDotnet' -Value "$UsingMono" -IsMultiJobVariable $false
Write-PipelineSetVariable -Name 'iOSLlvmBuild' -Value "$iOSLlvmBuild" -IsMultiJobVariable $false
+Write-PipelineSetVariable -Name 'iOSStripSymbols' -Value "$iOSStripSymbols" -IsMultiJobVariable $false
# Helix Arguments
Write-PipelineSetVariable -Name 'Creator' -Value "$Creator" -IsMultiJobVariable $false
diff --git a/eng/testing/performance/performance-setup.sh b/eng/testing/performance/performance-setup.sh
index 871a745c3375d5..38fbbc1a980de3 100755
--- a/eng/testing/performance/performance-setup.sh
+++ b/eng/testing/performance/performance-setup.sh
@@ -36,7 +36,10 @@ use_latest_dotnet=false
logical_machine=
javascript_engine="v8"
iosmono=false
+iosnativeaot=false
+runtimetype=""
iosllvmbuild=""
+iosstripsymbols=""
maui_version=""
use_local_commit_time=false
only_sanity=false
@@ -139,16 +142,8 @@ while (($# > 0)); do
wasmaot=true
shift 1
;;
- --nopgo)
- nopgo=true
- shift 1
- ;;
- --dynamicpgo)
- dynamicpgo=true
- shift 1
- ;;
- --fullpgo)
- fullpgo=true
+ --nodynamicpgo)
+ nodynamicpgo=true
shift 1
;;
--compare)
@@ -171,10 +166,18 @@ while (($# > 0)); do
iosmono=true
shift 1
;;
+ --iosnativeaot)
+ iosnativeaot=true
+ shift 1
+ ;;
--iosllvmbuild)
iosllvmbuild=$2
shift 2
;;
+ --iosstripsymbols)
+ iosstripsymbols=$2
+ shift 2
+ ;;
--mauiversion)
maui_version=$2
shift 2
@@ -222,12 +225,12 @@ while (($# > 0)); do
echo " --dotnetversions Passed as '--dotnet-versions ' to the setup script"
echo " --alpine Set for runs on Alpine"
echo " --iosmono Set for ios Mono/Maui runs"
+ echo " --iosnativeaot Set for ios Native AOT runs"
echo " --iosllvmbuild Set LLVM for iOS Mono/Maui runs"
+ echo " --iosstripsymbols Set STRIP_DEBUG_SYMBOLS for iOS Mono/Maui runs"
echo " --mauiversion Set the maui version for Mono/Maui runs"
echo " --uselocalcommittime Pass local runtime commit time to the setup script"
- echo " --nopgo Set for No PGO runs"
- echo " --dynamicpgo Set for dynamic PGO runs"
- echo " --fullpgo Set for Full PGO runs"
+ echo " --nodynamicpgo Set for No dynamic PGO runs"
echo ""
exit 1
;;
@@ -329,21 +332,23 @@ if [[ "$monoaot" == "true" ]]; then
fi
if [[ "$iosmono" == "true" ]]; then
- configurations="$configurations iOSLlvmBuild=$iosllvmbuild"
+ runtimetype="Mono"
+ configurations="$configurations iOSLlvmBuild=$iosllvmbuild iOSStripSymbols=$iosstripsymbols RuntimeType=$runtimetype"
extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments"
fi
-if [[ "$nopgo" == "true" ]]; then
- configurations="$configurations PGOType=nopgo"
-fi
-if [[ "$dynamicpgo" == "true" ]]; then
- configurations="$configurations PGOType=dynamicpgo"
+if [[ "$iosnativeaot" == "true" ]]; then
+ runtimetype="NativeAOT"
+ configurations="$configurations iOSStripSymbols=$iosstripsymbols RuntimeType=$runtimetype"
+ extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments"
fi
-if [[ "$fullpgo" == "true" ]]; then
- configurations="$configurations PGOType=fullpgo"
- extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments --category-exclusion-filter NoAOT"
+
+if [[ "$nodynamicpgo" == "true" ]]; then
+ configurations="$configurations PGOType=nodynamicpgo"
fi
+
+
cleaned_branch_name="main"
if [[ $branch == *"refs/heads/release"* ]]; then
cleaned_branch_name=${branch/refs\/heads\//}
@@ -406,14 +411,8 @@ if [[ -n "$dotnet_versions" ]]; then
setup_arguments="$setup_arguments --dotnet-versions $dotnet_versions"
fi
-if [[ "$nopgo" == "true" ]]; then
- setup_arguments="$setup_arguments --no-pgo"
-fi
-if [[ "$dynamicpgo" == "true" ]]; then
- setup_arguments="$setup_arguments --dynamic-pgo"
-fi
-if [[ "$fullpgo" == "true" ]]; then
- setup_arguments="$setup_arguments --full-pgo"
+if [[ "$nodynamicpgo" == "true" ]]; then
+ setup_arguments="$setup_arguments --no-dynamic-pgo"
fi
if [[ "$monoaot" == "true" ]]; then
@@ -434,16 +433,11 @@ if [[ "$use_baseline_core_run" == true ]]; then
mv $baseline_core_root_directory $new_baseline_core_root
fi
-if [[ "$iosmono" == "true" ]]; then
- if [[ "$iosllvmbuild" == "True" ]]; then
- # LLVM Mono .app
- mkdir -p $payload_directory/iosHelloWorld && cp -rv $source_directory/iosHelloWorld/llvm $payload_directory/iosHelloWorld
- mkdir -p $payload_directory/iosHelloWorldZip/llvmzip && cp -rv $source_directory/iosHelloWorldZip/llvmzip $payload_directory/iosHelloWorldZip
- else
- # NoLLVM Mono .app
- mkdir -p $payload_directory/iosHelloWorld && cp -rv $source_directory/iosHelloWorld/nollvm $payload_directory/iosHelloWorld
- mkdir -p $payload_directory/iosHelloWorldZip/nollvmzip && cp -rv $source_directory/iosHelloWorldZip/nollvmzip $payload_directory/iosHelloWorldZip
- fi
+if [[ "$iosmono" == "true" || "$iosnativeaot" == "true" ]]; then
+ mkdir -p $payload_directory/iosHelloWorld && cp -rv $source_directory/iosHelloWorld $payload_directory/iosHelloWorld
+ mkdir -p $payload_directory/iosHelloWorldZip && cp -rv $source_directory/iosHelloWorldZip $payload_directory/iosHelloWorldZip
+
+ find "$payload_directory/iosHelloWorldZip/" -type f -name "*.zip" -execdir mv {} "$payload_directory/iosHelloWorldZip/iOSSampleApp.zip" \;
fi
ci=true
@@ -477,6 +471,8 @@ Write-PipelineSetVariable -name "Compare" -value "$compare" -is_multi_job_variab
Write-PipelineSetVariable -name "MonoDotnet" -value "$using_mono" -is_multi_job_variable false
Write-PipelineSetVariable -name "WasmDotnet" -value "$using_wasm" -is_multi_job_variable false
Write-PipelineSetVariable -Name 'iOSLlvmBuild' -Value "$iosllvmbuild" -is_multi_job_variable false
+Write-PipelineSetVariable -Name 'iOSStripSymbols' -Value "$iosstripsymbols" -is_multi_job_variable false
+Write-PipelineSetVariable -Name 'RuntimeType' -Value "$runtimetype" -is_multi_job_variable false
Write-PipelineSetVariable -name "OnlySanityCheck" -value "$only_sanity" -is_multi_job_variable false
# Put it back to what was set on top of this script
diff --git a/eng/testing/scenarios/BuildWasmAppsJobsList.txt b/eng/testing/scenarios/BuildWasmAppsJobsList.txt
index 38e288f81cfa14..914e6a2f956ee1 100644
--- a/eng/testing/scenarios/BuildWasmAppsJobsList.txt
+++ b/eng/testing/scenarios/BuildWasmAppsJobsList.txt
@@ -11,6 +11,7 @@ Wasm.Build.Tests.BuildPublishTests
Wasm.Build.Tests.CleanTests
Wasm.Build.Tests.ConfigSrcTests
Wasm.Build.Tests.IcuShardingTests
+Wasm.Build.Tests.HybridGlobalizationTests
Wasm.Build.Tests.InvariantGlobalizationTests
Wasm.Build.Tests.MainWithArgsTests
Wasm.Build.Tests.NativeBuildTests
diff --git a/eng/testing/tests.singlefile.targets b/eng/testing/tests.singlefile.targets
index 4db0bf2f448955..4f4bcb481073ac 100644
--- a/eng/testing/tests.singlefile.targets
+++ b/eng/testing/tests.singlefile.targets
@@ -29,10 +29,10 @@
$(CoreCLRAotSdkDir)$(NetCoreAppCurrentTestHostSharedFrameworkPath)$(NetCoreAppCurrentTestHostSharedFrameworkPath)
- $(NoWarn);IL1005;IL2105;IL3000;IL3001;IL3002;IL3003
+ $(NoWarn);IL1005;IL2105;IL3000;IL3001;IL3002;IL3003;IL3050;IL3051;IL3052;IL3053partialtrue
- true
+ false
diff --git a/global.json b/global.json
index d24eda92f8b044..778d73c6fe7963 100644
--- a/global.json
+++ b/global.json
@@ -8,9 +8,9 @@
"dotnet": "8.0.100-preview.4.23260.5"
},
"msbuild-sdks": {
- "Microsoft.DotNet.Arcade.Sdk": "8.0.0-beta.23268.1",
- "Microsoft.DotNet.Helix.Sdk": "8.0.0-beta.23268.1",
- "Microsoft.DotNet.SharedFramework.Sdk": "8.0.0-beta.23268.1",
+ "Microsoft.DotNet.Arcade.Sdk": "8.0.0-beta.23302.3",
+ "Microsoft.DotNet.Helix.Sdk": "8.0.0-beta.23302.3",
+ "Microsoft.DotNet.SharedFramework.Sdk": "8.0.0-beta.23302.3",
"Microsoft.Build.NoTargets": "3.7.0",
"Microsoft.Build.Traversal": "3.4.0",
"Microsoft.NET.Sdk.IL": "8.0.0-preview.5.23279.2"
diff --git a/src/coreclr/.nuget/Directory.Build.props b/src/coreclr/.nuget/Directory.Build.props
index 150e9ccfb0b82b..885f7f10739850 100644
--- a/src/coreclr/.nuget/Directory.Build.props
+++ b/src/coreclr/.nuget/Directory.Build.props
@@ -110,9 +110,6 @@
x86
-
- arm
- arm64
diff --git a/src/coreclr/System.Private.CoreLib/src/Internal/Runtime/InteropServices/ComActivator.cs b/src/coreclr/System.Private.CoreLib/src/Internal/Runtime/InteropServices/ComActivator.cs
index 0ee478fa675458..bc06bf44bac99d 100644
--- a/src/coreclr/System.Private.CoreLib/src/Internal/Runtime/InteropServices/ComActivator.cs
+++ b/src/coreclr/System.Private.CoreLib/src/Internal/Runtime/InteropServices/ComActivator.cs
@@ -620,15 +620,15 @@ public void CreateInstance(
ref Guid riid,
out IntPtr ppvObject)
{
- Type interfaceType = BasicClassFactory.GetValidatedInterfaceType(_classType, ref riid, pUnkOuter);
+ Type interfaceType = GetValidatedInterfaceType(_classType, ref riid, pUnkOuter);
object obj = Activator.CreateInstance(_classType)!;
if (pUnkOuter != null)
{
- obj = BasicClassFactory.CreateAggregatedObject(pUnkOuter, obj);
+ obj = CreateAggregatedObject(pUnkOuter, obj);
}
- ppvObject = BasicClassFactory.GetObjectAsInterface(obj, interfaceType);
+ ppvObject = GetObjectAsInterface(obj, interfaceType);
}
public void LockServer([MarshalAs(UnmanagedType.Bool)] bool fLock)
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Attribute.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Attribute.CoreCLR.cs
index 258d0470e71b0e..9365f8ba0b8695 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Attribute.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Attribute.CoreCLR.cs
@@ -515,10 +515,11 @@ public static bool IsDefined(MemberInfo element, Type attributeType, bool inheri
if (attrib == null || attrib.Length == 0)
return null;
+ Attribute match = attrib[0];
if (attrib.Length == 1)
- return attrib[0];
+ return match;
- throw new AmbiguousMatchException(SR.RFLCT_AmbigCust);
+ throw ThrowHelper.GetAmbiguousMatchException(match);
}
#endregion
@@ -614,10 +615,11 @@ public static bool IsDefined(ParameterInfo element, Type attributeType, bool inh
if (attrib == null || attrib.Length == 0)
return null;
+ Attribute match = attrib[0];
if (attrib.Length == 1)
- return attrib[0];
+ return match;
- throw new AmbiguousMatchException(SR.RFLCT_AmbigCust);
+ throw ThrowHelper.GetAmbiguousMatchException(match);
}
#endregion
@@ -683,10 +685,11 @@ public static bool IsDefined(Module element, Type attributeType, bool inherit)
if (attrib == null || attrib.Length == 0)
return null;
+ Attribute match = attrib[0];
if (attrib.Length == 1)
- return attrib[0];
+ return match;
- throw new AmbiguousMatchException(SR.RFLCT_AmbigCust);
+ throw ThrowHelper.GetAmbiguousMatchException(match);
}
#endregion
@@ -752,10 +755,11 @@ public static bool IsDefined(Assembly element, Type attributeType, bool inherit)
if (attrib == null || attrib.Length == 0)
return null;
+ Attribute match = attrib[0];
if (attrib.Length == 1)
- return attrib[0];
+ return match;
- throw new AmbiguousMatchException(SR.RFLCT_AmbigCust);
+ throw ThrowHelper.GetAmbiguousMatchException(match);
}
#endregion
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/Comparer.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/Comparer.CoreCLR.cs
index f6d531485b2925..5c2e18b3382b65 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/Comparer.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/Comparer.CoreCLR.cs
@@ -16,7 +16,7 @@ internal sealed partial class EnumComparer : Comparer where T : struct, En
{
public override int Compare(T x, T y)
{
- return System.Runtime.CompilerServices.RuntimeHelpers.EnumCompareTo(x, y);
+ return RuntimeHelpers.EnumCompareTo(x, y);
}
}
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/ComparerHelpers.cs b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/ComparerHelpers.cs
index b020c127629413..f07e70deaf8d88 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/ComparerHelpers.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/ComparerHelpers.cs
@@ -41,7 +41,7 @@ internal static object CreateDefaultComparer(Type type)
{
// Nullable does not implement IComparable directly because that would add an extra interface call per comparison.
var embeddedType = (RuntimeType)type.GetGenericArguments()[0];
- result = RuntimeTypeHandle.CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(NullableComparer), embeddedType);
+ result = CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(NullableComparer), embeddedType);
}
// The comparer for enums is specialized to avoid boxing.
else if (type.IsEnum)
@@ -78,7 +78,7 @@ internal static object CreateDefaultComparer(Type type)
case TypeCode.UInt32:
case TypeCode.Int64:
case TypeCode.UInt64:
- return RuntimeTypeHandle.CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(EnumComparer<>), enumType);
+ return CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(EnumComparer<>), enumType);
}
return null;
@@ -153,7 +153,7 @@ internal static object CreateDefaultEqualityComparer(Type type)
case TypeCode.Int64:
case TypeCode.UInt64:
case TypeCode.UInt16:
- return RuntimeTypeHandle.CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(EnumEqualityComparer<>), enumType);
+ return CreateInstanceForAnotherGenericParameter((RuntimeType)typeof(EnumEqualityComparer<>), enumType);
}
return null;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.CoreCLR.cs
index 2377566a05312a..bd4532bad23851 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.CoreCLR.cs
@@ -164,7 +164,7 @@ public sealed partial class EnumEqualityComparer : EqualityComparer where
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public override bool Equals(T x, T y)
{
- return System.Runtime.CompilerServices.RuntimeHelpers.EnumEquals(x, y);
+ return RuntimeHelpers.EnumEquals(x, y);
}
internal override int IndexOf(T[] array, T value, int startIndex, int count)
@@ -172,7 +172,7 @@ internal override int IndexOf(T[] array, T value, int startIndex, int count)
int endIndex = startIndex + count;
for (int i = startIndex; i < endIndex; i++)
{
- if (System.Runtime.CompilerServices.RuntimeHelpers.EnumEquals(array[i], value)) return i;
+ if (RuntimeHelpers.EnumEquals(array[i], value)) return i;
}
return -1;
}
@@ -182,7 +182,7 @@ internal override int LastIndexOf(T[] array, T value, int startIndex, int count)
int endIndex = startIndex - count + 1;
for (int i = startIndex; i >= endIndex; i--)
{
- if (System.Runtime.CompilerServices.RuntimeHelpers.EnumEquals(array[i], value)) return i;
+ if (RuntimeHelpers.EnumEquals(array[i], value)) return i;
}
return -1;
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Delegate.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Delegate.CoreCLR.cs
index e184ddfe89ed37..7fd8ed40d57b82 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Delegate.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Delegate.CoreCLR.cs
@@ -131,7 +131,7 @@ public override bool Equals([NotNullWhen(true)] object? obj)
// method ptrs don't match, go down long path
//
if (_methodBase == null || d._methodBase == null || !(_methodBase is MethodInfo) || !(d._methodBase is MethodInfo))
- return Delegate.InternalEqualMethodHandles(this, d);
+ return InternalEqualMethodHandles(this, d);
else
return _methodBase.Equals(d._methodBase);
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Eventing/NativeRuntimeEventSource.PortableThreadPool.NativeSinks.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Eventing/NativeRuntimeEventSource.PortableThreadPool.NativeSinks.CoreCLR.cs
index cbf6e4665e78ce..c618f1d5203e4e 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Eventing/NativeRuntimeEventSource.PortableThreadPool.NativeSinks.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Eventing/NativeRuntimeEventSource.PortableThreadPool.NativeSinks.CoreCLR.cs
@@ -34,7 +34,7 @@ internal sealed partial class NativeRuntimeEventSource : EventSource
[NonEvent]
[LibraryImport(RuntimeHelpers.QCall)]
- internal static partial void LogThreadPoolWorkerThreadAdjustmentAdjustment(double AverageThroughput, uint NewWorkerThreadCount, NativeRuntimeEventSource.ThreadAdjustmentReasonMap Reason, ushort ClrInstanceID);
+ internal static partial void LogThreadPoolWorkerThreadAdjustmentAdjustment(double AverageThroughput, uint NewWorkerThreadCount, ThreadAdjustmentReasonMap Reason, ushort ClrInstanceID);
[NonEvent]
[LibraryImport(RuntimeHelpers.QCall)]
diff --git a/src/coreclr/System.Private.CoreLib/src/System/GC.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/GC.CoreCLR.cs
index 20c7159380959e..d59ad132ef59ee 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/GC.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/GC.CoreCLR.cs
@@ -17,13 +17,14 @@
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading;
+using System.Runtime.Versioning;
namespace System
{
/// Specifies the behavior for a forced garbage collection.
public enum GCCollectionMode
{
- /// The default setting for this enumeration, which is currently .
+ /// The default setting for this enumeration, which is currently .
Default = 0,
/// Forces the garbage collection to occur immediately.
@@ -352,8 +353,8 @@ public static long GetTotalMemory(bool forceFullCollection)
float diff;
do
{
- GC.WaitForPendingFinalizers();
- GC.Collect();
+ WaitForPendingFinalizers();
+ Collect();
size = newSize;
newSize = GetTotalMemory();
diff = ((float)(newSize - size)) / size;
@@ -640,8 +641,8 @@ internal enum EnableNoGCRegionCallbackStatus
/// Register a callback to be invoked when we allocated a certain amount of memory in the no GC region.
/// The total size of the no GC region. Must be a number > 0 or an ArgumentOutOfRangeException will be thrown.
/// The callback to be executed when we allocated a certain amount of memory in the no GC region..
- /// The argument is less than or equal to 0.
- /// The argument is null.
+ /// The argument is less than or equal to 0.
+ /// The argument is null.
/// The GC is not currently under a NoGC region.
/// -or-
/// Another callback is already registered.
@@ -907,7 +908,7 @@ internal enum RefreshMemoryStatus
/// If the hard limit is invalid. This can happen, for example, with negative heap hard limit percentages."
///
///
- [System.Runtime.Versioning.RequiresPreviewFeaturesAttribute("RefreshMemoryLimit is in preview.")]
+ [RequiresPreviewFeatures("RefreshMemoryLimit is in preview.")]
public static void RefreshMemoryLimit()
{
ulong heapHardLimit = (AppContext.GetData("GCHeapHardLimit") as ulong?) ?? ulong.MaxValue;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/IO/FileLoadException.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/IO/FileLoadException.CoreCLR.cs
index 4933f9189f133b..439689ca9adc26 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/IO/FileLoadException.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/IO/FileLoadException.CoreCLR.cs
@@ -23,7 +23,7 @@ internal static string FormatFileLoadExceptionMessage(string? fileName, int hRes
GetFileLoadExceptionMessage(hResult, new StringHandleOnStack(ref format));
string? message = null;
- if (hResult == System.HResults.COR_E_BADEXEFORMAT)
+ if (hResult == HResults.COR_E_BADEXEFORMAT)
message = SR.Arg_BadImageFormatException;
else
GetMessageForHR(hResult, new StringHandleOnStack(ref message));
diff --git a/src/coreclr/System.Private.CoreLib/src/System/MulticastDelegate.cs b/src/coreclr/System.Private.CoreLib/src/System/MulticastDelegate.cs
index 5410a6c2460732..038d8db6084119 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/MulticastDelegate.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/MulticastDelegate.cs
@@ -5,6 +5,7 @@
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Reflection;
+using System.Reflection.Emit;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Runtime.Serialization;
@@ -43,7 +44,7 @@ internal bool IsUnmanagedFunctionPtr()
internal bool InvocationListLogicallyNull()
{
- return (_invocationList == null) || (_invocationList is LoaderAllocator) || (_invocationList is System.Reflection.Emit.DynamicResolver);
+ return (_invocationList == null) || (_invocationList is LoaderAllocator) || (_invocationList is DynamicResolver);
}
[Obsolete(Obsoletions.LegacyFormatterImplMessage, DiagnosticId = Obsoletions.LegacyFormatterImplDiagId, UrlFormat = Obsoletions.SharedUrlFormat)]
@@ -59,7 +60,7 @@ public sealed override bool Equals([NotNullWhen(true)] object? obj)
{
if (obj == null)
return false;
- if (object.ReferenceEquals(this, obj))
+ if (ReferenceEquals(this, obj))
return true;
if (!InternalEqualTypes(this, obj))
return false;
@@ -158,7 +159,7 @@ private bool InvocationListEquals(MulticastDelegate d)
private static bool TrySetSlot(object?[] a, int index, object o)
{
- if (a[index] == null && System.Threading.Interlocked.CompareExchange(ref a[index], o, null) == null)
+ if (a[index] == null && Threading.Interlocked.CompareExchange(ref a[index], o, null) == null)
return true;
// The slot may be already set because we have added and removed the same method before.
@@ -565,13 +566,13 @@ protected override MethodInfo GetMethodImpl()
// this should help inlining
[DoesNotReturn]
- [System.Diagnostics.DebuggerNonUserCode]
+ [DebuggerNonUserCode]
private static void ThrowNullThisInDelegateToInstance() =>
throw new ArgumentException(SR.Arg_DlgtNullInst);
#pragma warning disable IDE0060
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorClosed(object target, IntPtr methodPtr)
{
if (target == null)
@@ -580,24 +581,24 @@ private void CtorClosed(object target, IntPtr methodPtr)
this._methodPtr = methodPtr;
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorClosedStatic(object target, IntPtr methodPtr)
{
this._target = target;
this._methodPtr = methodPtr;
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorRTClosed(object target, IntPtr methodPtr)
{
this._target = target;
this._methodPtr = AdjustTarget(target, methodPtr);
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorOpened(object target, IntPtr methodPtr, IntPtr shuffleThunk)
{
this._target = this;
@@ -605,8 +606,8 @@ private void CtorOpened(object target, IntPtr methodPtr, IntPtr shuffleThunk)
this._methodPtrAux = methodPtr;
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorVirtualDispatch(object target, IntPtr methodPtr, IntPtr shuffleThunk)
{
this._target = this;
@@ -614,33 +615,33 @@ private void CtorVirtualDispatch(object target, IntPtr methodPtr, IntPtr shuffle
this._methodPtrAux = GetCallStub(methodPtr);
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorCollectibleClosedStatic(object target, IntPtr methodPtr, IntPtr gchandle)
{
this._target = target;
this._methodPtr = methodPtr;
- this._methodBase = System.Runtime.InteropServices.GCHandle.InternalGet(gchandle);
+ this._methodBase = GCHandle.InternalGet(gchandle);
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorCollectibleOpened(object target, IntPtr methodPtr, IntPtr shuffleThunk, IntPtr gchandle)
{
this._target = this;
this._methodPtr = shuffleThunk;
this._methodPtrAux = methodPtr;
- this._methodBase = System.Runtime.InteropServices.GCHandle.InternalGet(gchandle);
+ this._methodBase = GCHandle.InternalGet(gchandle);
}
- [System.Diagnostics.DebuggerNonUserCode]
- [System.Diagnostics.DebuggerStepThrough]
+ [DebuggerNonUserCode]
+ [DebuggerStepThrough]
private void CtorCollectibleVirtualDispatch(object target, IntPtr methodPtr, IntPtr shuffleThunk, IntPtr gchandle)
{
this._target = this;
this._methodPtr = shuffleThunk;
this._methodPtrAux = GetCallStub(methodPtr);
- this._methodBase = System.Runtime.InteropServices.GCHandle.InternalGet(gchandle);
+ this._methodBase = GCHandle.InternalGet(gchandle);
}
#pragma warning restore IDE0060
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Assembly.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Assembly.CoreCLR.cs
index a2aa467d9a8c38..a0d06e8e8197bc 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Assembly.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Assembly.CoreCLR.cs
@@ -6,6 +6,7 @@
using System.Runtime.InteropServices;
using System.Runtime.Loader;
using System.Runtime.Serialization;
+using System.Security;
using StackCrawlMark = System.Threading.StackCrawlMark;
namespace System.Reflection
@@ -14,7 +15,7 @@ public abstract partial class Assembly : ICustomAttributeProvider, ISerializable
{
// Locate an assembly by the long form of the assembly name.
// eg. "Toolbox.dll, version=1.1.10.1220, locale=en, publickey=1234567890123456789012345678901234567890"
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Assembly Load(string assemblyString)
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
@@ -22,7 +23,7 @@ public static Assembly Load(string assemblyString)
}
[Obsolete("Assembly.LoadWithPartialName has been deprecated. Use Assembly.Load() instead.")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Assembly? LoadWithPartialName(string partialName)
{
ArgumentNullException.ThrowIfNull(partialName);
@@ -43,7 +44,7 @@ public static Assembly Load(string assemblyString)
// Locate an assembly by its name. The name can be strong or
// weak. The assembly is loaded into the domain of the caller.
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Assembly Load(AssemblyName assemblyRef)
{
ArgumentNullException.ThrowIfNull(assemblyRef);
@@ -63,14 +64,14 @@ internal static RuntimeAssembly GetExecutingAssembly(ref StackCrawlMark stackMar
}
// Get the assembly that the current code is running from.
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Assembly GetExecutingAssembly()
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
return GetExecutingAssembly(ref stackMark);
}
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Assembly GetCallingAssembly()
{
// LookForMyCallersCaller is not guaranteed to return the correct stack frame
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeAssemblyBuilder.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeAssemblyBuilder.cs
index dd6c84158a01fe..d008fed2a107b5 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeAssemblyBuilder.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeAssemblyBuilder.cs
@@ -22,7 +22,7 @@ public partial class AssemblyBuilder
[RequiresDynamicCode("Defining a dynamic assembly requires dynamic code.")]
[DynamicSecurityMethod] // Required to make Assembly.GetCallingAssembly reliable.
public static AssemblyBuilder DefineDynamicAssembly(AssemblyName name, AssemblyBuilderAccess access)
- => DefineDynamicAssembly(name, access, null, Assembly.GetCallingAssembly());
+ => DefineDynamicAssembly(name, access, null, GetCallingAssembly());
[RequiresDynamicCode("Defining a dynamic assembly requires dynamic code.")]
[DynamicSecurityMethod] // Required to make Assembly.GetCallingAssembly reliable.
@@ -30,7 +30,7 @@ public static AssemblyBuilder DefineDynamicAssembly(
AssemblyName name,
AssemblyBuilderAccess access,
IEnumerable? assemblyAttributes)
- => DefineDynamicAssembly(name, access, assemblyAttributes, Assembly.GetCallingAssembly());
+ => DefineDynamicAssembly(name, access, assemblyAttributes, GetCallingAssembly());
private static RuntimeAssemblyBuilder DefineDynamicAssembly(
AssemblyName name,
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeEnumBuilder.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeEnumBuilder.cs
index 2e14270c0d0265..1d0703dc51c02e 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeEnumBuilder.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeEnumBuilder.cs
@@ -334,7 +334,7 @@ internal RuntimeEnumBuilder(
// Client should not set any bits other than the visibility bits.
if ((visibility & ~TypeAttributes.VisibilityMask) != 0)
throw new ArgumentException(SR.Argument_ShouldOnlySetVisibilityFlags, nameof(name));
- m_typeBuilder = new RuntimeTypeBuilder(name, visibility | TypeAttributes.Sealed, typeof(System.Enum), null, module, PackingSize.Unspecified, TypeBuilder.UnspecifiedTypeSize, null);
+ m_typeBuilder = new RuntimeTypeBuilder(name, visibility | TypeAttributes.Sealed, typeof(Enum), null, module, PackingSize.Unspecified, TypeBuilder.UnspecifiedTypeSize, null);
// Define the underlying field for the enum. It will be a non-static, private field with special name bit set.
m_underlyingField = m_typeBuilder.DefineField("value__", underlyingType, FieldAttributes.Public | FieldAttributes.SpecialName | FieldAttributes.RTSpecialName);
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeMethodBuilder.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeMethodBuilder.cs
index f0cf9e6440341c..3c2daf38b082f2 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeMethodBuilder.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeMethodBuilder.cs
@@ -718,7 +718,7 @@ private static bool IsKnownCA(ConstructorInfo con)
private void ParseCA(ConstructorInfo con)
{
Type? caType = con.DeclaringType;
- if (caType == typeof(System.Runtime.CompilerServices.MethodImplAttribute))
+ if (caType == typeof(MethodImplAttribute))
{
// dig through the blob looking for the MethodImplAttributes flag
// that must be in the MethodCodeType field
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeTypeBuilder.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeTypeBuilder.cs
index 79aaaebd328b13..c9147ca420a28e 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeTypeBuilder.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Emit/RuntimeTypeBuilder.cs
@@ -574,7 +574,7 @@ private FieldBuilder DefineDataHelper(string name, byte[]? data, int size, Field
typeAttributes = TypeAttributes.Public | TypeAttributes.ExplicitLayout | TypeAttributes.Class | TypeAttributes.Sealed | TypeAttributes.AnsiClass;
// Define the backing value class
- valueClassType = m_module.DefineType(strValueClassName, typeAttributes, typeof(System.ValueType), PackingSize.Size1, size);
+ valueClassType = m_module.DefineType(strValueClassName, typeAttributes, typeof(ValueType), PackingSize.Size1, size);
valueClassType.CreateType();
}
@@ -827,7 +827,7 @@ public override Type[] GetInterfaces()
if (m_typeInterfaces == null)
{
- return Type.EmptyTypes;
+ return EmptyTypes;
}
return m_typeInterfaces.ToArray();
@@ -1134,7 +1134,7 @@ protected override GenericTypeParameterBuilder[] DefineGenericParametersCore(par
return m_inst;
}
- public override Type[] GetGenericArguments() => m_inst ?? Type.EmptyTypes;
+ public override Type[] GetGenericArguments() => m_inst ?? EmptyTypes;
// If a TypeBuilder is generic, it must be a generic type definition
// All instantiated generic types are TypeBuilderInstantiation.
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/MdFieldInfo.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/MdFieldInfo.cs
index 7d5cd5328e81f1..cda066bba16e20 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/MdFieldInfo.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/MdFieldInfo.cs
@@ -64,22 +64,22 @@ public override int GetHashCode() =>
public override bool IsSecuritySafeCritical => DeclaringType!.IsSecuritySafeCritical;
public override bool IsSecurityTransparent => DeclaringType!.IsSecurityTransparent;
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override object? GetValueDirect(TypedReference obj)
{
return GetValue(null);
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override void SetValueDirect(TypedReference obj, object value)
{
throw new FieldAccessException(SR.Acc_ReadOnly);
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override object? GetValue(object? obj)
{
return GetValue(false);
@@ -99,8 +99,8 @@ public override void SetValueDirect(TypedReference obj, object value)
return value;
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override void SetValue(object? obj, object? value, BindingFlags invokeAttr, Binder? binder, CultureInfo? culture)
{
throw new FieldAccessException(SR.Acc_ReadOnly);
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs
index bae43fd0e9147d..c3dbc2a73f2d6c 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs
@@ -20,7 +20,7 @@ public static partial class AssemblyExtensions
/// When this method returns, contains the length of the metadata section blob.
///
/// if the metadata is retrieved successfully; upon failure.
- /// The metadata might not be available for some assemblies, such as , AOT images, etc.
+ /// The metadata might not be available for some assemblies, such as , AOT images, etc.
///
///
/// Callers should not write to the metadata blob.
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBase.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBase.CoreCLR.cs
index afed33fe1f3bc2..e8bb0fa1814939 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBase.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBase.CoreCLR.cs
@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.
using System.Diagnostics.CodeAnalysis;
+using System.Security;
using System.Threading;
namespace System.Reflection
@@ -34,7 +35,7 @@ public abstract partial class MethodBase : MemberInfo
}
[RequiresUnreferencedCode("Metadata for the method might be incomplete or removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static MethodBase? GetCurrentMethod()
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/ModifiedType.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/ModifiedType.CoreCLR.cs
index 60ab5dd00d102d..394b6b4ecffa9e 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/ModifiedType.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/ModifiedType.CoreCLR.cs
@@ -30,6 +30,6 @@ internal static Type Create(Type unmodifiedType, Signature? signature, int param
private Type[] GetCustomModifiers(bool required) =>
(_typeSignature._signature != null) ?
- _typeSignature._signature.GetCustomModifiersAtOffset(_typeSignature._offset, required) : Type.EmptyTypes;
+ _typeSignature._signature.GetCustomModifiersAtOffset(_typeSignature._offset, required) : EmptyTypes;
}
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeAssembly.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeAssembly.cs
index 10fd2c215dd6ac..26e75846401dfc 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeAssembly.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeAssembly.cs
@@ -14,6 +14,7 @@
using System.Runtime.CompilerServices;
using System.Runtime.Serialization;
using System.Threading;
+using System.Security;
namespace System.Reflection
{
@@ -344,7 +345,7 @@ public override void GetObjectData(SerializationInfo info, StreamingContext cont
public override Module ManifestModule =>
// We don't need to return the "external" ModuleBuilder because
// it is meant to be read-only
- RuntimeAssembly.GetManifestModule(GetNativeHandle());
+ GetManifestModule(GetNativeHandle());
public override object[] GetCustomAttributes(bool inherit)
{
@@ -657,7 +658,7 @@ public override Assembly GetSatelliteAssembly(CultureInfo culture, Version? vers
return InternalGetSatelliteAssembly(culture, version, throwOnFileNotFound: true)!;
}
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
internal Assembly? InternalGetSatelliteAssembly(CultureInfo culture,
Version? version,
bool throwOnFileNotFound)
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeCustomAttributeData.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeCustomAttributeData.cs
index c601a2d7d08997..f3578517a4df78 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeCustomAttributeData.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeCustomAttributeData.cs
@@ -928,7 +928,7 @@ internal static object[] GetCustomAttributes(RuntimeType type, RuntimeType caTyp
// if we are asked to go up the hierarchy chain we have to do it now and regardless of the
// attribute usage for the specific attribute because a derived attribute may override the usage...
// ... however if the attribute is sealed we can rely on the attribute usage
- if (!inherit || (caType.IsSealed && !CustomAttribute.GetAttributeUsage(caType).Inherited))
+ if (!inherit || (caType.IsSealed && !GetAttributeUsage(caType).Inherited))
{
object[] attributes = GetCustomAttributes(type.GetRuntimeModule(), type.MetadataToken, pcas.Count, caType);
if (pcas.Count > 0) pcas.CopyTo(attributes, attributes.Length - pcas.Count);
@@ -970,7 +970,7 @@ internal static object[] GetCustomAttributes(RuntimeMethodInfo method, RuntimeTy
// if we are asked to go up the hierarchy chain we have to do it now and regardless of the
// attribute usage for the specific attribute because a derived attribute may override the usage...
// ... however if the attribute is sealed we can rely on the attribute usage
- if (!inherit || (caType.IsSealed && !CustomAttribute.GetAttributeUsage(caType).Inherited))
+ if (!inherit || (caType.IsSealed && !GetAttributeUsage(caType).Inherited))
{
object[] attributes = GetCustomAttributes(method.GetRuntimeModule(), method.MetadataToken, pcas.Count, caType);
if (pcas.Count > 0) pcas.CopyTo(attributes, attributes.Length - pcas.Count);
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs
index 0d25286564e343..24634ed9715fe2 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs
@@ -454,7 +454,7 @@ public override MethodInfo MakeGenericMethod(params Type[] methodInstantiation)
for (int iCopy = 0; iCopy < methodInstantiation.Length; iCopy++)
methodInstantiationCopy[iCopy] = methodInstantiation[iCopy];
methodInstantiation = methodInstantiationCopy;
- return System.Reflection.Emit.MethodBuilderInstantiation.MakeGenericMethod(this, methodInstantiation);
+ return Emit.MethodBuilderInstantiation.MakeGenericMethod(this, methodInstantiation);
}
methodInstantionRuntimeType[i] = rtMethodInstantiationElem;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeModule.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeModule.cs
index 0f8683c6633e47..a3a98b78af9a5a 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeModule.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeModule.cs
@@ -512,7 +512,7 @@ public override string Name
{
string s = GetFullyQualifiedName();
- int i = s.LastIndexOf(System.IO.Path.DirectorySeparatorChar);
+ int i = s.LastIndexOf(IO.Path.DirectorySeparatorChar);
if (i < 0)
return s;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeParameterInfo.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeParameterInfo.cs
index 6ae15363e51175..5af77b790f49c3 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeParameterInfo.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeParameterInfo.cs
@@ -259,9 +259,7 @@ public override bool HasDefaultValue
if (m_noMetadata || m_noDefaultValue)
return false;
- object? defaultValue = GetDefaultValueInternal(false);
-
- return defaultValue != DBNull.Value;
+ return TryGetDefaultValueInternal(false, out _);
}
}
@@ -277,9 +275,7 @@ public override bool HasDefaultValue
return null;
// for dynamic method we pretend to have cached the value so we do not go to metadata
- object? defaultValue = GetDefaultValueInternal(raw);
-
- if (defaultValue == DBNull.Value)
+ if (!TryGetDefaultValueInternal(raw, out object? defaultValue))
{
#region Handle case if no default value was found
if (IsOptional)
@@ -295,7 +291,7 @@ public override bool HasDefaultValue
private object? GetDefaultValueFromCustomAttributeData()
{
- foreach (CustomAttributeData attributeData in RuntimeCustomAttributeData.GetCustomAttributes(this))
+ foreach (CustomAttributeData attributeData in CustomAttributeData.GetCustomAttributes(this))
{
Type attributeType = attributeData.AttributeType;
if (attributeType == typeof(DecimalConstantAttribute))
@@ -328,23 +324,21 @@ public override bool HasDefaultValue
}
// returns DBNull.Value if the parameter doesn't have a default value
- private object? GetDefaultValueInternal(bool raw)
+ private bool TryGetDefaultValueInternal(bool raw, out object? defaultValue)
{
Debug.Assert(!m_noMetadata);
- if (m_noDefaultValue)
- return DBNull.Value;
-
- object? defaultValue = null;
+ if (m_noDefaultValue || MdToken.IsNullToken(m_tkParamDef))
+ {
+ defaultValue = DBNull.Value;
+ m_noDefaultValue = true;
+ return false;
+ }
// Prioritize metadata constant over custom attribute constant
#region Look for a default value in metadata
- if (!MdToken.IsNullToken(m_tkParamDef))
- {
- // This will return DBNull.Value if no constant value is defined on m_tkParamDef in the metadata.
- defaultValue = MdConstant.GetValue(m_scope, m_tkParamDef, ParameterType.TypeHandle, raw);
- }
- #endregion
+ // This will return DBNull.Value if no constant value is defined on m_tkParamDef in the metadata.
+ defaultValue = MdConstant.GetValue(m_scope, m_tkParamDef, ParameterType.TypeHandle, raw);
// If default value is not specified in metadata, look for it in custom attributes
if (defaultValue == DBNull.Value)
@@ -357,18 +351,23 @@ public override bool HasDefaultValue
// IMPORTANT: Please note that there is a subtle difference in order custom attributes are inspected for
// RawDefaultValue and DefaultValue.
defaultValue = raw ? GetDefaultValueFromCustomAttributeData() : GetDefaultValueFromCustomAttributes();
+
+ if (defaultValue == DBNull.Value)
+ {
+ m_noDefaultValue = true;
+ return false;
+ }
}
- if (defaultValue == DBNull.Value)
- m_noDefaultValue = true;
+ return true;
- return defaultValue;
+ #endregion
}
private static decimal GetRawDecimalConstant(CustomAttributeData attr)
{
Debug.Assert(attr.Constructor.DeclaringType == typeof(DecimalConstantAttribute));
- System.Collections.Generic.IList args = attr.ConstructorArguments;
+ IList args = attr.ConstructorArguments;
Debug.Assert(args.Count == 5);
return new decimal(
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimePropertyInfo.cs b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimePropertyInfo.cs
index c48c24dd7e7a34..d2731883b02f89 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimePropertyInfo.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimePropertyInfo.cs
@@ -331,26 +331,26 @@ internal ParameterInfo[] GetIndexParametersNoCopy()
#endregion
#region Dynamic
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override object? GetValue(object? obj, object?[]? index)
{
return GetValue(obj, BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Static,
null, index, null);
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override object? GetValue(object? obj, BindingFlags invokeAttr, Binder? binder, object?[]? index, CultureInfo? culture)
{
RuntimeMethodInfo? m = GetGetMethod(true);
if (m == null)
- throw new ArgumentException(System.SR.Arg_GetMethNotFnd);
+ throw new ArgumentException(SR.Arg_GetMethNotFnd);
return m.Invoke(obj, invokeAttr, binder, index, null);
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override void SetValue(object? obj, object? value, object?[]? index)
{
SetValue(obj,
@@ -361,14 +361,14 @@ public override void SetValue(object? obj, object? value, object?[]? index)
null);
}
- [DebuggerStepThroughAttribute]
- [Diagnostics.DebuggerHidden]
+ [DebuggerStepThrough]
+ [DebuggerHidden]
public override void SetValue(object? obj, object? value, BindingFlags invokeAttr, Binder? binder, object?[]? index, CultureInfo? culture)
{
RuntimeMethodInfo? m = GetSetMethod(true);
if (m == null)
- throw new ArgumentException(System.SR.Arg_SetMethNotFnd);
+ throw new ArgumentException(SR.Arg_SetMethNotFnd);
if (index is null)
{
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs
index 7a44b24f5b6741..831582694aa534 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs
@@ -49,7 +49,7 @@ public static partial class RuntimeHelpers
// This call will generate an exception if the specified class constructor threw an
// exception when it ran.
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "ReflectionInvocation_RunClassConstructor")]
+ [LibraryImport(QCall, EntryPoint = "ReflectionInvocation_RunClassConstructor")]
private static partial void RunClassConstructor(QCallTypeHandle type);
[RequiresUnreferencedCode("Trimmer can't guarantee existence of class constructor")]
@@ -70,7 +70,7 @@ public static void RunClassConstructor(RuntimeTypeHandle type)
// This call will generate an exception if the specified module constructor threw an
// exception when it ran.
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "ReflectionInvocation_RunModuleConstructor")]
+ [LibraryImport(QCall, EntryPoint = "ReflectionInvocation_RunModuleConstructor")]
private static partial void RunModuleConstructor(QCallModule module);
public static void RunModuleConstructor(ModuleHandle module)
@@ -82,10 +82,10 @@ public static void RunModuleConstructor(ModuleHandle module)
RunModuleConstructor(new QCallModule(ref rm));
}
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "ReflectionInvocation_CompileMethod")]
+ [LibraryImport(QCall, EntryPoint = "ReflectionInvocation_CompileMethod")]
internal static partial void CompileMethod(RuntimeMethodHandleInternal method);
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "ReflectionInvocation_PrepareMethod")]
+ [LibraryImport(QCall, EntryPoint = "ReflectionInvocation_PrepareMethod")]
private static unsafe partial void PrepareMethod(RuntimeMethodHandleInternal method, IntPtr* pInstantiation, int cInstantiation);
public static void PrepareMethod(RuntimeMethodHandle method) => PrepareMethod(method, null);
@@ -133,7 +133,7 @@ public static int OffsetToStringData
{
// This offset is baked in by string indexer intrinsic, so there is no harm
// in getting it baked in here as well.
- [System.Runtime.Versioning.NonVersionable]
+ [NonVersionable]
get =>
// Number of bytes from the address pointed to by a reference to
// a String to the first 16-bit character in the String. Skip
@@ -182,7 +182,7 @@ public static object GetUninitializedObject(
return obj!;
}
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "ReflectionSerialization_GetUninitializedObject")]
+ [LibraryImport(QCall, EntryPoint = "ReflectionSerialization_GetUninitializedObject")]
private static partial void GetUninitializedObject(QCallTypeHandle type, ObjectHandleOnStack retObject);
[MethodImpl(MethodImplOptions.InternalCall)]
@@ -309,13 +309,13 @@ internal static unsafe bool ObjectHasComponentSize(object obj)
}
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "MethodTable_AreTypesEquivalent")]
+ [LibraryImport(QCall, EntryPoint = "MethodTable_AreTypesEquivalent")]
[return: MarshalAs(UnmanagedType.Bool)]
internal static unsafe partial bool AreTypesEquivalent(MethodTable* pMTa, MethodTable* pMTb);
///
/// Allocate memory that is associated with the and
- /// will be freed if and when the is unloaded.
+ /// will be freed if and when the is unloaded.
///
/// Type associated with the allocated memory.
/// Amount of memory in bytes to allocate.
@@ -330,7 +330,7 @@ public static IntPtr AllocateTypeAssociatedMemory(Type type, int size)
return AllocateTypeAssociatedMemory(new QCallTypeHandle(ref rt), (uint)size);
}
- [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "RuntimeTypeHandle_AllocateTypeAssociatedMemory")]
+ [LibraryImport(QCall, EntryPoint = "RuntimeTypeHandle_AllocateTypeAssociatedMemory")]
private static partial IntPtr AllocateTypeAssociatedMemory(QCallTypeHandle type, uint size);
[MethodImpl(MethodImplOptions.InternalCall)]
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/ControlledExecution.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/ControlledExecution.CoreCLR.cs
index b6c437b351b7b1..67971b90caa1b6 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/ControlledExecution.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/ControlledExecution.CoreCLR.cs
@@ -21,12 +21,12 @@ public static partial class ControlledExecution
///
/// The delegate that represents the code to execute.
/// The cancellation token that may be used to abort execution.
- /// The method is not supported on this platform.
- /// The argument is null.
- ///
- /// The current thread is already running the method.
+ /// The method is not supported on this platform.
+ /// The argument is null.
+ ///
+ /// The current thread is already running the method.
///
- /// The execution was aborted.
+ /// The execution was aborted.
///
/// This method enables aborting arbitrary managed code in a non-cooperative manner by throwing an exception
/// in the thread executing that code. While the exception may be caught by the code, it is re-thrown at the end
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComTypes/IEnumerable.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComTypes/IEnumerable.cs
index bfe2ad30092cf6..b3ab4f4056b1ce 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComTypes/IEnumerable.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComTypes/IEnumerable.cs
@@ -15,6 +15,6 @@ namespace System.Runtime.InteropServices.ComTypes
internal interface IEnumerable
{
[DispId(-4)]
- System.Collections.IEnumerator GetEnumerator();
+ Collections.IEnumerator GetEnumerator();
}
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComWrappers.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComWrappers.cs
index e3293c6e12c541..9a334d044d5d77 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComWrappers.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/ComWrappers.cs
@@ -235,7 +235,7 @@ public object GetOrCreateObjectForComInstance(IntPtr externalComObject, CreateOb
/// The to be used as the wrapper for the external object
/// Returns a managed object associated with the supplied external COM object.
///
- /// If the instance already has an associated external object a will be thrown.
+ /// If the instance already has an associated external object a will be thrown.
///
public object GetOrRegisterObjectForComInstance(IntPtr externalComObject, CreateObjectFlags flags, object wrapper)
{
@@ -256,7 +256,7 @@ public object GetOrRegisterObjectForComInstance(IntPtr externalComObject, Create
/// in an unknown apartment state. If the supplied inner is not known to be a free-threaded instance then
/// it is advised to not supply the inner.
///
- /// If the instance already has an associated external object a will be thrown.
+ /// If the instance already has an associated external object a will be thrown.
///
public object GetOrRegisterObjectForComInstance(IntPtr externalComObject, CreateObjectFlags flags, object wrapper, IntPtr inner)
{
@@ -315,7 +315,7 @@ internal static void CallReleaseObjects(ComWrappers? comWrappersImpl, IEnumerabl
/// Instance to register
///
/// This function can only be called a single time. Subsequent calls to this function will result
- /// in a being thrown.
+ /// in a being thrown.
///
/// Scenarios where this global instance may be used are:
/// * Object tracking via the and flags.
@@ -343,7 +343,7 @@ public static void RegisterForTrackerSupport(ComWrappers instance)
/// Instance to register
///
/// This function can only be called a single time. Subsequent calls to this function will result
- /// in a being thrown.
+ /// in a being thrown.
///
/// Scenarios where this global instance may be used are:
/// * Usage of COM-related Marshal APIs
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumerableViewOfDispatch.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumerableViewOfDispatch.cs
index d9d941138ca609..fa94eefcfb6a8d 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumerableViewOfDispatch.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumerableViewOfDispatch.cs
@@ -6,7 +6,7 @@
namespace System.Runtime.InteropServices.CustomMarshalers
{
- internal sealed class EnumerableViewOfDispatch : ICustomAdapter, System.Collections.IEnumerable
+ internal sealed class EnumerableViewOfDispatch : ICustomAdapter, Collections.IEnumerable
{
// Reserved DISPID slot for getting an enumerator from an IDispatch-implementing COM interface.
private const int DISPID_NEWENUM = -4;
@@ -20,7 +20,7 @@ public EnumerableViewOfDispatch(object dispatch)
private IDispatch Dispatch => (IDispatch)_dispatch;
- public System.Collections.IEnumerator GetEnumerator()
+ public Collections.IEnumerator GetEnumerator()
{
Variant result;
unsafe
@@ -50,7 +50,7 @@ public System.Collections.IEnumerator GetEnumerator()
}
enumVariantPtr = Marshal.GetIUnknownForObject(enumVariant);
- return (System.Collections.IEnumerator)EnumeratorToEnumVariantMarshaler.GetInstance(null).MarshalNativeToManaged(enumVariantPtr);
+ return (Collections.IEnumerator)EnumeratorToEnumVariantMarshaler.GetInstance(null).MarshalNativeToManaged(enumVariantPtr);
}
finally
{
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumeratorViewOfEnumVariant.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumeratorViewOfEnumVariant.cs
index a62fad1a0dc543..25f03f14133e97 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumeratorViewOfEnumVariant.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/CustomMarshalers/EnumeratorViewOfEnumVariant.cs
@@ -5,7 +5,7 @@
namespace System.Runtime.InteropServices.CustomMarshalers
{
- internal sealed class EnumeratorViewOfEnumVariant : ICustomAdapter, System.Collections.IEnumerator
+ internal sealed class EnumeratorViewOfEnumVariant : ICustomAdapter, Collections.IEnumerator
{
private readonly IEnumVARIANT _enumVariantObject;
private bool _fetchedLastObject;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/DynamicInterfaceCastableHelpers.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/DynamicInterfaceCastableHelpers.cs
index da3cd7f1059e27..588ba04b139616 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/DynamicInterfaceCastableHelpers.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/DynamicInterfaceCastableHelpers.cs
@@ -1,6 +1,8 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+using System.Diagnostics;
+
namespace System.Runtime.InteropServices
{
///
@@ -10,7 +12,7 @@ namespace System.Runtime.InteropServices
///
internal static class DynamicInterfaceCastableHelpers
{
- [Diagnostics.StackTraceHidden]
+ [StackTraceHidden]
internal static bool IsInterfaceImplemented(IDynamicInterfaceCastable castable, RuntimeType interfaceType, bool throwIfNotImplemented)
{
bool isImplemented= castable.IsInterfaceImplemented(new RuntimeTypeHandle(interfaceType), throwIfNotImplemented);
@@ -20,7 +22,7 @@ internal static bool IsInterfaceImplemented(IDynamicInterfaceCastable castable,
return isImplemented;
}
- [Diagnostics.StackTraceHidden]
+ [StackTraceHidden]
internal static RuntimeType? GetInterfaceImplementation(IDynamicInterfaceCastable castable, RuntimeType interfaceType)
{
RuntimeTypeHandle handle = castable.GetInterfaceImplementation(new RuntimeTypeHandle(interfaceType));
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs
index f5650b4e1ff7b5..8e2596f994fd1a 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs
@@ -133,7 +133,7 @@ public static void WriteByte(object ptr, int ofs, byte val)
[RequiresDynamicCode("Marshalling code for the object might not be available")]
public static void WriteInt16(object ptr, int ofs, short val)
{
- WriteValueSlow(ptr, ofs, val, Marshal.WriteInt16);
+ WriteValueSlow(ptr, ofs, val, WriteInt16);
}
[EditorBrowsable(EditorBrowsableState.Never)]
@@ -141,7 +141,7 @@ public static void WriteInt16(object ptr, int ofs, short val)
[RequiresDynamicCode("Marshalling code for the object might not be available")]
public static void WriteInt32(object ptr, int ofs, int val)
{
- WriteValueSlow(ptr, ofs, val, Marshal.WriteInt32);
+ WriteValueSlow(ptr, ofs, val, WriteInt32);
}
[EditorBrowsable(EditorBrowsableState.Never)]
@@ -149,7 +149,7 @@ public static void WriteInt32(object ptr, int ofs, int val)
[RequiresDynamicCode("Marshalling code for the object might not be available")]
public static void WriteInt64(object ptr, int ofs, long val)
{
- WriteValueSlow(ptr, ofs, val, Marshal.WriteInt64);
+ WriteValueSlow(ptr, ofs, val, WriteInt64);
}
///
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs
index 75ce9685828218..b3b05a381218c4 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs
@@ -156,7 +156,7 @@ private static IntPtr ResolveUnmanagedDllUsingEvent(string unmanagedDllName, Ass
{
// If the load context is returned null, then the assembly was bound using the TPA binder
// and we shall return reference to the "Default" binder.
- loadContextForAssembly = AssemblyLoadContext.Default;
+ loadContextForAssembly = Default;
}
else
{
@@ -217,7 +217,7 @@ private static void StopAssemblyLoad(ref Guid activityId)
///
private static void InitializeDefaultContext()
{
- _ = AssemblyLoadContext.Default;
+ _ = Default;
}
}
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/RuntimeHandles.cs b/src/coreclr/System.Private.CoreLib/src/System/RuntimeHandles.cs
index 38125e88795cd9..bd1e343398f9b9 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/RuntimeHandles.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/RuntimeHandles.cs
@@ -312,7 +312,7 @@ internal RuntimeType GetRuntimeType()
public ModuleHandle GetModuleHandle()
{
- return new ModuleHandle(RuntimeTypeHandle.GetModule(m_type));
+ return new ModuleHandle(GetModule(m_type));
}
[MethodImpl(MethodImplOptions.InternalCall)]
@@ -350,7 +350,7 @@ internal struct IntroducedMethodEnumerator
internal IntroducedMethodEnumerator(RuntimeType type)
{
- _handle = RuntimeTypeHandle.GetFirstIntroducedMethod(type);
+ _handle = GetFirstIntroducedMethod(type);
_firstCall = true;
}
@@ -362,7 +362,7 @@ public bool MoveNext()
}
else if (_handle.Value != IntPtr.Zero)
{
- RuntimeTypeHandle.GetNextIntroducedMethod(ref _handle);
+ GetNextIntroducedMethod(ref _handle);
}
return !(_handle.Value == IntPtr.Zero);
}
@@ -457,7 +457,7 @@ internal static bool IsComObject(RuntimeType type, bool isGenericCOM)
if (isGenericCOM)
return type.TypeHandle.Value == typeof(__ComObject).TypeHandle.Value;
- return RuntimeTypeHandle.CanCastTo(type, (RuntimeType)typeof(__ComObject));
+ return CanCastTo(type, (RuntimeType)typeof(__ComObject));
#else
return false;
#endif
@@ -803,7 +803,7 @@ public override bool Equals(object? obj)
public static RuntimeMethodHandle FromIntPtr(IntPtr value)
{
var handle = new RuntimeMethodHandleInternal(value);
- var methodInfo = new RuntimeMethodInfoStub(handle, RuntimeMethodHandle.GetLoaderAllocator(handle));
+ var methodInfo = new RuntimeMethodInfoStub(handle, GetLoaderAllocator(handle));
return new RuntimeMethodHandle(methodInfo);
}
@@ -860,7 +860,7 @@ internal static partial Interop.BOOL IsCAVisibleFromDecoratedType(
internal static MethodAttributes GetAttributes(IRuntimeMethodInfo method)
{
- MethodAttributes retVal = RuntimeMethodHandle.GetAttributes(method.Value);
+ MethodAttributes retVal = GetAttributes(method.Value);
GC.KeepAlive(method);
return retVal;
}
@@ -885,7 +885,7 @@ internal static string ConstructInstantiation(IRuntimeMethodInfo method, TypeNam
internal static RuntimeType GetDeclaringType(IRuntimeMethodInfo method)
{
- RuntimeType type = RuntimeMethodHandle.GetDeclaringType(method.Value);
+ RuntimeType type = GetDeclaringType(method.Value);
GC.KeepAlive(method);
return type;
}
@@ -897,7 +897,7 @@ internal static int GetSlot(IRuntimeMethodInfo method)
{
Debug.Assert(method != null);
- int slot = RuntimeMethodHandle.GetSlot(method.Value);
+ int slot = GetSlot(method.Value);
GC.KeepAlive(method);
return slot;
}
@@ -910,7 +910,7 @@ internal static int GetSlot(IRuntimeMethodInfo method)
internal static string GetName(IRuntimeMethodInfo method)
{
- string name = RuntimeMethodHandle.GetName(method.Value);
+ string name = GetName(method.Value);
GC.KeepAlive(method);
return name;
}
@@ -965,7 +965,7 @@ internal static Type[] GetMethodInstantiationPublic(IRuntimeMethodInfo method)
internal static bool HasMethodInstantiation(IRuntimeMethodInfo method)
{
- bool fRet = RuntimeMethodHandle.HasMethodInstantiation(method.Value);
+ bool fRet = HasMethodInstantiation(method.Value);
GC.KeepAlive(method);
return fRet;
}
@@ -981,7 +981,7 @@ internal static bool HasMethodInstantiation(IRuntimeMethodInfo method)
internal static bool IsGenericMethodDefinition(IRuntimeMethodInfo method)
{
- bool fRet = RuntimeMethodHandle.IsGenericMethodDefinition(method.Value);
+ bool fRet = IsGenericMethodDefinition(method.Value);
GC.KeepAlive(method);
return fRet;
}
@@ -1157,7 +1157,7 @@ public bool Equals(RuntimeFieldHandle handle)
public static RuntimeFieldHandle FromIntPtr(IntPtr value)
{
var handle = new RuntimeFieldHandleInternal(value);
- var fieldInfo = new RuntimeFieldInfoStub(handle, RuntimeFieldHandle.GetLoaderAllocator(handle));
+ var fieldInfo = new RuntimeFieldInfoStub(handle, GetLoaderAllocator(handle));
return new RuntimeFieldHandle(fieldInfo);
}
@@ -1275,7 +1275,7 @@ public bool Equals(ModuleHandle handle)
public static bool operator !=(ModuleHandle left, ModuleHandle right) => !left.Equals(right);
[MethodImpl(MethodImplOptions.InternalCall)]
- internal static extern IRuntimeMethodInfo GetDynamicMethod(System.Reflection.Emit.DynamicMethod method, RuntimeModule module, string name, byte[] sig, Resolver resolver);
+ internal static extern IRuntimeMethodInfo GetDynamicMethod(Reflection.Emit.DynamicMethod method, RuntimeModule module, string name, byte[] sig, Resolver resolver);
[MethodImpl(MethodImplOptions.InternalCall)]
internal static extern int GetToken(RuntimeModule module);
diff --git a/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.ActivatorCache.cs b/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.ActivatorCache.cs
index abf4f4afa4410f..d82daa26f0ccf4 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.ActivatorCache.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.ActivatorCache.cs
@@ -11,7 +11,7 @@ internal sealed partial class RuntimeType
{
///
/// A cache which allows optimizing ,
- /// , and related APIs.
+ /// , and related APIs.
///
private sealed unsafe class ActivatorCache
{
diff --git a/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.CoreCLR.cs
index 872f9d08b1910e..12b85edffacb6f 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/RuntimeType.CoreCLR.cs
@@ -1114,7 +1114,7 @@ private RuntimeType[] PopulateNestedClasses(Filter filter)
{
nestedType = moduleHandle.ResolveTypeHandle(tkNestedClasses[i]).GetRuntimeType();
}
- catch (System.TypeLoadException)
+ catch (TypeLoadException)
{
// In a reflection emit scenario, we may have a token for a class which
// has not been baked and hence cannot be loaded.
@@ -1474,7 +1474,7 @@ private string ConstructName([NotNull] ref string? name, TypeNameFormatFlags for
private T[] GetMemberList(ref MemberInfoCache? m_cache, MemberListType listType, string? name, CacheType cacheType)
where T : MemberInfo
{
- MemberInfoCache existingCache = GetMemberCache(ref m_cache);
+ MemberInfoCache existingCache = GetMemberCache(ref m_cache);
return existingCache.GetMemberList(listType, name, cacheType);
}
@@ -1721,54 +1721,54 @@ internal MethodInfo GetGenericMethodInfo(RuntimeMethodHandleInternal genericMeth
internal RuntimeMethodInfo[] GetMethodList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_methodInfoCache, listType, name, CacheType.Method);
+ return GetMemberList(ref m_methodInfoCache, listType, name, CacheType.Method);
}
internal RuntimeConstructorInfo[] GetConstructorList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_constructorInfoCache, listType, name, CacheType.Constructor);
+ return GetMemberList(ref m_constructorInfoCache, listType, name, CacheType.Constructor);
}
internal RuntimePropertyInfo[] GetPropertyList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_propertyInfoCache, listType, name, CacheType.Property);
+ return GetMemberList(ref m_propertyInfoCache, listType, name, CacheType.Property);
}
internal RuntimeEventInfo[] GetEventList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_eventInfoCache, listType, name, CacheType.Event);
+ return GetMemberList(ref m_eventInfoCache, listType, name, CacheType.Event);
}
internal RuntimeFieldInfo[] GetFieldList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_fieldInfoCache, listType, name, CacheType.Field);
+ return GetMemberList(ref m_fieldInfoCache, listType, name, CacheType.Field);
}
internal RuntimeType[] GetInterfaceList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_interfaceCache, listType, name, CacheType.Interface);
+ return GetMemberList(ref m_interfaceCache, listType, name, CacheType.Interface);
}
internal RuntimeType[] GetNestedTypeList(MemberListType listType, string? name)
{
- return GetMemberList(ref m_nestedClassesCache, listType, name, CacheType.NestedType);
+ return GetMemberList(ref m_nestedClassesCache, listType, name, CacheType.NestedType);
}
internal MethodBase GetMethod(RuntimeType declaringType, RuntimeMethodHandleInternal method)
{
- GetMemberCache(ref m_methodInfoCache);
+ GetMemberCache(ref m_methodInfoCache);
return m_methodInfoCache!.AddMethod(declaringType, method, CacheType.Method);
}
internal MethodBase GetConstructor(RuntimeType declaringType, RuntimeMethodHandleInternal constructor)
{
- GetMemberCache(ref m_constructorInfoCache);
+ GetMemberCache(ref m_constructorInfoCache);
return m_constructorInfoCache!.AddMethod(declaringType, constructor, CacheType.Constructor);
}
internal FieldInfo GetField(RuntimeFieldHandleInternal field)
{
- GetMemberCache(ref m_fieldInfoCache);
+ GetMemberCache(ref m_fieldInfoCache);
return m_fieldInfoCache!.AddField(field);
}
@@ -2395,7 +2395,7 @@ private static bool FilterApplyMethodBase(
private IntPtr m_cache;
internal IntPtr m_handle;
- internal static readonly RuntimeType ValueType = (RuntimeType)typeof(System.ValueType);
+ internal static readonly RuntimeType ValueType = (RuntimeType)typeof(ValueType);
private static readonly RuntimeType ObjectType = (RuntimeType)typeof(object);
private static readonly RuntimeType StringType = (RuntimeType)typeof(string);
@@ -2822,9 +2822,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
{
MethodInfo methodInfo = candidates[j];
if (!System.DefaultBinder.CompareMethodSig(methodInfo, firstCandidate))
- {
- throw new AmbiguousMatchException();
- }
+ throw ThrowHelper.GetAmbiguousMatchException(firstCandidate);
}
// All the methods have the exact same name and sig so return the most derived one.
@@ -2878,10 +2876,10 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
if (types == null || types.Length == 0)
{
// no arguments
+ PropertyInfo firstCandidate = candidates[0];
+
if (candidates.Count == 1)
{
- PropertyInfo firstCandidate = candidates[0];
-
if (returnType is not null && !returnType.IsEquivalentTo(firstCandidate.PropertyType))
return null;
@@ -2891,7 +2889,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
{
if (returnType is null)
// if we are here we have no args or property type to select over and we have more than one property with that name
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(firstCandidate);
}
}
@@ -2920,7 +2918,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
if ((bindingAttr & eventInfo.BindingFlags) == eventInfo.BindingFlags)
{
if (match != null)
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
match = eventInfo;
}
@@ -2950,7 +2948,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
if (match != null)
{
if (ReferenceEquals(fieldInfo.DeclaringType, match.DeclaringType))
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
if ((match.DeclaringType!.IsInterface) && (fieldInfo.DeclaringType!.IsInterface))
multipleStaticFieldMatches = true;
@@ -2962,7 +2960,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
}
if (multipleStaticFieldMatches && match!.DeclaringType!.IsInterface)
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
return match;
}
@@ -2998,7 +2996,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
if (FilterApplyType(iface, bindingAttr, name, false, ns))
{
if (match != null)
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
match = iface;
}
@@ -3027,7 +3025,7 @@ public override InterfaceMapping GetInterfaceMap([DynamicallyAccessedMembers(Dyn
if (FilterApplyType(nestedType, bindingAttr, name, false, ns))
{
if (match != null)
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
match = nestedType;
}
@@ -3395,7 +3393,7 @@ public override unsafe bool IsEnum
if (th.IsTypeDesc)
return IsSubclassOf(typeof(Enum));
- bool isEnum = th.AsMethodTable()->ParentMethodTable == System.Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
+ bool isEnum = th.AsMethodTable()->ParentMethodTable == Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
GC.KeepAlive(this);
return isEnum;
}
@@ -3409,7 +3407,7 @@ internal unsafe bool IsActualEnum
{
TypeHandle th = GetNativeTypeHandle();
- bool isEnum = !th.IsTypeDesc && th.AsMethodTable()->ParentMethodTable == System.Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
+ bool isEnum = !th.IsTypeDesc && th.AsMethodTable()->ParentMethodTable == Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
GC.KeepAlive(this);
return isEnum;
}
@@ -3419,7 +3417,7 @@ internal unsafe bool IsDelegate()
{
TypeHandle th = GetNativeTypeHandle();
- bool isDelegate = !th.IsTypeDesc && th.AsMethodTable()->ParentMethodTable == System.Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
+ bool isDelegate = !th.IsTypeDesc && th.AsMethodTable()->ParentMethodTable == Runtime.CompilerServices.TypeHandle.TypeHandleOf().AsMethodTable();
GC.KeepAlive(this);
return isDelegate;
}
@@ -3501,7 +3499,7 @@ internal RuntimeType[] GetGenericArgumentsInternal()
public override Type[] GetGenericArguments()
{
Type[] types = GetRootElementType().TypeHandle.GetInstantiationPublic();
- return types ?? Type.EmptyTypes;
+ return types ?? EmptyTypes;
}
[RequiresUnreferencedCode("If some of the generic arguments are annotated (either with DynamicallyAccessedMembersAttribute, or generic constraints), trimming can't validate that the requirements of those annotations are met.")]
@@ -3559,7 +3557,7 @@ public override Type MakeGenericType(Type[] instantiation)
if (foundSigType)
return new SignatureConstructedGenericType(this, instantiation);
- return System.Reflection.Emit.TypeBuilderInstantiation.MakeGenericType(this, (Type[])(instantiation.Clone()));
+ return Reflection.Emit.TypeBuilderInstantiation.MakeGenericType(this, (Type[])(instantiation.Clone()));
}
SanityCheckGenericArguments(instantiationRuntimeType, genericParameters);
@@ -3598,7 +3596,7 @@ public override Type[] GetGenericParameterConstraints()
throw new InvalidOperationException(SR.Arg_NotGenericParameter);
Type[] constraints = new RuntimeTypeHandle(this).GetConstraints();
- return constraints ?? Type.EmptyTypes;
+ return constraints ?? EmptyTypes;
}
#endregion
diff --git a/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs b/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs
index 1ef0563e7ebd06..431c2f724a02ba 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs
@@ -1294,7 +1294,7 @@ internal static void CheckStringLength(uint length)
#endif
[Intrinsic]
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
+ [MethodImpl(MethodImplOptions.InternalCall)]
internal static extern IntPtr NextCallReturnAddress();
} // class StubHelpers
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Text/StringBuilder.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Text/StringBuilder.CoreCLR.cs
index 40973d0e30e950..4988ae32548518 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Text/StringBuilder.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Text/StringBuilder.CoreCLR.cs
@@ -12,7 +12,7 @@ private int GetReplaceBufferCapacity(int requiredCapacity)
{
// This function assumes that required capacity will be less
// than the max capacity of the StringBuilder
- Diagnostics.Debug.Assert(requiredCapacity <= m_MaxCapacity);
+ Debug.Assert(requiredCapacity <= m_MaxCapacity);
int newCapacity = Capacity;
// Round the current capacity to the nearest multiple of 2
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Threading/Interlocked.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Threading/Interlocked.CoreCLR.cs
index 2e683dcf73c8ac..6ee5eb20c8f646 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Threading/Interlocked.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Threading/Interlocked.CoreCLR.cs
@@ -60,22 +60,6 @@ public static long Decrement(ref long location) =>
[MethodImpl(MethodImplOptions.InternalCall)]
public static extern long Exchange(ref long location1, long value);
- /// Sets a single-precision floating point number to a specified value and returns the original value, as an atomic operation.
- /// The variable to set to the specified value.
- /// The value to which the parameter is set.
- /// The original value of .
- /// The address of location1 is a null pointer.
- [MethodImpl(MethodImplOptions.InternalCall)]
- public static extern float Exchange(ref float location1, float value);
-
- /// Sets a double-precision floating point number to a specified value and returns the original value, as an atomic operation.
- /// The variable to set to the specified value.
- /// The value to which the parameter is set.
- /// The original value of .
- /// The address of location1 is a null pointer.
- [MethodImpl(MethodImplOptions.InternalCall)]
- public static extern double Exchange(ref double location1, double value);
-
/// Sets an object to the specified value and returns a reference to the original object, as an atomic operation.
/// The variable to set to the specified value.
/// The value to which the parameter is set.
@@ -122,24 +106,6 @@ public static T Exchange([NotNullIfNotNull(nameof(value))] ref T location1, T
[MethodImpl(MethodImplOptions.InternalCall)]
public static extern long CompareExchange(ref long location1, long value, long comparand);
- /// Compares two single-precision floating point numbers for equality and, if they are equal, replaces the first value.
- /// The destination, whose value is compared with and possibly replaced.
- /// The value that replaces the destination value if the comparison results in equality.
- /// The value that is compared to the value at .
- /// The original value in .
- /// The address of is a null pointer.
- [MethodImpl(MethodImplOptions.InternalCall)]
- public static extern float CompareExchange(ref float location1, float value, float comparand);
-
- /// Compares two double-precision floating point numbers for equality and, if they are equal, replaces the first value.
- /// The destination, whose value is compared with and possibly replaced.
- /// The value that replaces the destination value if the comparison results in equality.
- /// The value that is compared to the value at .
- /// The original value in .
- /// The address of is a null pointer.
- [MethodImpl(MethodImplOptions.InternalCall)]
- public static extern double CompareExchange(ref double location1, double value, double comparand);
-
/// Compares two objects for reference equality and, if they are equal, replaces the first object.
/// The destination object that is compared by reference with and possibly replaced.
/// The object that replaces the destination object if the reference comparison results in equality.
@@ -208,25 +174,7 @@ public static long Read(ref long location) =>
CompareExchange(ref location, 0, 0);
#endregion
- #region MemoryBarrier
- ///
- /// Synchronizes memory access as follows:
- /// The processor that executes the current thread cannot reorder instructions in such a way that memory accesses before
- /// the call to execute after memory accesses that follow the call to .
- ///
- [Intrinsic]
- [MethodImpl(MethodImplOptions.InternalCall)]
- public static extern void MemoryBarrier();
-
- ///
- /// Synchronizes memory access as follows:
- /// The processor that executes the current thread cannot reorder instructions in such a way that memory reads before
- /// the call to execute after memory accesses that follow the call to .
- ///
- [Intrinsic]
- [MethodImpl(MethodImplOptions.InternalCall)]
- internal static extern void ReadMemoryBarrier();
-
+ #region MemoryBarrierProcessWide
[LibraryImport(RuntimeHelpers.QCall, EntryPoint = "Interlocked_MemoryBarrierProcessWide")]
private static partial void _MemoryBarrierProcessWide();
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs
index 7a52db2be2de64..64ba59f672ec09 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs
@@ -248,7 +248,7 @@ private bool SetApartmentStateUnchecked(ApartmentState state, bool throwOnError)
// Once we CoUninitialize the thread, the OS will still
// report the thread as implicitly in the MTA if any
// other thread in the process is CoInitialized.
- if ((state == System.Threading.ApartmentState.Unknown) && (retState == System.Threading.ApartmentState.MTA))
+ if ((state == ApartmentState.Unknown) && (retState == ApartmentState.MTA))
{
return true;
}
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Type.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Type.CoreCLR.cs
index fe4b8622b56336..b858949e73eada 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Type.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Type.CoreCLR.cs
@@ -5,6 +5,7 @@
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.Versioning;
+using System.Security;
using StackCrawlMark = System.Threading.StackCrawlMark;
namespace System
@@ -22,7 +23,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(string typeName, bool throwOnError, bool ignoreCase)
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
@@ -31,7 +32,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(string typeName, bool throwOnError)
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
@@ -40,7 +41,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(string typeName)
{
StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
@@ -48,7 +49,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(
string typeName,
Func? assemblyResolver,
@@ -60,7 +61,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(
string typeName,
Func? assemblyResolver,
@@ -74,7 +75,7 @@ public bool IsInterface
}
[RequiresUnreferencedCode("The type might be removed")]
- [System.Security.DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
+ [DynamicSecurityMethod] // Methods containing StackCrawlMark local var has to be marked DynamicSecurityMethod
public static Type? GetType(
string typeName,
Func? assemblyResolver,
diff --git a/src/coreclr/System.Private.CoreLib/src/System/TypedReference.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/TypedReference.CoreCLR.cs
index 46b9d2bd9fb2e5..25538c0a0b4a39 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/TypedReference.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/TypedReference.CoreCLR.cs
@@ -6,10 +6,11 @@
using System.Diagnostics;
using System.Runtime.CompilerServices;
+using System.Runtime.Versioning;
namespace System
{
- [System.Runtime.Versioning.NonVersionable] // This only applies to field layout
+ [NonVersionable] // This only applies to field layout
public ref partial struct TypedReference
{
private readonly ref byte _value;
diff --git a/src/coreclr/System.Private.CoreLib/src/System/ValueType.cs b/src/coreclr/System.Private.CoreLib/src/System/ValueType.cs
index 880de9ac6af953..cc13e37e083f01 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/ValueType.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/ValueType.cs
@@ -17,7 +17,7 @@
namespace System
{
[Serializable]
- [System.Runtime.CompilerServices.TypeForwardedFrom("mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089")]
+ [TypeForwardedFrom("mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089")]
public abstract class ValueType
{
[UnconditionalSuppressMessage("ReflectionAnalysis", "IL2075:UnrecognizedReflectionPattern",
diff --git a/src/coreclr/System.Private.CoreLib/src/System/Variant.cs b/src/coreclr/System.Private.CoreLib/src/System/Variant.cs
index 9b7f27f8d10929..0f55a7ef6a0224 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Variant.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Variant.cs
@@ -67,8 +67,8 @@ internal struct Variant
internal const int ArrayBitMask = 0x10000;
internal static Variant Empty => default;
- internal static Variant Missing => new Variant(Variant.CV_MISSING, Type.Missing, 0);
- internal static Variant DBNull => new Variant(Variant.CV_NULL, System.DBNull.Value, 0);
+ internal static Variant Missing => new Variant(CV_MISSING, Type.Missing, 0);
+ internal static Variant DBNull => new Variant(CV_NULL, System.DBNull.Value, 0);
//
// Native Methods
diff --git a/src/coreclr/clrdefinitions.cmake b/src/coreclr/clrdefinitions.cmake
index b5a43f5057a47b..172d68e8964777 100644
--- a/src/coreclr/clrdefinitions.cmake
+++ b/src/coreclr/clrdefinitions.cmake
@@ -185,9 +185,9 @@ endif(FEATURE_ENABLE_NO_ADDRESS_SPACE_RANDOMIZATION)
add_definitions(-DFEATURE_SVR_GC)
add_definitions(-DFEATURE_SYMDIFF)
add_compile_definitions(FEATURE_TIERED_COMPILATION)
-if (CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_ARM64)
+if (CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_ARM64 OR CLR_CMAKE_TARGET_ARCH_LOONGARCH64)
add_compile_definitions(FEATURE_ON_STACK_REPLACEMENT)
-endif (CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_ARM64)
+endif (CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_ARM64 OR CLR_CMAKE_TARGET_ARCH_LOONGARCH64)
add_compile_definitions(FEATURE_PGO)
if (CLR_CMAKE_TARGET_WIN32)
add_definitions(-DFEATURE_TYPEEQUIVALENCE)
diff --git a/src/coreclr/crossgen-corelib.proj b/src/coreclr/crossgen-corelib.proj
index 0ee3d5185c1934..747864a0047a63 100644
--- a/src/coreclr/crossgen-corelib.proj
+++ b/src/coreclr/crossgen-corelib.proj
@@ -122,7 +122,7 @@
-
+
diff --git a/src/coreclr/dlls/mscorpe/ceefilegenwriter.cpp b/src/coreclr/dlls/mscorpe/ceefilegenwriter.cpp
index e6135077675039..b6cc342c741d80 100644
--- a/src/coreclr/dlls/mscorpe/ceefilegenwriter.cpp
+++ b/src/coreclr/dlls/mscorpe/ceefilegenwriter.cpp
@@ -180,24 +180,11 @@ inline IMAGE_SYMBOL* GetSymbolEntry(IMAGE_SYMBOL* pHead, SIZE_T idx)
}
//*****************************************************************************
-// To get a new instance, call CreateNewInstance() or CreateNewInstanceEx() instead of new
+// To get a new instance, call CreateNewInstance() instead of new
//*****************************************************************************
-HRESULT CeeFileGenWriter::CreateNewInstance(CCeeGen *pCeeFileGenFrom,
- CeeFileGenWriter* & pGenWriter,
+HRESULT CeeFileGenWriter::CreateNewInstance(CeeFileGenWriter* & pGenWriter,
DWORD createFlags)
-{
- return CreateNewInstanceEx(pCeeFileGenFrom, pGenWriter, createFlags);
-}
-
-//
-// Seed file is used as the base file. The new file data will be "appended" to the seed file
-//
-
-HRESULT CeeFileGenWriter::CreateNewInstanceEx(CCeeGen *pCeeFileGenFrom,
- CeeFileGenWriter* & pGenWriter,
- DWORD createFlags,
- LPCWSTR seedFileName)
{
HRESULT hr = S_OK;
ULONG preallocatedOffset = 0;
@@ -213,10 +200,7 @@ HRESULT CeeFileGenWriter::CreateNewInstanceEx(CCeeGen *pCeeFileGenFrom,
if (pPEWriter == NULL)
IfFailGo(E_OUTOFMEMORY);
- //workaround
- //What's really the correct thing to be doing here?
- //HRESULT hr = pPEWriter->Init(pCeeFileGenFrom ? pCeeFileGenFrom->getPESectionMan() : NULL);
- hr = pPEWriter->Init(NULL, createFlags, seedFileName);
+ hr = pPEWriter->Init(NULL, createFlags);
IfFailGo(hr);
//Create the general PEWriter.
@@ -224,16 +208,13 @@ HRESULT CeeFileGenWriter::CreateNewInstanceEx(CCeeGen *pCeeFileGenFrom,
hr = pPrivateGenWriter->Init(); // base class member to finish init
IfFailGo(hr);
- if (!seedFileName) // Use base file's preferred base (if present)
+ if (pPEWriter->isPE32())
{
- if (pPEWriter->isPE32())
- {
- pPrivateGenWriter->setImageBase((DWORD) CEE_IMAGE_BASE_32); // use same default as linker
- }
- else
- {
- pPrivateGenWriter->setImageBase64((ULONGLONG) CEE_IMAGE_BASE_64); // use same default as linker
- }
+ pPrivateGenWriter->setImageBase((DWORD) CEE_IMAGE_BASE_32); // use same default as linker
+ }
+ else
+ {
+ pPrivateGenWriter->setImageBase64((ULONGLONG) CEE_IMAGE_BASE_64); // use same default as linker
}
pPrivateGenWriter->setSubsystem(IMAGE_SUBSYSTEM_WINDOWS_CUI, CEE_IMAGE_SUBSYSTEM_MAJOR_VERSION, CEE_IMAGE_SUBSYSTEM_MINOR_VERSION);
@@ -247,11 +228,6 @@ HRESULT CeeFileGenWriter::CreateNewInstanceEx(CCeeGen *pCeeFileGenFrom,
hr = pPrivateGenWriter->allocateCorHeader(); // get COR header near front
IfFailGo(hr);
- //If we were passed a CCeeGen at the beginning, copy it's data now.
- if (pCeeFileGenFrom) {
- pCeeFileGenFrom->cloneInstance((CCeeGen*)pPrivateGenWriter);
- }
-
hr = pPrivateGenWriter->getSectionCreate(".text0", sdExecute, &corHeaderSection);
IfFailGo(hr);
preallocatedOffset = corHeaderSection->dataLen();
diff --git a/src/coreclr/dlls/mscorpe/iceefilegen.cpp b/src/coreclr/dlls/mscorpe/iceefilegen.cpp
index 6c40d9699fce91..152fe9b3fcaf91 100644
--- a/src/coreclr/dlls/mscorpe/iceefilegen.cpp
+++ b/src/coreclr/dlls/mscorpe/iceefilegen.cpp
@@ -34,42 +34,19 @@ HRESULT ICeeFileGen::CreateCeeFile (HCEEFILE *ceeFile)
}
HRESULT ICeeFileGen::CreateCeeFileEx (HCEEFILE *ceeFile, DWORD createFlags)
-{
- return CreateCeeFileEx2(ceeFile, createFlags, NULL);
-}
-
-//
-// Seed file is used as the base file. The new file data will be "appended" to the seed file
-//
-
-HRESULT ICeeFileGen::CreateCeeFileEx2 (HCEEFILE *ceeFile, DWORD createFlags, LPCWSTR seedFileName)
{
if (!ceeFile)
return E_POINTER;
CeeFileGenWriter *gen = NULL;
HRESULT hr;
- IfFailRet(CeeFileGenWriter::CreateNewInstanceEx(NULL, gen, createFlags, seedFileName));
+ IfFailRet(CeeFileGenWriter::CreateNewInstance(gen, createFlags));
TESTANDRETURN(gen != NULL, E_OUTOFMEMORY);
*ceeFile = gen;
return S_OK;
}
-HRESULT ICeeFileGen::CreateCeeFileFromICeeGen(ICeeGenInternal *pICeeGen, HCEEFILE *ceeFile, DWORD createFlags)
-{
- if (!ceeFile)
- return E_POINTER;
- CCeeGen *genFrom = reinterpret_cast(pICeeGen);
- CeeFileGenWriter *gen = NULL;
- HRESULT hr = CeeFileGenWriter::CreateNewInstance(genFrom, gen, createFlags);
- if (FAILED(hr))
- return hr;
- TESTANDRETURN(gen != NULL, E_OUTOFMEMORY);
- *ceeFile = gen;
- return S_OK;
-}
-
HRESULT ICeeFileGen::DestroyCeeFile(HCEEFILE *ceeFile)
{
if (!ceeFile)
@@ -262,12 +239,6 @@ HRESULT ICeeFileGen::SetSubsystem(HCEEFILE ceeFile, DWORD subsystem, DWORD major
return S_OK;
}
-HRESULT ICeeFileGen::GetIMapTokenIface(HCEEFILE ceeFile, IMetaDataEmit *emitter, IUnknown **pIMapToken)
-{
- _ASSERTE(!"This is an obsolete function!");
- return E_NOTIMPL;
-}
-
HRESULT ICeeFileGen::GetMethodRVA(HCEEFILE ceeFile, ULONG codeOffset, ULONG *codeRVA)
{
TESTANDRETURNARG(ceeFile != 0);
@@ -394,25 +365,6 @@ HRESULT ICeeFileGen::EmitMetaDataAt (HCEEFILE ceeFile, IMetaDataEmit *emitter, H
return(gen->emitMetaData(emitter, sec, offset, buffer, buffLen));
}
-HRESULT ICeeFileGen::GetIMapTokenIfaceEx(HCEEFILE ceeFile, IMetaDataEmit *emitter, IUnknown **pIMapToken)
-{
- TESTANDRETURNPOINTER(ceeFile);
- TESTANDRETURNPOINTER(pIMapToken);
-
- CeeFileGenWriter *gen = reinterpret_cast(ceeFile);
- return gen->getMapTokenIface(pIMapToken);
-}
-
-HRESULT ICeeFileGen::AddNotificationHandler(HCEEFILE ceeFile,
- IUnknown *pHandler)
-{
- TESTANDRETURNPOINTER(ceeFile);
- TESTANDRETURNPOINTER(pHandler);
-
- CeeFileGenWriter *gen = reinterpret_cast(ceeFile);
- return gen->addNotificationHandler(pHandler);
-}
-
HRESULT ICeeFileGen::SetManifestEntry(HCEEFILE ceeFile, ULONG size, ULONG offset)
{
TESTANDRETURNPOINTER(ceeFile);
diff --git a/src/coreclr/dlls/mscorpe/pewriter.cpp b/src/coreclr/dlls/mscorpe/pewriter.cpp
index 2f40d3cd5b5939..a9f7a8fddf593e 100644
--- a/src/coreclr/dlls/mscorpe/pewriter.cpp
+++ b/src/coreclr/dlls/mscorpe/pewriter.cpp
@@ -686,41 +686,7 @@ HRESULT PEWriterSection::applyRelocs(IMAGE_NT_HEADERS * pNtHeaders,
}
/******************************************************************/
-
-PESeedSection::PESeedSection(PEDecoder * peDecoder,
- IMAGE_SECTION_HEADER * seedSection)
- : PEWriterSection((const char *)seedSection->Name,
- VAL32(seedSection->Characteristics),
- VAL32(seedSection->SizeOfRawData),
- 0),
- m_pSeedFileDecoder(peDecoder),
- m_pSeedSectionHeader(seedSection)
-{
- m_baseRVA = VAL32(seedSection->VirtualAddress);
-}
-
-HRESULT PESeedSection::write(HANDLE file) {
- ULONG sizeOfSection = VAL32(m_pSeedSectionHeader->SizeOfRawData);
- LPCVOID sectionData = PBYTE(m_pSeedFileDecoder->GetBase()) + m_pSeedSectionHeader->PointerToRawData;
-
- DWORD dwWritten = 0;
- if (!WriteFile(file, sectionData, sizeOfSection, &dwWritten, NULL)) {
- return HRESULT_FROM_GetLastError();
- }
- _ASSERTE(dwWritten == sizeOfSection);
- return S_OK;
-}
-
-unsigned PESeedSection::writeMem(void ** pMem) {
- ULONG sizeOfSection = VAL32(m_pSeedSectionHeader->SizeOfRawData);
- LPCVOID sectionData = PBYTE(m_pSeedFileDecoder->GetBase()) + m_pSeedSectionHeader->PointerToRawData;
-
- COPY_AND_ADVANCE(*pMem, sectionData, sizeOfSection);
- return sizeOfSection;
-}
-
-/******************************************************************/
-HRESULT PEWriter::Init(PESectionMan *pFrom, DWORD createFlags, LPCWSTR seedFileName)
+HRESULT PEWriter::Init(PESectionMan *pFrom, DWORD createFlags)
{
if (pFrom)
*(PESectionMan*)this = *pFrom;
@@ -878,103 +844,12 @@ HRESULT PEWriter::Init(PESectionMan *pFrom, DWORD createFlags, LPCWSTR seedFileN
m_file = INVALID_HANDLE_VALUE;
- //
- // Seed file
- //
-
- m_hSeedFile = INVALID_HANDLE_VALUE;
- m_hSeedFileMap = INVALID_HANDLE_VALUE;
- m_pSeedFileDecoder = NULL;
- m_iSeedSections = 0;
- m_pSeedSectionToAdd = NULL;
-
- if (seedFileName)
- {
- HandleHolder hFile (WszCreateFile(seedFileName,
- GENERIC_READ,
- FILE_SHARE_READ,
- NULL,
- OPEN_EXISTING,
- FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN,
- NULL));
-
- if (hFile == INVALID_HANDLE_VALUE)
- return HRESULT_FROM_GetLastError();
-
- MapViewHolder hMapFile (WszCreateFileMapping(hFile, NULL, PAGE_READONLY, 0, 0, NULL));
- DWORD dwFileLen = SafeGetFileSize(hFile, 0);
- if (dwFileLen == 0xffffffff)
- return HRESULT_FROM_GetLastError();
-
- if (hMapFile == NULL)
- return HRESULT_FROM_GetLastError();
-
- BYTE * baseFileView = (BYTE*) MapViewOfFile(hMapFile, FILE_MAP_READ, 0, 0, 0);
-
- PEDecoder * pPEDecoder = new (nothrow) PEDecoder(baseFileView, (COUNT_T)dwFileLen);
- if (pPEDecoder == NULL) return E_OUTOFMEMORY;
-
- if (pPEDecoder->Has32BitNTHeaders())
- {
- if ((createFlags & ICEE_CREATE_FILE_PE32) == 0)
- return E_FAIL;
-
- setImageBase32(DWORD(size_t(pPEDecoder->GetPreferredBase())));
- }
- else
- {
- if ((createFlags & ICEE_CREATE_FILE_PE64) == 0)
- return E_FAIL;
-
- setImageBase64(UINT64((intptr_t) pPEDecoder->GetPreferredBase()));
- }
-
- setFileAlignment (VAL32(pPEDecoder->GetFileAlignment()));
- setSectionAlignment(VAL32(pPEDecoder->GetSectionAlignment()));
-
- hFile.SuppressRelease();
- hMapFile.SuppressRelease();
-
- m_hSeedFile = hFile;
- m_hSeedFileMap = hMapFile;
- m_pSeedFileDecoder = pPEDecoder;
-
-#ifdef HOST_64BIT
- m_pSeedFileNTHeaders = pPEDecoder->GetNTHeaders64();
-#else
- m_pSeedFileNTHeaders = pPEDecoder->GetNTHeaders32();
-#endif
-
- // Add the seed sections
-
- m_pSeedSections = m_pSeedFileDecoder->FindFirstSection();
-
- m_pSeedSectionToAdd = m_pSeedSections;
- m_iSeedSections = m_pSeedFileDecoder->GetNumberOfSections();
-
- for (unsigned i = 0; i < m_iSeedSections; m_pSeedSectionToAdd++, i++) {
- PESection * dummy;
- getSectionCreate((const char *)(m_pSeedSectionToAdd->Name),
- VAL32(m_pSeedSectionToAdd->Characteristics),
- &dummy);
- }
-
- m_pSeedSectionToAdd = NULL;
- }
-
return S_OK;
}
/******************************************************************/
HRESULT PEWriter::Cleanup() {
- if (m_hSeedFile != INVALID_HANDLE_VALUE)
- {
- CloseHandle(m_hSeedFile);
- CloseHandle(m_hSeedFileMap);
- delete m_pSeedFileDecoder;
- }
-
if (isPE32())
{
delete ntHeaders32();
@@ -993,43 +868,10 @@ HRESULT PEWriter::Cleanup() {
return PESectionMan::Cleanup();
}
-PESection* PEWriter::getSection(const char* name)
-{
- int len = (int)strlen(name);
-
- // the section name can be at most 8 characters including the null.
- if (len < 8)
- len++;
- else
- len = 8;
-
- // dbPrintf(("looking for section %s\n", name));
- // Skip over the seed sections
-
- for(PESection** cur = sectStart+m_iSeedSections; cur < sectCur; cur++) {
- // dbPrintf(("searching section %s\n", (*cur)->m_ame));
- if (strncmp((*cur)->m_name, name, len) == 0) {
- // dbPrintf(("found section %s\n", (*cur)->m_name));
- return(*cur);
- }
- }
- return(0);
-}
-
HRESULT PEWriter::newSection(const char* name, PESection **section,
unsigned flags, unsigned estSize,
unsigned estRelocs)
{
- if (m_pSeedSectionToAdd) {
- _ASSERTE(strcmp((const char *)(m_pSeedSectionToAdd->Name), name) == 0 &&
- VAL32(m_pSeedSectionToAdd->Characteristics) == flags);
-
- PESeedSection * ret = new (nothrow) PESeedSection(m_pSeedFileDecoder, m_pSeedSectionToAdd);
- *section = ret;
- TESTANDRETURNMEMORY(ret);
- return S_OK;
- }
-
PEWriterSection * ret = new (nothrow) PEWriterSection(name, flags, estSize, estRelocs);
*section = ret;
TESTANDRETURNMEMORY(ret);
@@ -1165,15 +1007,13 @@ class SectionNameSorter : protected CQuickSort
entry * m_entries;
PEWriterSection ** m_sections;
unsigned m_count;
- unsigned m_seedCount;
public:
- SectionNameSorter(entry *entries, PEWriterSection ** sections, int count, unsigned seedSections)
+ SectionNameSorter(entry *entries, PEWriterSection ** sections, int count)
: CQuickSort(entries, count),
m_entries(entries),
m_sections(sections),
- m_count(unsigned(count)),
- m_seedCount(seedSections)
+ m_count(unsigned(count))
{}
// Sorts the entries according to alphabetical + numerical order
@@ -1183,16 +1023,6 @@ class SectionNameSorter : protected CQuickSort
PEWriterSection * firstSection = m_sections[first->arrayIndex];
PEWriterSection * secondSection = m_sections[second->arrayIndex];
- // Seed sections are always at the start, in the order they were
- // added to the PEWriter
-
- if (firstSection->isSeedSection() || secondSection->isSeedSection()) {
- if (firstSection->isSeedSection() && secondSection->isSeedSection())
- return first->arrayIndex - second->arrayIndex;
-
- return firstSection->isSeedSection() ? -1 : 1;
- }
-
// Sort the names
int lenDiff = first->nameLength - second->nameLength;
@@ -1225,11 +1055,7 @@ class SectionNameSorter : protected CQuickSort
for (unsigned i = 1; i < m_count; i++, ePrev = e, e++) {
- // Seed sections should stay at the front
- _ASSERTE(i >= m_seedCount || i == e->arrayIndex);
-
- if (!m_sections[ePrev->arrayIndex]->isSeedSection() &&
- (ePrev->nameLength == e->nameLength) &&
+ if ((ePrev->nameLength == e->nameLength) &&
strncmp(ePrev->name, e->name, e->nameLength) == 0)
{
continue;
@@ -1317,7 +1143,7 @@ HRESULT PEWriter::linkSortSections(entry * entries,
// Sort the entries according to alphabetical + numerical order
//
- SectionNameSorter sorter(entries, getSectStart(), int(e - entries), m_iSeedSections);
+ SectionNameSorter sorter(entries, getSectStart(), int(e - entries));
*piUniqueSections = sorter.SortSections();
*piEntries = unsigned(e - entries);
@@ -1367,7 +1193,6 @@ HRESULT PEWriter::linkSortHeaders(entry * entries, unsigned iEntries, unsigned i
for (entry * e = entries ; e < entriesEnd; e++)
{
if (ePrev != NULL
- && !getSectStart()[ePrev->arrayIndex]->isSeedSection()
&& e->nameLength == ePrev->nameLength
&& strncmp(e->name, ePrev->name, e->nameLength) == 0)
{
@@ -1433,10 +1258,6 @@ HRESULT PEWriter::linkPlaceSections(entry * entries, unsigned iEntries)
entry * e = entries + VAL32(h->FirstEntryIndex);
PEWriterSection *s = getSectStart()[e->arrayIndex];
- if (s->isSeedSection()) {
- virtualPos = s->getBaseRVA();
- }
-
h->VirtualAddress = VAL32(virtualPos);
h->PointerToRawData = VAL32(filePos);
@@ -1501,11 +1322,6 @@ HRESULT PEWriter::linkPlaceSections(entry * entries, unsigned iEntries)
void PEWriter::setSectionIndex(IMAGE_SECTION_HEADER * h, unsigned sectionIndex) {
- if (getSectStart()[sectionIndex]->isSeedSection()) {
- h->SectionIndex = VAL32(sectionIndex);
- return;
- }
-
//
// Reserve some dummy "array index" values for special sections
// at the start of the image (after the seed sections)
@@ -1523,7 +1339,7 @@ void PEWriter::setSectionIndex(IMAGE_SECTION_HEADER * h, unsigned sectionIndex)
}
else if (strcmp((char *) h->Name, *s) == 0)
{
- h->SectionIndex = VAL32(m_iSeedSections + DWORD(s - SpecialNames));
+ h->SectionIndex = VAL32(DWORD(s - SpecialNames));
break;
}
}
@@ -1598,17 +1414,6 @@ HRESULT PEWriter::link() {
virtualPos = roundUp(filePos, VAL32(m_ntHeaders->OptionalHeader.SectionAlignment));
- if (m_hSeedFile != INVALID_HANDLE_VALUE) {
- // We do not support relocating/sliding down the seed sections
- if (filePos > VAL32(m_pSeedSections->VirtualAddress) ||
- virtualPos > VAL32(m_pSeedSections->VirtualAddress))
- return E_FAIL;
-
- if (virtualPos < VAL32(m_pSeedSections->VirtualAddress)) {
- virtualPos = VAL32(m_pSeedSections->VirtualAddress);
- }
- }
-
// Now finally assign RVAs to the sections
IfFailRet(linkPlaceSections(entries, iEntries));
@@ -2014,8 +1819,6 @@ UINT64 PEWriter::getImageBase64()
void PEWriter::setImageBase32(DWORD imageBase)
{
- _ASSERTE(m_hSeedFile == INVALID_HANDLE_VALUE);
-
_ASSERTE(isPE32());
ntHeaders32()->OptionalHeader.ImageBase = VAL32(imageBase);
}
diff --git a/src/coreclr/dlls/mscorpe/pewriter.h b/src/coreclr/dlls/mscorpe/pewriter.h
index 11bba04a71b3f0..089f624bdaa5bb 100644
--- a/src/coreclr/dlls/mscorpe/pewriter.h
+++ b/src/coreclr/dlls/mscorpe/pewriter.h
@@ -28,12 +28,9 @@ class PEWriter : public PESectionMan
public:
// See ICeeFileGen.h for definition of createFlags
- HRESULT Init(PESectionMan *pFrom, DWORD createFlags, LPCWSTR seedFileName = NULL);
+ HRESULT Init(PESectionMan *pFrom, DWORD createFlags);
HRESULT Cleanup();
- // Finds section with given name. returns 0 if not found
- virtual PESection* getSection(const char* name);
-
// Create a new section
virtual HRESULT newSection(const char* name, PESection **section,
unsigned flags=sdNone, unsigned estSize=0x10000,
@@ -118,17 +115,6 @@ class PEWriter : public PESectionMan
HANDLE m_file;
- // "Seed" file information. The new file data will be "appended" to the seed file
- // These are valid only if m_hSeedFile is valid
-
- HANDLE m_hSeedFile;
- HANDLE m_hSeedFileMap;
- PEDecoder * m_pSeedFileDecoder;
- IMAGE_NT_HEADERS * m_pSeedFileNTHeaders;
- unsigned m_iSeedSections;
- IMAGE_SECTION_HEADER*m_pSeedSections;
- IMAGE_SECTION_HEADER*m_pSeedSectionToAdd; // used only by newSection()
-
PEWriterSection **getSectStart() {
return (PEWriterSection**)sectStart;
}
@@ -221,73 +207,13 @@ class PEWriterSection : public PESection {
virtual HRESULT write (HANDLE file);
virtual unsigned writeMem (void ** pMem);
- virtual bool isSeedSection() { return false; }
-
-};
-
-// This is for sections from the seed file. Their order needs to be maintained and
-// they need to be written to the output file.
-
-class PESeedSection : public PEWriterSection {
-
-public:
-
- PESeedSection(PEDecoder * peDecoder, IMAGE_SECTION_HEADER * seedSection);
-
- // PESection methods
-
- unsigned dataLen() { return m_pSeedSectionHeader->SizeOfRawData; }
- HRESULT applyRelocs(CeeGenTokenMapper *pTokenMapper) { return S_OK; }
- char* getBlock(unsigned len, unsigned align) { _ASSERTE(!"PESeedSection"); return NULL; }
- HRESULT truncate(unsigned newLen) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
- void writeSectReloc(unsigned val, CeeSection& relativeTo,
- CeeSectionRelocType reloc,
- CeeSectionRelocExtra *extra) { _ASSERTE(!"PESeedSection"); return; }
- HRESULT addSectReloc(unsigned offset, CeeSection& relativeTo,
- CeeSectionRelocType reloc,
- CeeSectionRelocExtra *extra) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
- HRESULT addSectReloc(unsigned offset, PESection *relativeTo,
- CeeSectionRelocType reloc,
- CeeSectionRelocExtra *extra) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
- HRESULT addBaseReloc(unsigned offset, CeeSectionRelocType reloc,
- CeeSectionRelocExtra *extra) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
-// unsigned char *name();
-// unsigned flags();
-// unsigned getBaseRVA();
- int getDirEntry() { _ASSERTE(!"PESeedSection"); return 0; }
- HRESULT directoryEntry(unsigned num) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
- char * computePointer(unsigned offset) const { _ASSERTE(!"PESeedSection"); return NULL; }
- BOOL containsPointer(_In_ char *ptr) const { _ASSERTE(!"PESeedSection"); return FALSE; }
- unsigned computeOffset(_In_ char *ptr) const { _ASSERTE(!"PESeedSection"); return 0; }
- HRESULT cloneInstance(PESection *destination) { _ASSERTE(!"PESeedSection"); return E_FAIL; }
-
- // PEWriterSection
-
- HRESULT applyRelocs(IMAGE_NT_HEADERS * pNtHeaders,
- PERelocSection * relocSection,
- CeeGenTokenMapper * pTokenMapper,
- DWORD rdataRvaBase,
- DWORD dataRvaBase,
- DWORD textRvaBase) { return S_OK; }
-
- HRESULT write(HANDLE file);
- unsigned writeMem(void ** pMem);
- bool isSeedSection() { return true; }
-
-protected:
-
- PEDecoder * m_pSeedFileDecoder;
- IMAGE_SECTION_HEADER * m_pSeedSectionHeader;
-
};
inline DWORD PEWriter::getSectionAlignment() {
- return VAL32(m_ntHeaders->OptionalHeader.FileAlignment);
+ return VAL32(m_ntHeaders->OptionalHeader.SectionAlignment);
}
inline void PEWriter::setSectionAlignment(DWORD SectionAlignment) {
-
- _ASSERTE(m_hSeedFile == INVALID_HANDLE_VALUE);
m_ntHeaders->OptionalHeader.SectionAlignment = VAL32(SectionAlignment);
}
@@ -296,7 +222,6 @@ inline DWORD PEWriter::getFileAlignment() {
}
inline void PEWriter::setFileAlignment(DWORD fileAlignment) {
- _ASSERTE(m_hSeedFile == INVALID_HANDLE_VALUE);
m_ntHeaders->OptionalHeader.FileAlignment = VAL32(fileAlignment);
}
diff --git a/src/coreclr/gc/gc.cpp b/src/coreclr/gc/gc.cpp
index 5409d99a5380d9..453cff7b31e921 100644
--- a/src/coreclr/gc/gc.cpp
+++ b/src/coreclr/gc/gc.cpp
@@ -7841,7 +7841,7 @@ void gc_heap::fix_allocation_context_heaps (gc_alloc_context* gc_context, void*)
// make sure no allocation contexts point to idle heaps
void gc_heap::fix_allocation_contexts_heaps()
{
- GCToEEInterface::GcEnumAllocContexts (fix_allocation_context_heaps, nullptr);
+ GCToEEInterface::GcEnumAllocContexts (fix_allocation_context_heaps, nullptr);
}
#endif //MULTIPLE_HEAPS
@@ -10596,7 +10596,7 @@ static int __cdecl cmp_mark_list_item (const void* vkey, const void* vdatum)
#endif // _DEBUG
#ifdef USE_REGIONS
-uint8_t** gc_heap::get_region_mark_list (uint8_t* start, uint8_t* end, uint8_t*** mark_list_end_ptr)
+uint8_t** gc_heap::get_region_mark_list (BOOL& use_mark_list, uint8_t* start, uint8_t* end, uint8_t*** mark_list_end_ptr)
{
size_t region_number = get_basic_region_index_for_address (start);
size_t source_number = region_number;
@@ -10726,6 +10726,13 @@ void gc_heap::merge_mark_lists (size_t total_mark_list_size)
// blast this piece to the mark list
append_to_mark_list(source[lowest_source], x);
+#ifdef USE_REGIONS
+ if (mark_list_index > mark_list_end)
+ {
+ use_mark_list = false;
+ return nullptr;
+ }
+#endif //USE_REGIONS
piece_count++;
source[lowest_source] = x;
@@ -10745,6 +10752,13 @@ void gc_heap::merge_mark_lists (size_t total_mark_list_size)
}
// we're left with just one source that we copy
append_to_mark_list(source[0], source_end[0]);
+#ifdef USE_REGIONS
+ if (mark_list_index > mark_list_end)
+ {
+ use_mark_list = false;
+ return nullptr;
+ }
+#endif //USE_REGIONS
piece_count++;
}
@@ -10801,7 +10815,7 @@ static uint8_t** binary_search (uint8_t** left, uint8_t** right, uint8_t* e)
return a + l;
}
-uint8_t** gc_heap::get_region_mark_list (uint8_t* start, uint8_t* end, uint8_t*** mark_list_end_ptr)
+uint8_t** gc_heap::get_region_mark_list (BOOL& use_mark_list, uint8_t* start, uint8_t* end, uint8_t*** mark_list_end_ptr)
{
// do a binary search over the sorted marked list to find start and end of the
// mark list for this region
@@ -14170,7 +14184,7 @@ gc_heap::init_semi_shared()
#ifdef MULTIPLE_HEAPS
mark_list_size = min (100*1024, max (8192, soh_segment_size/(2*10*32)));
#ifdef DYNAMIC_HEAP_COUNT
- if (GCConfig::GetGCDynamicAdaptation() && GCConfig::GetHeapCount() == 0)
+ if (GCConfig::GetGCDynamicAdaptationMode() == 1 && GCConfig::GetHeapCount() == 0)
{
// we'll actually start with one heap in this case
g_mark_list_total_size = mark_list_size;
@@ -14380,7 +14394,7 @@ gc_heap::init_semi_shared()
#endif //FEATURE_EVENT_TRACE
conserve_mem_setting = (int)GCConfig::GetGCConserveMem();
- if (conserve_mem_setting == 0 && GCConfig::GetGCDynamicAdaptation())
+ if (conserve_mem_setting == 0 && GCConfig::GetGCDynamicAdaptationMode() == 1)
conserve_mem_setting = 5;
if (conserve_mem_setting < 0)
conserve_mem_setting = 0;
@@ -15837,9 +15851,9 @@ void allocator::count_items (gc_heap* this_hp, size_t* fl_items_count, size_t* f
assert (((CObjectHeader*)free_item)->IsFree());
num_fl_items++;
- // Get the heap its region belongs to see if we need to put it back.
+ // Get the heap its region belongs to see if we need to put it back.
heap_segment* region = gc_heap::region_of (free_item);
- dprintf (3, ("b#%2d FL %Ix region %Ix heap %d -> %d",
+ dprintf (3, ("b#%2d FL %Ix region %Ix heap %d -> %d",
i, free_item, (size_t)region, this_hp->heap_number, region->heap->heap_number));
if (region->heap != this_hp)
{
@@ -15848,7 +15862,7 @@ void allocator::count_items (gc_heap* this_hp, size_t* fl_items_count, size_t* f
//if ((num_fl_items_rethread % 1000) == 0)
//{
// end_us = GetHighPrecisionTimeStamp();
- // dprintf (8888, ("%Id items rethreaded out of %Id items in %I64d us, current fl: %Ix",
+ // dprintf (8888, ("%Id items rethreaded out of %Id items in %I64d us, current fl: %Ix",
// num_fl_items_rethread, num_fl_items, (end_us - start_us), free_item));
// start_us = end_us;
//}
@@ -15940,9 +15954,9 @@ void allocator::rethread_items (size_t* num_total_fl_items, size_t* num_total_fl
assert (((CObjectHeader*)free_item)->IsFree());
num_fl_items++;
- // Get the heap its region belongs to see if we need to put it back.
+ // Get the heap its region belongs to see if we need to put it back.
heap_segment* region = gc_heap::region_of (free_item);
- dprintf (3, ("b#%2d FL %Ix region %Ix heap %d -> %d",
+ dprintf (3, ("b#%2d FL %Ix region %Ix heap %d -> %d",
i, free_item, (size_t)region, current_heap->heap_number, region->heap->heap_number));
// need to keep track of heap and only check if it's not from our heap!!
if (region->heap != current_heap)
@@ -15986,7 +16000,7 @@ void allocator::rethread_items (size_t* num_total_fl_items, size_t* num_total_fl
// merge buckets from min_fl_list to their corresponding buckets to this FL.
void allocator::merge_items (gc_heap* current_heap, int to_num_heaps, int from_num_heaps)
{
- int this_hn = current_heap->heap_number;
+ int this_hn = current_heap->heap_number;
for (unsigned int i = 0; i < num_buckets; i++)
{
@@ -22481,7 +22495,7 @@ void gc_heap::gc1()
bool gc_heap::prepare_rethread_fl_items()
{
if (!min_fl_list)
- {
+ {
min_fl_list = new (nothrow) min_fl_list_info [MAX_BUCKET_COUNT * n_max_heaps];
if (min_fl_list == nullptr)
return false;
@@ -24882,10 +24896,10 @@ void gc_heap::check_heap_count ()
return;
}
- if (!GCConfig::GetGCDynamicAdaptation())
+ if (GCConfig::GetGCDynamicAdaptationMode() == 0)
{
// don't change the heap count dynamically if the feature isn't explicitly enabled
- return;
+ return;
}
if (heap_number == 0)
@@ -25875,8 +25889,8 @@ BOOL gc_heap::background_mark (uint8_t* o, uint8_t* low, uint8_t* high)
{ \
for (ptrdiff_t __i = 0; __i > cnt; __i--) \
{ \
- HALF_SIZE_T skip = cur->val_serie[__i].skip; \
- HALF_SIZE_T nptrs = cur->val_serie[__i].nptrs; \
+ HALF_SIZE_T skip = (cur->val_serie + __i)->skip; \
+ HALF_SIZE_T nptrs = (cur->val_serie + __i)->nptrs; \
uint8_t** ppstop = parm + nptrs; \
if (!start_useful || (uint8_t*)ppstop > (start)) \
{ \
@@ -30947,21 +30961,21 @@ void gc_heap::process_remaining_regions (int current_plan_gen_num, generation* c
//
// + if the pinned surv of a region is >= demotion_pinned_ratio_th (this will be dynamically tuned based on memory load),
// it will be promoted to its normal planned generation unconditionally.
- //
+ //
// + if the pinned surv is < demotion_pinned_ratio_th, we will always demote it to gen0. We will record how many regions
// have no survival at all - those will be empty and can be used to plan any non gen0 generation if needed.
- //
+ //
// Note! We could actually promote a region with non zero pinned survivors to whichever generation we'd like (eg, we could
// promote a gen0 region to gen2). However it means we'd need to set cards on those objects because we will not have a chance
// later. The benefit of doing this is small in general as when we get into this method, it's very rare we don't already
// have planned regions in higher generations. So I don't think it's worth the complexicity for now. We may consider it
// for the future.
- //
+ //
// + if after we are done walking the remaining regions, we still haven't successfully planned all the needed generations,
// we check to see if we have enough in the regions that will be empty (note that we call set_region_plan_gen_num on
// these regions which means they are planned in gen0. So we need to make sure at least gen0 has 1 region). If so
// thread_final_regions will naturally get one from there so we don't need to call set_region_plan_gen_num to replace the
- // plan gen num.
+ // plan gen num.
//
// + if we don't have enough in regions that will be empty, we'll need to ask for new regions and if we can't, we fall back
// to the special sweep mode.
@@ -31012,7 +31026,7 @@ void gc_heap::process_remaining_regions (int current_plan_gen_num, generation* c
heap_segment_plan_allocated (nseg) = generation_allocation_pointer (consing_gen);
decide_on_demotion_pin_surv (nseg, &to_be_empty_regions);
-
+
heap_segment* next_seg = heap_segment_next_non_sip (nseg);
if ((next_seg == 0) && (heap_segment_gen_num (nseg) > 0))
@@ -31610,7 +31624,7 @@ void gc_heap::plan_phase (int condemned_gen_number)
uint8_t** mark_list_index = nullptr;
uint8_t** mark_list_next = nullptr;
if (use_mark_list)
- mark_list_next = get_region_mark_list (x, end, &mark_list_index);
+ mark_list_next = get_region_mark_list (use_mark_list, x, end, &mark_list_index);
#else // USE_REGIONS
assert (!marked (x));
uint8_t** mark_list_next = &mark_list[0];
@@ -31898,7 +31912,7 @@ void gc_heap::plan_phase (int condemned_gen_number)
current_brick = brick_of (x);
#ifdef USE_REGIONS
if (use_mark_list)
- mark_list_next = get_region_mark_list (x, end, &mark_list_index);
+ mark_list_next = get_region_mark_list (use_mark_list, x, end, &mark_list_index);
if (should_sweep_in_plan (seg1))
{
@@ -31968,7 +31982,7 @@ void gc_heap::plan_phase (int condemned_gen_number)
current_brick = brick_of (x);
if (use_mark_list)
- mark_list_next = get_region_mark_list (x, end, &mark_list_index);
+ mark_list_next = get_region_mark_list (use_mark_list, x, end, &mark_list_index);
if (should_sweep_in_plan (seg1))
{
@@ -46217,7 +46231,7 @@ void gc_heap::descr_generations (const char* msg)
dprintf (1, ("[%5s] GC#%5Id total heap size: %Idmb (F: %Idmb %d%%) commit size: %Idmb, %0.3f min, %d,%d new in plan, %d in threading\n",
msg, idx, alloc_size, frag_size,
(int)((double)frag_size * 100.0 / (double)alloc_size),
- commit_size,
+ commit_size,
(double)elapsed_time_so_far / (double)1000000 / (double)60,
total_new_gen0_regions_in_plns, total_new_regions_in_prr, total_new_regions_in_threading));
}
@@ -48067,7 +48081,7 @@ HRESULT GCHeap::Initialize()
{
#ifdef DYNAMIC_HEAP_COUNT
// if no heap count was specified, and we are told to adjust heap count dynamically ...
- if (GCConfig::GetHeapCount() == 0 && GCConfig::GetGCDynamicAdaptation())
+ if (GCConfig::GetHeapCount() == 0 && GCConfig::GetGCDynamicAdaptationMode() == 1)
{
// ... start with only 1 heap
gc_heap::g_heaps[0]->change_heap_count (1);
@@ -50509,7 +50523,7 @@ size_t gc_heap::get_gen0_min_size()
int n_heaps = 1;
#endif //SERVER_GC
- if (GCConfig::GetGCConserveMem() != 0 || GCConfig::GetGCDynamicAdaptation())
+ if (GCConfig::GetGCConserveMem() != 0 || GCConfig::GetGCDynamicAdaptationMode() == 1)
{
// if we are asked to be stingy with memory, limit gen 0 size
gen0size = min (gen0size, (4*1024*1024));
diff --git a/src/coreclr/gc/gcconfig.h b/src/coreclr/gc/gcconfig.h
index e5a551baa37d5e..72786778d5a978 100644
--- a/src/coreclr/gc/gcconfig.h
+++ b/src/coreclr/gc/gcconfig.h
@@ -138,7 +138,7 @@ class GCConfigStringHolder
INT_CONFIG (GCWriteBarrier, "GCWriteBarrier", NULL, 0, "Specifies whether GC should use more precise but slower write barrier") \
STRING_CONFIG(GCName, "GCName", "System.GC.Name", "Specifies the path of the standalone GC implementation.") \
INT_CONFIG (GCSpinCountUnit, "GCSpinCountUnit", 0, 0, "Specifies the spin count unit used by the GC.") \
- BOOL_CONFIG (GCDynamicAdaptation, "GCDynamicAdaptation", NULL, false, "Enables varying the heap count dynamically in Server GC.")
+ INT_CONFIG (GCDynamicAdaptationMode, "GCDynamicAdaptationMode", "System.GC.DynamicAdaptationMode", 0, "Enable the GC to dynamically adapt to application sizes.")
// This class is responsible for retreiving configuration information
// for how the GC should operate.
class GCConfig
diff --git a/src/coreclr/gc/gcdesc.h b/src/coreclr/gc/gcdesc.h
index 14cee1db9a4cc1..54a13dfdb8cdff 100644
--- a/src/coreclr/gc/gcdesc.h
+++ b/src/coreclr/gc/gcdesc.h
@@ -216,7 +216,7 @@ class CGCDesc
/* Handle the repeating case - array of valuetypes */
for (ptrdiff_t __i = 0; __i > cnt; __i--)
{
- NumOfPointers += cur->val_serie[__i].nptrs;
+ NumOfPointers += (cur->val_serie + __i)->nptrs;
}
NumOfPointers *= NumComponents;
diff --git a/src/coreclr/gc/gcpriv.h b/src/coreclr/gc/gcpriv.h
index 14256523e02a0c..1263610d4bf456 100644
--- a/src/coreclr/gc/gcpriv.h
+++ b/src/coreclr/gc/gcpriv.h
@@ -3160,7 +3160,7 @@ class gc_heap
PER_HEAP_ISOLATED_METHOD void grow_mark_list();
#ifdef USE_REGIONS
- PER_HEAP_METHOD uint8_t** get_region_mark_list (uint8_t* start, uint8_t* end, uint8_t*** mark_list_end);
+ PER_HEAP_METHOD uint8_t** get_region_mark_list (BOOL& use_mark_list, uint8_t* start, uint8_t* end, uint8_t*** mark_list_end);
#endif //USE_REGIONS
#ifdef BACKGROUND_GC
diff --git a/src/coreclr/hosts/corerun/corerun.cpp b/src/coreclr/hosts/corerun/corerun.cpp
index 83f3774936d6b5..4e506b95c9746b 100644
--- a/src/coreclr/hosts/corerun/corerun.cpp
+++ b/src/coreclr/hosts/corerun/corerun.cpp
@@ -247,7 +247,9 @@ static int run(const configuration& config)
// Check if debugger attach scenario was requested.
if (config.wait_to_debug)
wait_for_debugger();
-
+
+ config.dotenv_configuration.load_into_current_process();
+
string_t exe_path = pal::get_exe_path();
// Determine the managed application's path.
@@ -303,8 +305,6 @@ static int run(const configuration& config)
}
}
- config.dotenv_configuration.load_into_current_process();
-
actions.before_coreclr_load();
// Attempt to load CoreCLR.
diff --git a/src/coreclr/inc/ceefilegenwriter.h b/src/coreclr/inc/ceefilegenwriter.h
index 06e20a6bd1fbd8..0364767cd5f471 100644
--- a/src/coreclr/inc/ceefilegenwriter.h
+++ b/src/coreclr/inc/ceefilegenwriter.h
@@ -69,11 +69,8 @@ class CeeFileGenWriter : public CCeeGen
HRESULT allocateIAT();
public:
// Create with one of these two methods, not operator new
- static HRESULT CreateNewInstance(CCeeGen *pCeeFileGenFrom, CeeFileGenWriter* & pGenWriter,
- DWORD createFlags = ICEE_CREATE_FILE_PURE_IL);
- // See ICeeFileGen.h for the definition of the bits used in createFlags
- static HRESULT CreateNewInstanceEx(CCeeGen *pCeeFileGenFrom, CeeFileGenWriter* & pGenWriter,
- DWORD createFlags, LPCWSTR seedFileName = NULL);
+ static HRESULT CreateNewInstance(CeeFileGenWriter* & pGenWriter,
+ DWORD createFlags = ICEE_CREATE_FILE_PURE_IL);
virtual HRESULT Cleanup();
diff --git a/src/coreclr/inc/ceegen.h b/src/coreclr/inc/ceegen.h
index cb86a18c3f8277..67cb10d646b1ac 100644
--- a/src/coreclr/inc/ceegen.h
+++ b/src/coreclr/inc/ceegen.h
@@ -185,7 +185,6 @@ class CCeeGen : public ICeeGenInternal {
CeeGenTokenMapper *m_pTokenMap;
BOOLEAN m_fTokenMapSupported; // temporary to support both models
- IMapToken *m_pRemapHandler;
CeeSection **m_sections;
short m_numSections;
@@ -238,9 +237,6 @@ class CCeeGen : public ICeeGenInternal {
ULONG RVA,
UCHAR **lpBuffer);
- STDMETHODIMP GetIMapTokenIface (
- IUnknown **pIMapToken);
-
STDMETHODIMP GenerateCeeFile ();
STDMETHODIMP GetIlSection (
@@ -276,8 +272,6 @@ class CCeeGen : public ICeeGenInternal {
UCHAR **lpBuffer); // [OUT] Returned buffer
- STDMETHODIMP AddNotificationHandler(IUnknown *pHandler);
-
// Write the metadata in "emitter" to the default metadata section is "section" is 0
// If 'section != 0, it will put the data in 'buffer'. This
// buffer is assumed to be in 'section' at 'offset' and of size 'buffLen'
@@ -307,14 +301,6 @@ class CCeeGen : public ICeeGenInternal {
LIMITED_METHOD_CONTRACT;
return m_pTokenMap;
}
-
- virtual HRESULT addNotificationHandler(IUnknown *pHandler);
-
- //Clone is actually a misnomer here. This method will copy all of the
- //instance variables and then do a deep copy (as necessary) of the sections.
- //Section data will be appended onto any information already in the section.
- //This is done to support the DynamicIL -> PersistedIL transform.
- virtual HRESULT cloneInstance(CCeeGen *destination);
};
// ***** CeeSection inline methods
diff --git a/src/coreclr/inc/ceegentokenmapper.h b/src/coreclr/inc/ceegentokenmapper.h
index a1bb9b7ef4e4cc..02d98c92d53c3b 100644
--- a/src/coreclr/inc/ceegentokenmapper.h
+++ b/src/coreclr/inc/ceegentokenmapper.h
@@ -48,7 +48,7 @@ friend class PESectionMan;
static int IndexForType(mdToken tk);
- CeeGenTokenMapper() : m_pIImport(0), m_cRefs(1), m_pIMapToken(NULL) { LIMITED_METHOD_CONTRACT; }
+ CeeGenTokenMapper() : m_pIImport(0), m_cRefs(1) { LIMITED_METHOD_CONTRACT; }
virtual ~CeeGenTokenMapper() {}
//*****************************************************************************
@@ -66,12 +66,6 @@ friend class PESectionMan;
ULONG cRefs = --m_cRefs;
if (cRefs == 0)
{
- if (m_pIMapToken)
- {
- m_pIMapToken->Release();
- m_pIMapToken = NULL;
- }
-
delete this;
}
return cRefs;
@@ -104,28 +98,6 @@ friend class PESectionMan;
//*****************************************************************************
virtual HRESULT GetMetaData(IMetaDataImport **ppIImport);
-//*****************************************************************************
-// Add another token mapper.
-//*****************************************************************************
- virtual HRESULT AddTokenMapper(IMapToken *pIMapToken)
- {
- STATIC_CONTRACT_NOTHROW;
- STATIC_CONTRACT_FORBID_FAULT;
-
- // Add the token mapper, if there isn't already one.
- if (m_pIMapToken == NULL)
- {
- m_pIMapToken = pIMapToken;
- m_pIMapToken->AddRef();
- return S_OK;
- }
- else
- {
- _ASSERTE(!"Token mapper already set!");
- return E_FAIL;
- }
- }
-
protected:
// m_rgMap is an array indexed by token type. For each type, an array of
// tokens is kept, indexed by from rid. To see if a token has been moved,
@@ -134,8 +106,6 @@ friend class PESectionMan;
TOKENMAP m_rgMap[MAX_TOKENMAP];
IMetaDataImport *m_pIImport;
ULONG m_cRefs; // Ref count.
- IMapToken *m_pIMapToken;
-
};
#endif // __CeeGenTokenMapper_h__
diff --git a/src/coreclr/inc/clrconfigvalues.h b/src/coreclr/inc/clrconfigvalues.h
index 184b8f6b759c61..824be81ec313be 100644
--- a/src/coreclr/inc/clrconfigvalues.h
+++ b/src/coreclr/inc/clrconfigvalues.h
@@ -320,17 +320,6 @@ RETAIL_CONFIG_STRING_INFO(EXTERNAL_AltJitOs, W("AltJitOS"), "Sets target OS for
RETAIL_CONFIG_STRING_INFO(EXTERNAL_AltJitExcludeAssemblies, W("AltJitExcludeAssemblies"), "Do not use AltJit on this semicolon-delimited list of assemblies.")
#endif // defined(ALLOW_SXS_JIT)
-#if defined(FEATURE_STACK_SAMPLING)
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_StackSamplingEnabled, W("StackSamplingEnabled"), 0, "Is stack sampling based tracking of evolving hot methods enabled.")
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_StackSamplingAfter, W("StackSamplingAfter"), 0, "When to start sampling (for some sort of app steady state), i.e., initial delay for sampling start in milliseconds.")
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_StackSamplingEvery, W("StackSamplingEvery"), 100, "How frequent should thread stacks be sampled in milliseconds.")
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_StackSamplingNumMethods, W("StackSamplingNumMethods"), 32, "Number of evolving methods to track as hot and JIT them in the background at a given point of execution.")
-#endif // defined(FEATURE_JIT_SAMPLING)
-
-#if defined(ALLOW_SXS_JIT_NGEN)
-RETAIL_CONFIG_STRING_INFO(INTERNAL_AltJitNgen, W("AltJitNgen"), "Enables AltJit for NGEN and selectively limits it to the specified methods.")
-#endif // defined(ALLOW_SXS_JIT_NGEN)
-
RETAIL_CONFIG_DWORD_INFO(EXTERNAL_JitHostMaxSlabCache, W("JitHostMaxSlabCache"), 0x1000000, "Sets jit host max slab cache size, 16MB default")
RETAIL_CONFIG_DWORD_INFO(EXTERNAL_JitOptimizeType, W("JitOptimizeType"), 0 /* OPT_DEFAULT */, "")
@@ -357,12 +346,6 @@ RETAIL_CONFIG_DWORD_INFO(EXTERNAL_JitRegisterFP, W("JitRegisterFP"), 3, "Control
RETAIL_CONFIG_DWORD_INFO(INTERNAL_JitELTHookEnabled, W("JitELTHookEnabled"), 0, "On ARM, setting this will emit Enter/Leave/TailCall callbacks")
RETAIL_CONFIG_DWORD_INFO(INTERNAL_JitMemStats, W("JitMemStats"), 0, "Display JIT memory usage statistics")
RETAIL_CONFIG_DWORD_INFO(INTERNAL_JitVNMapSelBudget, W("JitVNMapSelBudget"), 100, "Max # of MapSelect's considered for a particular top-level invocation.")
-#if defined(TARGET_AMD64) || defined(TARGET_X86) || defined(TARGET_ARM64)
-#define EXTERNAL_FeatureSIMD_Default 1
-#else // !(defined(TARGET_AMD64) || defined(TARGET_X86) || defined(TARGET_ARM64))
-#define EXTERNAL_FeatureSIMD_Default 0
-#endif // !(defined(TARGET_AMD64) || defined(TARGET_X86) || defined(TARGET_ARM64))
-RETAIL_CONFIG_DWORD_INFO(INTERNAL_SIMD16ByteOnly, W("SIMD16ByteOnly"), 0, "Limit maximum SIMD vector length to 16 bytes (used by x64_arm64_altjit)")
RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TrackDynamicMethodDebugInfo, W("TrackDynamicMethodDebugInfo"), 0, "Specifies whether debug info should be generated and tracked for dynamic methods")
#ifdef FEATURE_MULTICOREJIT
@@ -577,11 +560,11 @@ RETAIL_CONFIG_DWORD_INFO(INTERNAL_HillClimbing_GainExponent,
#endif // _DEBUG
RETAIL_CONFIG_DWORD_INFO(EXTERNAL_TieredCompilation, W("TieredCompilation"), 1, "Enables tiered compilation")
RETAIL_CONFIG_DWORD_INFO(EXTERNAL_TC_QuickJit, W("TC_QuickJit"), 1, "For methods that would be jitted, enable using quick JIT when appropriate.")
-#if defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#if defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TC_QuickJitForLoops, W("TC_QuickJitForLoops"), 1, "When quick JIT is enabled, quick JIT may also be used for methods that contain loops.")
-#else // !(defined(TARGET_AMD64) || defined(TARGET_ARM64))
+#else // !(defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64))
RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TC_QuickJitForLoops, W("TC_QuickJitForLoops"), 0, "When quick JIT is enabled, quick JIT may also be used for methods that contain loops.")
-#endif // defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#endif // defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
RETAIL_CONFIG_DWORD_INFO(EXTERNAL_TC_AggressiveTiering, W("TC_AggressiveTiering"), 0, "Transition through tiers aggressively.")
RETAIL_CONFIG_DWORD_INFO(INTERNAL_TC_BackgroundWorkerTimeoutMs, W("TC_BackgroundWorkerTimeoutMs"), TC_BackgroundWorkerTimeoutMs, "How long in milliseconds the background worker thread may remain idle before exiting.")
RETAIL_CONFIG_DWORD_INFO(INTERNAL_TC_CallCountThreshold, W("TC_CallCountThreshold"), TC_CallCountThreshold, "Number of times a method must be called in tier 0 after which it is promoted to the next tier.")
@@ -745,15 +728,17 @@ RETAIL_CONFIG_DWORD_INFO(INTERNAL_GDBJitEmitDebugFrame, W("GDBJitEmitDebugFrame"
#endif
#endif
+RETAIL_CONFIG_DWORD_INFO(EXTERNAL_MaxVectorTBitWidth, W("MaxVectorTBitWidth"), 0, "The maximum width, in bits, that Vector is allowed to be. A value less than 128 is treated as the system default.")
+
//
// Hardware Intrinsic ISAs; keep in sync with jitconfigvalues.h
//
#if defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
//TODO: should implement LoongArch64's features.
//TODO-RISCV64-CQ: should implement RISCV64's features.
-RETAIL_CONFIG_DWORD_INFO(EXTERNAL_EnableHWIntrinsic, W("EnableHWIntrinsic"), 0, "Allows Base+ hardware intrinsics to be disabled")
+RETAIL_CONFIG_DWORD_INFO(EXTERNAL_EnableHWIntrinsic, W("EnableHWIntrinsic"), 0, "Allows Base+ hardware intrinsics to be disabled")
#else
-RETAIL_CONFIG_DWORD_INFO(EXTERNAL_EnableHWIntrinsic, W("EnableHWIntrinsic"), 1, "Allows Base+ hardware intrinsics to be disabled")
+RETAIL_CONFIG_DWORD_INFO(EXTERNAL_EnableHWIntrinsic, W("EnableHWIntrinsic"), 1, "Allows Base+ hardware intrinsics to be disabled")
#endif // defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
#if defined(TARGET_AMD64) || defined(TARGET_X86)
diff --git a/src/coreclr/inc/corerror.xml b/src/coreclr/inc/corerror.xml
index afeaf4f602a4ad..81c7bab5ea1f0a 100644
--- a/src/coreclr/inc/corerror.xml
+++ b/src/coreclr/inc/corerror.xml
@@ -1603,7 +1603,7 @@
COR_E_TARGET"Attempt to invoke non-static method with a null Object."
- - If you attempt to invoke a non-static method with a null Object - If you atte
+ If you attempt to invoke a non-static method with a null Object
diff --git a/src/coreclr/inc/corinfoinstructionset.h b/src/coreclr/inc/corinfoinstructionset.h
index 8462ab33413ef9..5aecf46bf3928c 100644
--- a/src/coreclr/inc/corinfoinstructionset.h
+++ b/src/coreclr/inc/corinfoinstructionset.h
@@ -14,7 +14,7 @@
enum CORINFO_InstructionSet
{
InstructionSet_ILLEGAL = 0,
- InstructionSet_NONE = 63,
+ InstructionSet_NONE = 127,
#ifdef TARGET_ARM64
InstructionSet_ArmBase=1,
InstructionSet_AdvSimd=2,
@@ -29,14 +29,15 @@ enum CORINFO_InstructionSet
InstructionSet_Vector128=11,
InstructionSet_Dczva=12,
InstructionSet_Rcpc=13,
- InstructionSet_ArmBase_Arm64=14,
- InstructionSet_AdvSimd_Arm64=15,
- InstructionSet_Aes_Arm64=16,
- InstructionSet_Crc32_Arm64=17,
- InstructionSet_Dp_Arm64=18,
- InstructionSet_Rdm_Arm64=19,
- InstructionSet_Sha1_Arm64=20,
- InstructionSet_Sha256_Arm64=21,
+ InstructionSet_VectorT128=14,
+ InstructionSet_ArmBase_Arm64=15,
+ InstructionSet_AdvSimd_Arm64=16,
+ InstructionSet_Aes_Arm64=17,
+ InstructionSet_Crc32_Arm64=18,
+ InstructionSet_Dp_Arm64=19,
+ InstructionSet_Rdm_Arm64=20,
+ InstructionSet_Sha1_Arm64=21,
+ InstructionSet_Sha256_Arm64=22,
#endif // TARGET_ARM64
#ifdef TARGET_AMD64
InstructionSet_X86Base=1,
@@ -71,35 +72,38 @@ enum CORINFO_InstructionSet
InstructionSet_AVX512DQ_VL=30,
InstructionSet_AVX512VBMI=31,
InstructionSet_AVX512VBMI_VL=32,
- InstructionSet_X86Base_X64=33,
- InstructionSet_SSE_X64=34,
- InstructionSet_SSE2_X64=35,
- InstructionSet_SSE3_X64=36,
- InstructionSet_SSSE3_X64=37,
- InstructionSet_SSE41_X64=38,
- InstructionSet_SSE42_X64=39,
- InstructionSet_AVX_X64=40,
- InstructionSet_AVX2_X64=41,
- InstructionSet_AES_X64=42,
- InstructionSet_BMI1_X64=43,
- InstructionSet_BMI2_X64=44,
- InstructionSet_FMA_X64=45,
- InstructionSet_LZCNT_X64=46,
- InstructionSet_PCLMULQDQ_X64=47,
- InstructionSet_POPCNT_X64=48,
- InstructionSet_AVXVNNI_X64=49,
- InstructionSet_MOVBE_X64=50,
- InstructionSet_X86Serialize_X64=51,
- InstructionSet_AVX512F_X64=52,
- InstructionSet_AVX512F_VL_X64=53,
- InstructionSet_AVX512BW_X64=54,
- InstructionSet_AVX512BW_VL_X64=55,
- InstructionSet_AVX512CD_X64=56,
- InstructionSet_AVX512CD_VL_X64=57,
- InstructionSet_AVX512DQ_X64=58,
- InstructionSet_AVX512DQ_VL_X64=59,
- InstructionSet_AVX512VBMI_X64=60,
- InstructionSet_AVX512VBMI_VL_X64=61,
+ InstructionSet_VectorT128=33,
+ InstructionSet_VectorT256=34,
+ InstructionSet_VectorT512=35,
+ InstructionSet_X86Base_X64=36,
+ InstructionSet_SSE_X64=37,
+ InstructionSet_SSE2_X64=38,
+ InstructionSet_SSE3_X64=39,
+ InstructionSet_SSSE3_X64=40,
+ InstructionSet_SSE41_X64=41,
+ InstructionSet_SSE42_X64=42,
+ InstructionSet_AVX_X64=43,
+ InstructionSet_AVX2_X64=44,
+ InstructionSet_AES_X64=45,
+ InstructionSet_BMI1_X64=46,
+ InstructionSet_BMI2_X64=47,
+ InstructionSet_FMA_X64=48,
+ InstructionSet_LZCNT_X64=49,
+ InstructionSet_PCLMULQDQ_X64=50,
+ InstructionSet_POPCNT_X64=51,
+ InstructionSet_AVXVNNI_X64=52,
+ InstructionSet_MOVBE_X64=53,
+ InstructionSet_X86Serialize_X64=54,
+ InstructionSet_AVX512F_X64=55,
+ InstructionSet_AVX512F_VL_X64=56,
+ InstructionSet_AVX512BW_X64=57,
+ InstructionSet_AVX512BW_VL_X64=58,
+ InstructionSet_AVX512CD_X64=59,
+ InstructionSet_AVX512CD_VL_X64=60,
+ InstructionSet_AVX512DQ_X64=61,
+ InstructionSet_AVX512DQ_VL_X64=62,
+ InstructionSet_AVX512VBMI_X64=63,
+ InstructionSet_AVX512VBMI_VL_X64=64,
#endif // TARGET_AMD64
#ifdef TARGET_X86
InstructionSet_X86Base=1,
@@ -134,35 +138,38 @@ enum CORINFO_InstructionSet
InstructionSet_AVX512DQ_VL=30,
InstructionSet_AVX512VBMI=31,
InstructionSet_AVX512VBMI_VL=32,
- InstructionSet_X86Base_X64=33,
- InstructionSet_SSE_X64=34,
- InstructionSet_SSE2_X64=35,
- InstructionSet_SSE3_X64=36,
- InstructionSet_SSSE3_X64=37,
- InstructionSet_SSE41_X64=38,
- InstructionSet_SSE42_X64=39,
- InstructionSet_AVX_X64=40,
- InstructionSet_AVX2_X64=41,
- InstructionSet_AES_X64=42,
- InstructionSet_BMI1_X64=43,
- InstructionSet_BMI2_X64=44,
- InstructionSet_FMA_X64=45,
- InstructionSet_LZCNT_X64=46,
- InstructionSet_PCLMULQDQ_X64=47,
- InstructionSet_POPCNT_X64=48,
- InstructionSet_AVXVNNI_X64=49,
- InstructionSet_MOVBE_X64=50,
- InstructionSet_X86Serialize_X64=51,
- InstructionSet_AVX512F_X64=52,
- InstructionSet_AVX512F_VL_X64=53,
- InstructionSet_AVX512BW_X64=54,
- InstructionSet_AVX512BW_VL_X64=55,
- InstructionSet_AVX512CD_X64=56,
- InstructionSet_AVX512CD_VL_X64=57,
- InstructionSet_AVX512DQ_X64=58,
- InstructionSet_AVX512DQ_VL_X64=59,
- InstructionSet_AVX512VBMI_X64=60,
- InstructionSet_AVX512VBMI_VL_X64=61,
+ InstructionSet_VectorT128=33,
+ InstructionSet_VectorT256=34,
+ InstructionSet_VectorT512=35,
+ InstructionSet_X86Base_X64=36,
+ InstructionSet_SSE_X64=37,
+ InstructionSet_SSE2_X64=38,
+ InstructionSet_SSE3_X64=39,
+ InstructionSet_SSSE3_X64=40,
+ InstructionSet_SSE41_X64=41,
+ InstructionSet_SSE42_X64=42,
+ InstructionSet_AVX_X64=43,
+ InstructionSet_AVX2_X64=44,
+ InstructionSet_AES_X64=45,
+ InstructionSet_BMI1_X64=46,
+ InstructionSet_BMI2_X64=47,
+ InstructionSet_FMA_X64=48,
+ InstructionSet_LZCNT_X64=49,
+ InstructionSet_PCLMULQDQ_X64=50,
+ InstructionSet_POPCNT_X64=51,
+ InstructionSet_AVXVNNI_X64=52,
+ InstructionSet_MOVBE_X64=53,
+ InstructionSet_X86Serialize_X64=54,
+ InstructionSet_AVX512F_X64=55,
+ InstructionSet_AVX512F_VL_X64=56,
+ InstructionSet_AVX512BW_X64=57,
+ InstructionSet_AVX512BW_VL_X64=58,
+ InstructionSet_AVX512CD_X64=59,
+ InstructionSet_AVX512CD_VL_X64=60,
+ InstructionSet_AVX512DQ_X64=61,
+ InstructionSet_AVX512DQ_VL_X64=62,
+ InstructionSet_AVX512VBMI_X64=63,
+ InstructionSet_AVX512VBMI_VL_X64=64,
#endif // TARGET_X86
};
@@ -170,7 +177,7 @@ enum CORINFO_InstructionSet
struct CORINFO_InstructionSetFlags
{
private:
- static const int32_t FlagsFieldCount = 1;
+ static const int32_t FlagsFieldCount = 2;
static const int32_t BitsPerFlagsField = sizeof(uint64_t) * 8;
uint64_t _flags[FlagsFieldCount] = { };
@@ -404,6 +411,8 @@ inline CORINFO_InstructionSetFlags EnsureInstructionSetFlagsAreValid(CORINFO_Ins
resultflags.RemoveInstructionSet(InstructionSet_Vector64);
if (resultflags.HasInstructionSet(InstructionSet_Vector128) && !resultflags.HasInstructionSet(InstructionSet_AdvSimd))
resultflags.RemoveInstructionSet(InstructionSet_Vector128);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT128) && !resultflags.HasInstructionSet(InstructionSet_AdvSimd))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT128);
#endif // TARGET_ARM64
#ifdef TARGET_AMD64
if (resultflags.HasInstructionSet(InstructionSet_X86Base) && !resultflags.HasInstructionSet(InstructionSet_X86Base_X64))
@@ -566,24 +575,46 @@ inline CORINFO_InstructionSetFlags EnsureInstructionSetFlagsAreValid(CORINFO_Ins
resultflags.RemoveInstructionSet(InstructionSet_X86Serialize);
if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX2))
resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_FMA))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet_AVX512F_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512CD) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512CD);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512CD))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512BW) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512DQ))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW))
resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512VBMI))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT128) && !resultflags.HasInstructionSet(InstructionSet_SSE2))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT128);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT256) && !resultflags.HasInstructionSet(InstructionSet_AVX2))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT256);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT512) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT512);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
#endif // TARGET_AMD64
#ifdef TARGET_X86
if (resultflags.HasInstructionSet(InstructionSet_SSE) && !resultflags.HasInstructionSet(InstructionSet_X86Base))
@@ -630,24 +661,46 @@ inline CORINFO_InstructionSetFlags EnsureInstructionSetFlagsAreValid(CORINFO_Ins
resultflags.RemoveInstructionSet(InstructionSet_X86Serialize);
if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX2))
resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_FMA))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet_AVX512F_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512CD) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512CD);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512CD))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512BW) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512DQ))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512F_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW))
resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512VBMI))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL);
if (resultflags.HasInstructionSet(InstructionSet_AVX512VBMI_VL) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL))
resultflags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT128) && !resultflags.HasInstructionSet(InstructionSet_SSE2))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT128);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT256) && !resultflags.HasInstructionSet(InstructionSet_AVX2))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT256);
+ if (resultflags.HasInstructionSet(InstructionSet_VectorT512) && !resultflags.HasInstructionSet(InstructionSet_AVX512F))
+ resultflags.RemoveInstructionSet(InstructionSet_VectorT512);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512BW_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512CD_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet_AVX512F) && !resultflags.HasInstructionSet(InstructionSet_AVX512DQ_VL))
+ resultflags.RemoveInstructionSet(InstructionSet_AVX512F);
#endif // TARGET_X86
} while (!oldflags.Equals(resultflags));
@@ -706,6 +759,8 @@ inline const char *InstructionSetToString(CORINFO_InstructionSet instructionSet)
return "Dczva";
case InstructionSet_Rcpc :
return "Rcpc";
+ case InstructionSet_VectorT128 :
+ return "VectorT128";
#endif // TARGET_ARM64
#ifdef TARGET_AMD64
case InstructionSet_X86Base :
@@ -830,6 +885,12 @@ inline const char *InstructionSetToString(CORINFO_InstructionSet instructionSet)
return "AVX512VBMI_VL";
case InstructionSet_AVX512VBMI_VL_X64 :
return "AVX512VBMI_VL_X64";
+ case InstructionSet_VectorT128 :
+ return "VectorT128";
+ case InstructionSet_VectorT256 :
+ return "VectorT256";
+ case InstructionSet_VectorT512 :
+ return "VectorT512";
#endif // TARGET_AMD64
#ifdef TARGET_X86
case InstructionSet_X86Base :
@@ -896,6 +957,12 @@ inline const char *InstructionSetToString(CORINFO_InstructionSet instructionSet)
return "AVX512VBMI";
case InstructionSet_AVX512VBMI_VL :
return "AVX512VBMI_VL";
+ case InstructionSet_VectorT128 :
+ return "VectorT128";
+ case InstructionSet_VectorT256 :
+ return "VectorT256";
+ case InstructionSet_VectorT512 :
+ return "VectorT512";
#endif // TARGET_X86
default:
@@ -926,6 +993,7 @@ inline CORINFO_InstructionSet InstructionSetFromR2RInstructionSet(ReadyToRunInst
case READYTORUN_INSTRUCTION_Sha256: return InstructionSet_Sha256;
case READYTORUN_INSTRUCTION_Atomics: return InstructionSet_Atomics;
case READYTORUN_INSTRUCTION_Rcpc: return InstructionSet_Rcpc;
+ case READYTORUN_INSTRUCTION_VectorT128: return InstructionSet_VectorT128;
#endif // TARGET_ARM64
#ifdef TARGET_AMD64
case READYTORUN_INSTRUCTION_X86Base: return InstructionSet_X86Base;
@@ -957,6 +1025,9 @@ inline CORINFO_InstructionSet InstructionSetFromR2RInstructionSet(ReadyToRunInst
case READYTORUN_INSTRUCTION_Avx512DQ_VL: return InstructionSet_AVX512DQ_VL;
case READYTORUN_INSTRUCTION_Avx512Vbmi: return InstructionSet_AVX512VBMI;
case READYTORUN_INSTRUCTION_Avx512Vbmi_VL: return InstructionSet_AVX512VBMI_VL;
+ case READYTORUN_INSTRUCTION_VectorT128: return InstructionSet_VectorT128;
+ case READYTORUN_INSTRUCTION_VectorT256: return InstructionSet_VectorT256;
+ case READYTORUN_INSTRUCTION_VectorT512: return InstructionSet_VectorT512;
#endif // TARGET_AMD64
#ifdef TARGET_X86
case READYTORUN_INSTRUCTION_X86Base: return InstructionSet_X86Base;
@@ -988,6 +1059,9 @@ inline CORINFO_InstructionSet InstructionSetFromR2RInstructionSet(ReadyToRunInst
case READYTORUN_INSTRUCTION_Avx512DQ_VL: return InstructionSet_AVX512DQ_VL;
case READYTORUN_INSTRUCTION_Avx512Vbmi: return InstructionSet_AVX512VBMI;
case READYTORUN_INSTRUCTION_Avx512Vbmi_VL: return InstructionSet_AVX512VBMI_VL;
+ case READYTORUN_INSTRUCTION_VectorT128: return InstructionSet_VectorT128;
+ case READYTORUN_INSTRUCTION_VectorT256: return InstructionSet_VectorT256;
+ case READYTORUN_INSTRUCTION_VectorT512: return InstructionSet_VectorT512;
#endif // TARGET_X86
default:
diff --git a/src/coreclr/inc/corjitflags.h b/src/coreclr/inc/corjitflags.h
index cc4ad7900e2435..6f7b00cb6f9197 100644
--- a/src/coreclr/inc/corjitflags.h
+++ b/src/coreclr/inc/corjitflags.h
@@ -70,7 +70,7 @@ class CORJIT_FLAGS
CORJIT_FLAG_BBINSTR_IF_LOOPS = 32, // JIT must instrument current method if it has loops
CORJIT_FLAG_PUBLISH_SECRET_PARAM = 33, // JIT must place stub secret param into local 0. (used by IL stubs)
CORJIT_FLAG_UNUSED14 = 34,
- CORJIT_FLAG_SAMPLING_JIT_BACKGROUND = 35, // JIT is being invoked as a result of stack sampling for hot methods in the background
+ CORJIT_FLAG_UNUSED37 = 35,
CORJIT_FLAG_USE_PINVOKE_HELPERS = 36, // The JIT should use the PINVOKE_{BEGIN,END} helpers instead of emitting inline transitions
CORJIT_FLAG_REVERSE_PINVOKE = 37, // The JIT should insert REVERSE_PINVOKE_{ENTER,EXIT} helpers into method prolog/epilog
CORJIT_FLAG_TRACK_TRANSITIONS = 38, // The JIT should insert the REVERSE_PINVOKE helper variants that track transitions.
diff --git a/src/coreclr/inc/corpriv.h b/src/coreclr/inc/corpriv.h
index 505ac83e1be984..9fa3fbb9f71d2c 100644
--- a/src/coreclr/inc/corpriv.h
+++ b/src/coreclr/inc/corpriv.h
@@ -403,9 +403,6 @@ DECLARE_INTERFACE_(ICeeGenInternal, IUnknown)
ULONG RVA, // [IN] RVA for method to return
UCHAR * *lpBuffer) PURE; // [OUT] Returned buffer
- STDMETHOD(GetIMapTokenIface) (
- IUnknown * *pIMapToken) PURE;
-
STDMETHOD(GenerateCeeFile) () PURE;
STDMETHOD(GetIlSection) (
diff --git a/src/coreclr/inc/iceefilegen.h b/src/coreclr/inc/iceefilegen.h
index f0fb5b091c9cac..c3dc26f4d952fd 100644
--- a/src/coreclr/inc/iceefilegen.h
+++ b/src/coreclr/inc/iceefilegen.h
@@ -115,19 +115,12 @@ class ICeeFileGen {
virtual HRESULT SetComImageFlags (HCEEFILE ceeFile, DWORD mask);
virtual HRESULT GetComImageFlags (HCEEFILE ceeFile, DWORD *mask);
- // get IMapToken interface for tracking mapped tokens
- virtual HRESULT GetIMapTokenIface(HCEEFILE ceeFile, IMetaDataEmit *emitter, IUnknown **pIMapToken);
virtual HRESULT SetDirectoryEntry (HCEEFILE ceeFile, HCEESECTION section, ULONG num, ULONG size, ULONG offset = 0);
// Write out the metadata in "emitter" to the metadata section in "ceeFile"
// Use EmitMetaDataAt() for more control
virtual HRESULT EmitMetaDataEx (HCEEFILE ceeFile, IMetaDataEmit *emitter);
- virtual HRESULT GetIMapTokenIfaceEx(HCEEFILE ceeFile, IMetaDataEmit *emitter, IUnknown **pIMapToken);
-
- virtual HRESULT CreateCeeFileFromICeeGen(
- ICeeGenInternal *pFromICeeGen, HCEEFILE *ceeFile, DWORD createFlags = ICEE_CREATE_FILE_PURE_IL); // call this to instantiate a file handle
-
virtual HRESULT SetManifestEntry(HCEEFILE ceeFile, ULONG size, ULONG offset);
virtual HRESULT ComputeSectionOffset(HCEESECTION section, _In_ char *ptr,
@@ -164,12 +157,6 @@ class ICeeFileGen {
virtual HRESULT GetFileTimeStamp (HCEEFILE ceeFile, DWORD *pTimeStamp);
- // Add a notification handler. If it implements an interface that
- // the ICeeFileGen understands, S_OK is returned. Otherwise,
- // E_NOINTERFACE.
- virtual HRESULT AddNotificationHandler(HCEEFILE ceeFile,
- IUnknown *pHandler);
-
virtual HRESULT SetFileAlignment(HCEEFILE ceeFile, ULONG fileAlignment);
virtual HRESULT ClearComImageFlags (HCEEFILE ceeFile, DWORD mask);
@@ -182,13 +169,6 @@ class ICeeFileGen {
PIMAGE_SECTION_HEADER *ppSections,
ULONG *pNumSections);
- // Seed file is a base file which is copied over into the output file
- // Note that there are restrictions on the seed file (the sections
- // cannot be relocated), and that the copy is not complete as the new
- // headers overwrite the seed file headers.
- virtual HRESULT CreateCeeFileEx2(HCEEFILE *ceeFile, ULONG createFlags,
- LPCWSTR seedFileName = NULL);
-
virtual HRESULT SetVTableEntry64(HCEEFILE ceeFile, ULONG size, void* ptr);
};
diff --git a/src/coreclr/inc/jiteeversionguid.h b/src/coreclr/inc/jiteeversionguid.h
index eee46d6068395a..ebcf4919cb6afb 100644
--- a/src/coreclr/inc/jiteeversionguid.h
+++ b/src/coreclr/inc/jiteeversionguid.h
@@ -43,11 +43,11 @@ typedef const GUID *LPCGUID;
#define GUID_DEFINED
#endif // !GUID_DEFINED
-constexpr GUID JITEEVersionIdentifier = { /* dfc41bc9-f134-4c50-897e-fc9304a82059 */
- 0xdfc41bc9,
- 0xf134,
- 0x4c50,
- {0x89, 0x7e, 0xfc, 0x93, 0x04, 0xa8, 0x20, 0x59}
+constexpr GUID JITEEVersionIdentifier = { /* fda2f9dd-6b3e-4ecd-a7b8-79e5edf1f072 */
+ 0xfda2f9dd,
+ 0x6b3e,
+ 0x4ecd,
+ {0xa7, 0xb8, 0x79, 0xe5, 0xed, 0xf1, 0xf0, 0x72}
};
//////////////////////////////////////////////////////////////////////////////////////////////////////////
diff --git a/src/coreclr/inc/pedecoder.h b/src/coreclr/inc/pedecoder.h
index 4ada5241163b7c..efd0555a4f7b9f 100644
--- a/src/coreclr/inc/pedecoder.h
+++ b/src/coreclr/inc/pedecoder.h
@@ -179,8 +179,6 @@ class PEDecoder
DWORD GetCheckSum() const;
WORD GetMachine() const;
WORD GetCharacteristics() const;
- DWORD GetFileAlignment() const;
- DWORD GetSectionAlignment() const;
SIZE_T GetSizeOfStackReserve() const;
SIZE_T GetSizeOfStackCommit() const;
SIZE_T GetSizeOfHeapReserve() const;
diff --git a/src/coreclr/inc/pedecoder.inl b/src/coreclr/inc/pedecoder.inl
index ea2440058f79d5..9b3c9c47238d15 100644
--- a/src/coreclr/inc/pedecoder.inl
+++ b/src/coreclr/inc/pedecoder.inl
@@ -383,36 +383,6 @@ inline DWORD PEDecoder::GetCheckSum() const
return VAL32(FindNTHeaders()->OptionalHeader.CheckSum);
}
-inline DWORD PEDecoder::GetFileAlignment() const
-{
- CONTRACTL
- {
- INSTANCE_CHECK;
- PRECONDITION(CheckNTHeaders());
- NOTHROW;
- GC_NOTRIGGER;
- }
- CONTRACTL_END;
-
- //even though some data in OptionalHeader is different for 32 and 64, this field is the same
- return VAL32(FindNTHeaders()->OptionalHeader.FileAlignment);
-}
-
-inline DWORD PEDecoder::GetSectionAlignment() const
-{
- CONTRACTL
- {
- INSTANCE_CHECK;
- PRECONDITION(CheckNTHeaders());
- NOTHROW;
- GC_NOTRIGGER;
- }
- CONTRACTL_END;
-
- //even though some data in OptionalHeader is different for 32 and 64, this field is the same
- return VAL32(FindNTHeaders()->OptionalHeader.SectionAlignment);
-}
-
inline WORD PEDecoder::GetMachine() const
{
CONTRACTL
diff --git a/src/coreclr/inc/pesectionman.h b/src/coreclr/inc/pesectionman.h
index 042a5dfc26ce77..79e70c7dc916dd 100644
--- a/src/coreclr/inc/pesectionman.h
+++ b/src/coreclr/inc/pesectionman.h
@@ -38,8 +38,6 @@ class PESectionMan
// Apply all the relocs for in memory conversion
HRESULT applyRelocs(CeeGenTokenMapper *pTokenMapper);
- HRESULT cloneInstance(PESectionMan *destination);
-
protected:
// Finds section with given name. returns 0 if not found
@@ -142,9 +140,6 @@ class PESection : public CeeSectionImpl {
// computes an offset as if this were an array
virtual unsigned computeOffset(_In_ char *ptr) const;
- // Make 'destination' a copy of the current PESection
- HRESULT cloneInstance(PESection *destination);
-
// Cause the section to allocate memory in smaller chunks
void SetInitialGrowth(unsigned growth);
diff --git a/src/coreclr/inc/readytoruninstructionset.h b/src/coreclr/inc/readytoruninstructionset.h
index faf4a5028cc1c9..0a9a78e03f6c15 100644
--- a/src/coreclr/inc/readytoruninstructionset.h
+++ b/src/coreclr/inc/readytoruninstructionset.h
@@ -47,6 +47,9 @@ enum ReadyToRunInstructionSet
READYTORUN_INSTRUCTION_Avx512DQ_VL=36,
READYTORUN_INSTRUCTION_Avx512Vbmi=37,
READYTORUN_INSTRUCTION_Avx512Vbmi_VL=38,
+ READYTORUN_INSTRUCTION_VectorT128=39,
+ READYTORUN_INSTRUCTION_VectorT256=40,
+ READYTORUN_INSTRUCTION_VectorT512=41,
};
diff --git a/src/coreclr/inc/switches.h b/src/coreclr/inc/switches.h
index 8b8ee515ef30b9..e19627fac1efad 100644
--- a/src/coreclr/inc/switches.h
+++ b/src/coreclr/inc/switches.h
@@ -74,10 +74,7 @@
#endif
// ALLOW_SXS_JIT enables AltJit support for JIT-ing, via DOTNET_AltJit / DOTNET_AltJitName.
-// ALLOW_SXS_JIT_NGEN enables AltJit support for NGEN, via DOTNET_AltJitNgen / DOTNET_AltJitName.
-// Note that if ALLOW_SXS_JIT_NGEN is defined, then ALLOW_SXS_JIT must be defined.
#define ALLOW_SXS_JIT
-#define ALLOW_SXS_JIT_NGEN
#if !defined(TARGET_UNIX)
// PLATFORM_SUPPORTS_THREADSUSPEND is defined for platforms where it is safe to call
@@ -168,9 +165,3 @@
#endif
#define FEATURE_MINIMETADATA_IN_TRIAGEDUMPS
-
-// If defined, support interpretation.
-
-#if !defined(TARGET_UNIX)
-#define FEATURE_STACK_SAMPLING
-#endif // defined (ALLOW_SXS_JIT)
diff --git a/src/coreclr/jit/block.cpp b/src/coreclr/jit/block.cpp
index 7c57df63822c7c..5284e747092bd1 100644
--- a/src/coreclr/jit/block.cpp
+++ b/src/coreclr/jit/block.cpp
@@ -68,91 +68,41 @@ unsigned SsaStressHashHelper()
}
#endif
-EHSuccessorIterPosition::EHSuccessorIterPosition(Compiler* comp, BasicBlock* block)
- : m_remainingRegSuccs(block->NumSucc(comp))
- , m_numRegSuccs(m_remainingRegSuccs)
- , m_curRegSucc(nullptr)
- , m_curTry(comp->ehGetBlockExnFlowDsc(block))
+//------------------------------------------------------------------------
+// AllSuccessorEnumerator: Construct an instance of the enumerator.
+//
+// Arguments:
+// comp - Compiler instance
+// block - The block whose successors are to be iterated
+//
+AllSuccessorEnumerator::AllSuccessorEnumerator(Compiler* comp, BasicBlock* block) : m_block(block)
{
- // If "block" is a "leave helper" block (the empty BBJ_ALWAYS block that pairs with a
- // preceding BBJ_CALLFINALLY block to implement a "leave" IL instruction), then no exceptions
- // can occur within it, so clear m_curTry if it's non-null.
- if (m_curTry != nullptr)
- {
- if (block->isBBCallAlwaysPairTail())
+ m_numSuccs = 0;
+ block->VisitAllSuccs(comp, [this](BasicBlock* succ) {
+ if (m_numSuccs < ArrLen(m_successors))
{
- m_curTry = nullptr;
+ m_successors[m_numSuccs] = succ;
}
- }
-
- if (m_curTry == nullptr && m_remainingRegSuccs > 0)
- {
- // Examine the successors to see if any are the start of try blocks.
- FindNextRegSuccTry(comp, block);
- }
-}
-void EHSuccessorIterPosition::FindNextRegSuccTry(Compiler* comp, BasicBlock* block)
-{
- assert(m_curTry == nullptr);
+ m_numSuccs++;
+ return BasicBlockVisit::Continue;
+ });
- // Must now consider the next regular successor, if any.
- while (m_remainingRegSuccs > 0)
+ if (m_numSuccs > ArrLen(m_successors))
{
- m_curRegSucc = block->GetSucc(m_numRegSuccs - m_remainingRegSuccs, comp);
- m_remainingRegSuccs--;
- if (comp->bbIsTryBeg(m_curRegSucc))
- {
- assert(m_curRegSucc->hasTryIndex()); // Since it is a try begin.
- unsigned newTryIndex = m_curRegSucc->getTryIndex();
+ m_pSuccessors = new (comp, CMK_BasicBlock) BasicBlock*[m_numSuccs];
- // If the try region started by "m_curRegSucc" (represented by newTryIndex) contains m_block,
- // we've already yielded its handler, as one of the EH handler successors of m_block itself.
- if (comp->bbInExnFlowRegions(newTryIndex, block))
- {
- continue;
- }
+ unsigned numSuccs = 0;
+ block->VisitAllSuccs(comp, [this, &numSuccs](BasicBlock* succ) {
+ assert(numSuccs < m_numSuccs);
+ m_pSuccessors[numSuccs++] = succ;
+ return BasicBlockVisit::Continue;
+ });
- // Otherwise, consider this try.
- m_curTry = comp->ehGetDsc(newTryIndex);
- break;
- }
+ assert(numSuccs == m_numSuccs);
}
}
-void EHSuccessorIterPosition::Advance(Compiler* comp, BasicBlock* block)
-{
- assert(m_curTry != nullptr);
- if (m_curTry->ebdEnclosingTryIndex != EHblkDsc::NO_ENCLOSING_INDEX)
- {
- m_curTry = comp->ehGetDsc(m_curTry->ebdEnclosingTryIndex);
-
- // If we've gone over into considering try's containing successors,
- // then the enclosing try must have the successor as its first block.
- if (m_curRegSucc == nullptr || m_curTry->ebdTryBeg == m_curRegSucc)
- {
- return;
- }
-
- // Otherwise, give up, try the next regular successor.
- m_curTry = nullptr;
- }
- else
- {
- m_curTry = nullptr;
- }
-
- // We've exhausted all try blocks.
- // See if there are any remaining regular successors that start try blocks.
- FindNextRegSuccTry(comp, block);
-}
-
-BasicBlock* EHSuccessorIterPosition::Current(Compiler* comp, BasicBlock* block)
-{
- assert(m_curTry != nullptr);
- return m_curTry->ExFlowBlock();
-}
-
FlowEdge* Compiler::BlockPredsWithEH(BasicBlock* blk)
{
BlockToFlowEdgeMap* ehPreds = GetBlockToEHPreds();
diff --git a/src/coreclr/jit/block.h b/src/coreclr/jit/block.h
index 425c5d9d5a892b..2cf6c1518c8d34 100644
--- a/src/coreclr/jit/block.h
+++ b/src/coreclr/jit/block.h
@@ -199,133 +199,6 @@ struct allMemoryKinds
}
};
-// This encapsulates the "exception handling" successors of a block. That is,
-// if a basic block BB1 occurs in a try block, we consider the first basic block
-// BB2 of the corresponding handler to be an "EH successor" of BB1. Because we
-// make the conservative assumption that control flow can jump from a try block
-// to its handler at any time, the immediate (regular control flow)
-// predecessor(s) of the first block of a try block are also considered to
-// have the first block of the handler as an EH successor. This makes variables that
-// are "live-in" to the handler become "live-out" for these try-predecessor block,
-// so that they become live-in to the try -- which we require.
-//
-// This class maintains the minimum amount of state necessary to implement
-// successor iteration. The basic block whose successors are enumerated and
-// the compiler need to be provided by Advance/Current's callers. In addition
-// to iterators, this allows the use of other approaches that are more space
-// efficient.
-class EHSuccessorIterPosition
-{
- // The number of "regular" (i.e., non-exceptional) successors that remain to
- // be considered. If BB1 has successor BB2, and BB2 is the first block of a
- // try block, then we consider the catch block of BB2's try to be an EH
- // successor of BB1. This captures the iteration over the successors of BB1
- // for this purpose. (In reverse order; we're done when this field is 0).
- unsigned m_remainingRegSuccs;
- unsigned m_numRegSuccs;
-
- // The current "regular" successor of "m_block" that we're considering.
- BasicBlock* m_curRegSucc;
-
- // The current try block. If non-null, then the current successor "m_curRegSucc"
- // is the first block of the handler of this block. While this try block has
- // enclosing try's that also start with "m_curRegSucc", the corresponding handlers will be
- // further EH successors.
- EHblkDsc* m_curTry;
-
- // Requires that "m_curTry" is NULL. Determines whether there is, as
- // discussed just above, a regular successor that's the first block of a
- // try; if so, sets "m_curTry" to that try block. (As noted above, selecting
- // the try containing the current regular successor as the "current try" may cause
- // multiple first-blocks of catches to be yielded as EH successors: trys enclosing
- // the current try are also included if they also start with the current EH successor.)
- void FindNextRegSuccTry(Compiler* comp, BasicBlock* block);
-
-public:
- // Constructs a position that "points" to the first EH successor of `block`.
- EHSuccessorIterPosition(Compiler* comp, BasicBlock* block);
-
- // Constructs a position that "points" past the last EH successor of `block` ("end" position).
- EHSuccessorIterPosition() : m_remainingRegSuccs(0), m_numRegSuccs(0), m_curTry(nullptr)
- {
- }
-
- // Go on to the next EH successor.
- void Advance(Compiler* comp, BasicBlock* block);
-
- // Returns the current EH successor.
- // Requires that "*this" is not equal to the "end" position.
- BasicBlock* Current(Compiler* comp, BasicBlock* block);
-
- // Returns "true" iff "*this" is equal to "ehsi".
- bool operator==(const EHSuccessorIterPosition& ehsi)
- {
- return m_curTry == ehsi.m_curTry && m_remainingRegSuccs == ehsi.m_remainingRegSuccs;
- }
-
- bool operator!=(const EHSuccessorIterPosition& ehsi)
- {
- return !((*this) == ehsi);
- }
-};
-
-// Yields both normal and EH successors (in that order) in one iteration.
-//
-// This class maintains the minimum amount of state necessary to implement
-// successor iteration. The basic block whose successors are enumerated and
-// the compiler need to be provided by Advance/Current's callers. In addition
-// to iterators, this allows the use of other approaches that are more space
-// efficient.
-class AllSuccessorIterPosition
-{
- // Normal successor position
- unsigned m_numNormSuccs;
- unsigned m_remainingNormSucc;
- // EH successor position
- EHSuccessorIterPosition m_ehIter;
-
- // True iff m_blk is a BBJ_CALLFINALLY block, and the current try block of m_ehIter,
- // the first block of whose handler would be next yielded, is the jump target of m_blk.
- inline bool CurTryIsBlkCallFinallyTarget(Compiler* comp, BasicBlock* block);
-
-public:
- // Constructs a position that "points" to the first successor of `block`.
- inline AllSuccessorIterPosition(Compiler* comp, BasicBlock* block);
-
- // Constructs a position that "points" past the last successor of `block` ("end" position).
- AllSuccessorIterPosition() : m_remainingNormSucc(0), m_ehIter()
- {
- }
-
- // Go on to the next successor.
- inline void Advance(Compiler* comp, BasicBlock* block);
-
- // Returns the current successor.
- // Requires that "*this" is not equal to the "end" position.
- inline BasicBlock* Current(Compiler* comp, BasicBlock* block);
-
- bool IsCurrentEH()
- {
- return m_remainingNormSucc == 0;
- }
-
- bool HasCurrent()
- {
- return *this != AllSuccessorIterPosition();
- }
-
- // Returns "true" iff "*this" is equal to "asi".
- bool operator==(const AllSuccessorIterPosition& asi)
- {
- return (m_remainingNormSucc == asi.m_remainingNormSucc) && (m_ehIter == asi.m_ehIter);
- }
-
- bool operator!=(const AllSuccessorIterPosition& asi)
- {
- return !((*this) == asi);
- }
-};
-
// PredEdgeList: adapter class for forward iteration of the predecessor edge linked list using range-based `for`,
// normally used via BasicBlock::PredEdges(), e.g.:
// for (FlowEdge* const edge : block->PredEdges()) ...
@@ -1347,16 +1220,6 @@ struct BasicBlock : private LIR::Range
}
};
- Successors GetEHSuccs(Compiler* comp)
- {
- return Successors(comp, this);
- }
-
- Successors GetAllSuccs(Compiler* comp)
- {
- return Successors(comp, this);
- }
-
template
BasicBlockVisit VisitAllSuccs(Compiler* comp, TFunc func);
@@ -1988,72 +1851,25 @@ inline PredBlockList::iterator& PredBlockList::iterator::operator++()
void* emitCodeGetCookie(BasicBlock* block);
-AllSuccessorIterPosition::AllSuccessorIterPosition(Compiler* comp, BasicBlock* block)
- : m_numNormSuccs(block->NumSucc(comp)), m_remainingNormSucc(m_numNormSuccs), m_ehIter(comp, block)
-{
- if (CurTryIsBlkCallFinallyTarget(comp, block))
- {
- m_ehIter.Advance(comp, block);
- }
-}
-
-bool AllSuccessorIterPosition::CurTryIsBlkCallFinallyTarget(Compiler* comp, BasicBlock* block)
-{
- return (block->bbJumpKind == BBJ_CALLFINALLY) && (m_ehIter != EHSuccessorIterPosition()) &&
- (block->bbJumpDest == m_ehIter.Current(comp, block));
-}
-
-void AllSuccessorIterPosition::Advance(Compiler* comp, BasicBlock* block)
-{
- if (m_remainingNormSucc > 0)
- {
- m_remainingNormSucc--;
- }
- else
- {
- m_ehIter.Advance(comp, block);
-
- // If the original block whose successors we're iterating over
- // is a BBJ_CALLFINALLY, that finally clause's first block
- // will be yielded as a normal successor. Don't also yield as
- // an exceptional successor.
- if (CurTryIsBlkCallFinallyTarget(comp, block))
- {
- m_ehIter.Advance(comp, block);
- }
- }
-}
-
-// Requires that "this" is not equal to the standard "end" iterator. Returns the
-// current successor.
-BasicBlock* AllSuccessorIterPosition::Current(Compiler* comp, BasicBlock* block)
-{
- if (m_remainingNormSucc > 0)
- {
- return block->GetSucc(m_numNormSuccs - m_remainingNormSucc, comp);
- }
- else
- {
- return m_ehIter.Current(comp, block);
- }
-}
-
-typedef BasicBlock::Successors::iterator EHSuccessorIter;
-typedef BasicBlock::Successors::iterator AllSuccessorIter;
-
// An enumerator of a block's all successors. In some cases (e.g. SsaBuilder::TopologicalSort)
// using iterators is not exactly efficient, at least because they contain an unnecessary
// member - a pointer to the Compiler object.
class AllSuccessorEnumerator
{
- BasicBlock* m_block;
- AllSuccessorIterPosition m_pos;
+ BasicBlock* m_block;
+ union {
+ // We store up to 4 successors inline in the enumerator. For ASP.NET
+ // and libraries.pmi this is enough in 99.7% of cases.
+ BasicBlock* m_successors[4];
+ BasicBlock** m_pSuccessors;
+ };
+
+ unsigned m_numSuccs;
+ unsigned m_curSucc = UINT_MAX;
public:
// Constructs an enumerator of all `block`'s successors.
- AllSuccessorEnumerator(Compiler* comp, BasicBlock* block) : m_block(block), m_pos(comp, block)
- {
- }
+ AllSuccessorEnumerator(Compiler* comp, BasicBlock* block);
// Gets the block whose successors are enumerated.
BasicBlock* Block()
@@ -2061,23 +1877,21 @@ class AllSuccessorEnumerator
return m_block;
}
- // Returns true if the next successor is an EH successor.
- bool IsNextEHSuccessor()
- {
- return m_pos.IsCurrentEH();
- }
-
// Returns the next available successor or `nullptr` if there are no more successors.
BasicBlock* NextSuccessor(Compiler* comp)
{
- if (!m_pos.HasCurrent())
+ m_curSucc++;
+ if (m_curSucc >= m_numSuccs)
{
return nullptr;
}
- BasicBlock* succ = m_pos.Current(comp, m_block);
- m_pos.Advance(comp, m_block);
- return succ;
+ if (m_numSuccs <= ArrLen(m_successors))
+ {
+ return m_successors[m_curSucc];
+ }
+
+ return m_pSuccessors[m_curSucc];
}
};
diff --git a/src/coreclr/jit/codegen.h b/src/coreclr/jit/codegen.h
index 5d89ef8d01832c..945b082a2dfb39 100644
--- a/src/coreclr/jit/codegen.h
+++ b/src/coreclr/jit/codegen.h
@@ -257,7 +257,7 @@ class CodeGen final : public CodeGenInterface
void genFnPrologCalleeRegArgs(regNumber xtraReg, bool* pXtraRegClobbered, RegState* regState);
#endif
void genEnregisterIncomingStackArgs();
-#if defined(TARGET_ARM64)
+#if defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
void genEnregisterOSRArgsAndLocals(regNumber initReg, bool* pInitRegZeroed);
#else
void genEnregisterOSRArgsAndLocals();
@@ -423,11 +423,11 @@ class CodeGen final : public CodeGenInterface
regMaskTP fiSaveRegs; // Set of callee-saved registers saved in the funclet prolog (includes RA)
int fiFunction_CallerSP_to_FP_delta; // Delta between caller SP and the frame pointer in the parent function
// (negative)
- int fiSP_to_FPRA_save_delta; // FP/RA register save offset from SP (positive)
+ int fiSP_to_CalleeSaved_delta; // CalleeSaved register save offset from SP (positive)
+ int fiCalleeSavedPadding; // CalleeSaved offset padding (positive)
int fiSP_to_PSP_slot_delta; // PSP slot offset from SP (positive)
int fiCallerSP_to_PSP_slot_delta; // PSP slot offset from Caller SP (negative)
- int fiFrameType; // Funclet frame types are numbered. See genFuncletProlog() for details.
- int fiSpDelta1; // Stack pointer delta 1 (negative)
+ int fiSpDelta; // Stack pointer delta (negative)
};
FuncletFrameInfoDsc genFuncletInfo;
diff --git a/src/coreclr/jit/codegencommon.cpp b/src/coreclr/jit/codegencommon.cpp
index 0d0aa1a2ed2999..eaf16b49be7ee3 100644
--- a/src/coreclr/jit/codegencommon.cpp
+++ b/src/coreclr/jit/codegencommon.cpp
@@ -4655,7 +4655,7 @@ void CodeGen::genZeroInitFrame(int untrLclHi, int untrLclLo, regNumber initReg,
// initReg -- scratch register to use if needed
// pInitRegZeroed -- [IN,OUT] if init reg is zero (on entry/exit)
//
-#if defined(TARGET_ARM64)
+#if defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
void CodeGen::genEnregisterOSRArgsAndLocals(regNumber initReg, bool* pInitRegZeroed)
#else
void CodeGen::genEnregisterOSRArgsAndLocals()
@@ -4796,7 +4796,7 @@ void CodeGen::genEnregisterOSRArgsAndLocals()
GetEmitter()->emitIns_R_AR(ins_Load(lclTyp), size, varDsc->GetRegNum(), genFramePointerReg(), offset);
-#elif defined(TARGET_ARM64)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// Patchpoint offset is from top of Tier0 frame
//
@@ -5436,7 +5436,7 @@ void CodeGen::genFnProlog()
psiBegProlog();
}
-#if defined(TARGET_ARM64)
+#if defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// For arm64 OSR, emit a "phantom prolog" to account for the actions taken
// in the tier0 frame that impact FP and SP on entry to the OSR method.
//
@@ -5451,7 +5451,7 @@ void CodeGen::genFnProlog()
// SP is tier0 method's SP.
compiler->unwindAllocStack(tier0FrameSize);
}
-#endif // defined(TARGET_ARM64)
+#endif // defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
#ifdef DEBUG
@@ -5774,6 +5774,14 @@ void CodeGen::genFnProlog()
{
initReg = REG_IP1;
}
+#elif defined(TARGET_LOONGARCH64)
+ // For loongarch64 OSR root frames, we may need a scratch register for large
+ // offset addresses. Use a register that won't be allocated.
+ //
+ if (isRoot && compiler->opts.IsOSR())
+ {
+ initReg = REG_SCRATCH;
+ }
#endif
noway_assert(!compiler->compMethodRequiresPInvokeFrame() || (initReg != REG_PINVOKE_FRAME));
@@ -6087,7 +6095,7 @@ void CodeGen::genFnProlog()
// Otherwise we'll do some of these fetches twice.
//
CLANG_FORMAT_COMMENT_ANCHOR;
-#if defined(TARGET_ARM64)
+#if defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
genEnregisterOSRArgsAndLocals(initReg, &initRegZeroed);
#else
genEnregisterOSRArgsAndLocals();
diff --git a/src/coreclr/jit/codegeninterface.h b/src/coreclr/jit/codegeninterface.h
index 1f8f4c6b0c7fe4..c2bcedb8ea9b7e 100644
--- a/src/coreclr/jit/codegeninterface.h
+++ b/src/coreclr/jit/codegeninterface.h
@@ -127,7 +127,9 @@ class CodeGenInterface
#define INST_FP 0x01 // is it a FP instruction?
public:
static bool instIsFP(instruction ins);
-
+#if defined(TARGET_XARCH)
+ static bool instIsEmbeddedBroadcastCompatible(instruction ins);
+#endif // TARGET_XARCH
//-------------------------------------------------------------------------
// Liveness-related fields & methods
public:
@@ -764,6 +766,10 @@ class CodeGenInterface
virtual const char* siStackVarName(size_t offs, size_t size, unsigned reg, unsigned stkOffs) = 0;
#endif // LATE_DISASM
+
+#if defined(TARGET_XARCH)
+ bool IsEmbeddedBroadcastEnabled(instruction ins, GenTree* op);
+#endif
};
#endif // _CODEGEN_INTERFACE_H_
diff --git a/src/coreclr/jit/codegenloongarch64.cpp b/src/coreclr/jit/codegenloongarch64.cpp
index 1617b7d24262ae..3135507a05c217 100644
--- a/src/coreclr/jit/codegenloongarch64.cpp
+++ b/src/coreclr/jit/codegenloongarch64.cpp
@@ -20,6 +20,7 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#include "lower.h"
#include "gcinfo.h"
#include "gcinfoencoder.h"
+#include "patchpointinfo.h"
/*
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
@@ -271,7 +272,7 @@ void CodeGen::genPrologSaveReg(regNumber reg1, int spOffset, int spDelta, regNum
if (spDelta != 0)
{
- // generate daddiu SP,SP,-imm
+ // generate addi.d SP,SP,-imm
genStackPointerAdjustment(spDelta, tmpReg, pTmpRegIsZero, /* reportUnwindData */ true);
}
@@ -330,7 +331,7 @@ void CodeGen::genEpilogRestoreRegPair(regNumber reg1,
GetEmitter()->emitIns_R_R_I(ins, EA_PTRSIZE, reg1, REG_SPBASE, spOffset);
compiler->unwindSaveReg(reg1, spOffset);
- // generate daddiu SP,SP,imm
+ // generate addi.d SP,SP,imm
genStackPointerAdjustment(spDelta, tmpReg, pTmpRegIsZero, /* reportUnwindData */ true);
}
else
@@ -372,11 +373,11 @@ void CodeGen::genEpilogRestoreReg(regNumber reg1, int spOffset, int spDelta, reg
if (spDelta != 0)
{
- // ld reg1, offset(SP)
+ // ld.d reg1,SP,offset
GetEmitter()->emitIns_R_R_I(ins, EA_PTRSIZE, reg1, REG_SPBASE, spOffset);
compiler->unwindSaveReg(reg1, spOffset);
- // generate add SP,SP,imm
+ // generate addi.d SP,SP,imm
genStackPointerAdjustment(spDelta, tmpReg, pTmpRegIsZero, /* reportUnwindData */ true);
}
else
@@ -675,25 +676,15 @@ void CodeGen::genRestoreCalleeSavedRegisterGroup(regMaskTP regsMask, int spDelta
// zero).
//
// Here's an example restore sequence:
-// ld s7, 88(sp)
-// ld s6, 80(sp)
-// ld s5, 72(sp)
-// ld s4, 64(sp)
-// ld s3, 56(sp)
-// ld s2, 48(sp)
-// ld s1, 40(sp)
-// ld s0, 32(sp)
-//
-// For the case of non-zero spDelta, we assume the base of the callee-save registers to restore is at SP, and
-// the last restore adjusts SP by the specified amount. For example:
-// ld s7, 56(sp)
-// ld s6, 48(sp)
-// ld s5, 40(sp)
-// ld s4, 32(sp)
-// ld s3, 24(sp)
-// ld s2, 16(sp)
-// ld s1, 88(sp)
-// ld s0, 80(sp)
+// ld.d s8,sp,#xxx
+// ld.d s7,sp,#xxx
+// ld.d s6,sp,#xxx
+// ld.d s5,sp,#xxx
+// ld.d s4,sp,#xxx
+// ld.d s3,sp,#xxx
+// ld.d s2,sp,#xxx
+// ld.d s1,sp,#xxx
+// ld.d s0,sp,#xxx
//
// Note you call the unwind functions specifying the prolog operation that is being un-done. So, for example, when
// generating a post-indexed load, you call the unwind function for specifying the corresponding preindexed store.
@@ -754,142 +745,45 @@ void CodeGen::genRestoreCalleeSavedRegistersHelp(regMaskTP regsToRestoreMask, in
*
* Funclets have the following incoming arguments:
*
- * catch: a0 = the exception object that was caught (see GT_CATCH_ARG)
- * filter: a0 = the exception object to filter (see GT_CATCH_ARG), a1 = CallerSP of the containing function
- * finally/fault: none
+ * catch: a0 = the exception object that was caught (see GT_CATCH_ARG)
+ * filter: a0 = the exception object to filter (see GT_CATCH_ARG), a1 = CallerSP of the containing function
+ * finally/fault: none
*
* Funclets set the following registers on exit:
*
- * catch: v0 = the address at which execution should resume (see BBJ_EHCATCHRET)
- * filter: v0 = non-zero if the handler should handle the exception, zero otherwise (see GT_RETFILT)
- * finally/fault: none
+ * catch: a0 = the address at which execution should resume (see BBJ_EHCATCHRET)
+ * filter: a0 = non-zero if the handler should handle the exception, zero otherwise (see GT_RETFILT)
+ * finally/fault: none
*
- * The LOONGARCH64 funclet prolog sequence is one of the following (Note: #framesz is total funclet frame size,
+ * The LOONGARCH64 funclet prolog is the following (Note: #framesz is total funclet frame size,
* including everything; #outsz is outgoing argument space. #framesz must be a multiple of 16):
*
- * Frame type 1:
- * For #framesz <= 32760 and FP/RA at bottom:
- * daddiu sp,sp,-#framesz ; establish the frame (predecrement by #framesz), save FP/RA
- * sd fp,#outsz(sp)
- * sd ra,#outsz+8(sp)
- * sd s0,#xxx-8(sp) ; save callee-saved registers, as necessary
- * sd s1,#xxx(sp)
+ * Frame type liking:
+ * addi_d sp,sp,-#framesz ; establish the frame, maybe multi-instruction.
+ * st.d s0,sp,#outsz ; save callee-saved registers, as necessary
+ * st.d s1,sp,#outsz+8
+ * st.d ra,sp,#outsz+?
+ * st.d fp,sp,#outsz+?
*
- * The funclet frame is thus:
+ * The funclet frame layout:
*
* | |
* |-----------------------|
* | incoming arguments |
* +=======================+ <---- Caller's SP
- * | Varargs regs space | // Only for varargs main functions; 64 bytes
- * |-----------------------|
- * |Callee saved registers | // multiple of 8 bytes
- * |-----------------------|
- * | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
- * |-----------------------|
- * ~ alignment padding ~ // To make the whole frame 16 byte aligned.
+ * | OSR padding | // If required
* |-----------------------|
- * | Saved FP, RA | // 16 bytes
- * |-----------------------|
- * | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
- * |-----------------------| <---- Ambient SP
- * | | |
- * ~ | Stack grows ~
- * | | downward |
- * V
- *
- * Frame type 2:
- * For #framesz <= 32760 and FP/RA at top:
- * daddiu sp,sp,-#framesz ; establish the frame
- * sd s0,xxx(sp) ; save callee-saved registers, as necessary
- * sd s1,xxx+8(sp)
- * sd s?,xxx+?(sp)
- * sd fp,xxx+?(sp) ; save FP/RA.
- * sd ra,xxx+?(sp)
- *
- * The funclet frame is thus:
- *
- * | |
- * |-----------------------|
- * | incoming arguments |
- * +=======================+ <---- Caller's SP
* | Varargs regs space | // Only for varargs main functions; 64 bytes
* |-----------------------|
- * | Saved FP, RA | // 16 bytes
- * |-----------------------|
- * |Callee saved registers | // multiple of 8 bytes
+ * | MonitorAcquired | // 8 bytes; for synchronized methods
* |-----------------------|
* | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
* |-----------------------|
- * ~ alignment padding ~ // To make the whole frame 16 byte aligned.
- * |-----------------------|
- * | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
- * |-----------------------| <---- Ambient SP
- * | | |
- * ~ | Stack grows ~
- * | | downward |
- * V
- *
- * Frame type 3:
- * For #framesz > 32760 and FP/RA at bottom:
- * ; for funclet, #framesz-#outsz will be less than 32760.
- *
- * daddiu sp,sp,-(#framesz-#FPRA_delta) ; note maybe 16byte-alignment.
- * sd fp, pad(sp) ; pad is depended on stack-16byte-alignment..
- * sd ra, pad+8(sp)
- * sd s0,#xxx(sp) ; save callee-saved registers, as necessary,
- * sd s1,#xxx+8(sp)
- * daddiu sp,sp,-#outsz ; create space for outgoing argument space, maybe 16byte-alignment.
- *
- * The funclet frame is thus:
- *
- * | |
- * |-----------------------|
- * | incoming arguments |
- * +=======================+ <---- Caller's SP
- * | Varargs regs space | // Only for varargs main functions; 64 bytes
- * |-----------------------|
- * |Callee saved registers | // multiple of 8 bytes
- * |-----------------------|
- * | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
- * |-----------------------|
- * ~ alignment padding ~
+ * ~ alignment padding ~ // To make the whole frame 16 byte aligned
* |-----------------------|
* | Saved FP, RA | // 16 bytes
* |-----------------------|
- * | Outgoing arg space | // multiple of 8 bytes
- * |-----------------------| <---- Ambient SP
- * | | |
- * ~ | Stack grows ~
- * | | downward |
- * V
- *
- * Frame type 4:
- * For #framesz > 32760 and FP/RA at top:
- * daddiu sp,sp,-#framesz+PSP_offset ; establish the frame, maybe 16byte-alignment.
- * sd s0,xxx(sp) ; save callee-saved registers, as necessary
- * sd s1,xxx+8(sp)
- * sd s?,xxx+?(sp)
- * sd fp,xxx+?(sp) ; save FP/RA.
- * sd ra,xxx+?(sp)
- *
- * daddiu sp,sp,-#PSP_offset ; establish the frame, maybe 16byte-alignment.
- *
- * The funclet frame is thus:
- *
- * | |
- * |-----------------------|
- * | incoming arguments |
- * +=======================+ <---- Caller's SP
- * | Varargs regs space | // Only for varargs main functions; 64 bytes
- * |-----------------------|
- * | Saved FP, RA | // 16 bytes
- * |-----------------------|
- * |Callee saved registers | // multiple of 8 bytes
- * |-----------------------|
- * | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
- * |-----------------------|
- * ~ alignment padding ~ // To make the whole frame 16 byte aligned.
+ * |Callee saved registers | // multiple of 8 bytes, not includting FP/RA
* |-----------------------|
* | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
* |-----------------------| <---- Ambient SP
@@ -906,10 +800,9 @@ void CodeGen::genRestoreCalleeSavedRegistersHelp(regMaskTP regsToRestoreMask, in
* FP,RA registers
* 9 int callee-saved register s0-s8
* 8 float callee-saved registers f24-f31
- * 8 saved integer argument registers a0-a7, if varargs function
+ * 8 saved integer argument registers a0-a7, if varargs function support.
* 1 PSP slot
- * 1 alignment slot, future maybe add gp
- * == 29 slots * 8 bytes = 232 bytes.
+ * == 20 slots * 8 bytes = 160 bytes.
*
* The outgoing argument size, however, can be very large, if we call a function that takes a large number of
* arguments (note that we currently use the same outgoing argument space size in the funclet as for the main
@@ -956,28 +849,28 @@ void CodeGen::genRestoreCalleeSavedRegistersHelp(regMaskTP regsToRestoreMask, in
* // be a funclet or main function. We won't know any time there is a filter protecting nested EH. To simplify, we just always
* // create a main function PSP for any function with a filter.
*
- * ld a1, CallerSP_to_PSP_slot_delta(a1) ; Load the CallerSP of the main function (stored in the PSP of the dynamically containing funclet or function)
- * sd a1, SP_to_PSP_slot_delta(sp) ; store the PSP
- * daddiu fp, a1, Function_CallerSP_to_FP_delta ; re-establish the frame pointer
+ * ld.d a1,a1, CallerSP_to_PSP_slot_delta ; Load the CallerSP of the main function (stored in the PSP of the dynamically containing funclet or function)
+ * st.d a1,sp, SP_to_PSP_slot_delta ; store the PSP
+ * addi.d fp, a1, Function_CallerSP_to_FP_delta ; re-establish the frame pointer
* }
* else
* {
* // This is NOT a filter funclet. The VM re-establishes the frame pointer on entry.
* // TODO-LOONGARCH64-CQ: if VM set a1 to CallerSP on entry, like for filters, we could save an instruction.
*
- * daddiu a3, fp, Function_FP_to_CallerSP_delta ; compute the CallerSP, given the frame pointer. a3 is scratch?
- * sd a3, SP_to_PSP_slot_delta(sp) ; store the PSP
+ * addi.d a3,fp,Function_FP_to_CallerSP_delta ; compute the CallerSP, given the frame pointer. a3 is scratch?
+ * st.d a3,sp,SP_to_PSP_slot_delta ; store the PSP
* }
*
* An example epilog sequence is then:
*
- * daddiu sp,sp,#outsz ; if any outgoing argument space
+ * addi.d sp,sp,#outsz ; if any outgoing argument space
* ... ; restore callee-saved registers
- * ld s0,#xxx-8(sp)
- * ld s1,#xxx(sp)
- * ld fp,#framesz-8(sp)
- * ld ra,#framesz(sp)
- * daddiu sp,sp,#framesz
+ * ld.d s0,sp,#xxx-8
+ * ld.d s1,sp,#xxx
+ * ld.d ra,sp,#xxx+?
+ * ld.d fp,sp,#xxx+?
+ * addi.d sp,sp,#framesz
* jr ra
*
*/
@@ -999,15 +892,9 @@ void CodeGen::genFuncletProlog(BasicBlock* block)
compiler->unwindBegProlog();
- regMaskTP maskSaveRegsFloat = genFuncletInfo.fiSaveRegs & RBM_ALLFLOAT;
- regMaskTP maskSaveRegsInt = genFuncletInfo.fiSaveRegs & ~maskSaveRegsFloat;
-
- // Funclets must always save RA and FP, since when we have funclets we must have an FP frame.
- assert((maskSaveRegsInt & RBM_RA) != 0);
- assert((maskSaveRegsInt & RBM_FP) != 0);
-
bool isFilter = (block->bbCatchTyp == BBCT_FILTER);
- int frameSize = genFuncletInfo.fiSpDelta1;
+ int frameSize = genFuncletInfo.fiSpDelta;
+ assert(frameSize < 0);
regMaskTP maskArgRegsLiveIn;
if (isFilter)
@@ -1023,62 +910,42 @@ void CodeGen::genFuncletProlog(BasicBlock* block)
maskArgRegsLiveIn = RBM_A0;
}
-#ifdef DEBUG
- if (compiler->opts.disAsm)
- {
- printf("DEBUG: CodeGen::genFuncletProlog, frameType:%d\n\n", genFuncletInfo.fiFrameType);
- }
-#endif
+ regMaskTP maskSaveRegs = genFuncletInfo.fiSaveRegs & RBM_CALLEE_SAVED;
+ int regsSavedSize = (compiler->compCalleeRegsPushed - 2) << 3;
- int offset = 0;
- if (genFuncletInfo.fiFrameType == 1)
+ int SP_to_CalleeSaved_delta = genFuncletInfo.fiSP_to_CalleeSaved_delta;
+ if ((SP_to_CalleeSaved_delta + regsSavedSize + genFuncletInfo.fiCalleeSavedPadding) <= 2040)
{
- // fiFrameType constraints:
- assert(frameSize < 0);
- assert(frameSize >= -2048);
+ SP_to_CalleeSaved_delta += genFuncletInfo.fiCalleeSavedPadding;
- assert(genFuncletInfo.fiSP_to_FPRA_save_delta < 2040);
genStackPointerAdjustment(frameSize, REG_R21, nullptr, /* reportUnwindData */ true);
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, genFuncletInfo.fiSP_to_FPRA_save_delta);
- compiler->unwindSaveReg(REG_FP, genFuncletInfo.fiSP_to_FPRA_save_delta);
+ genSaveCalleeSavedRegistersHelp(maskSaveRegs, SP_to_CalleeSaved_delta, 0);
+ SP_to_CalleeSaved_delta += regsSavedSize;
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE,
- genFuncletInfo.fiSP_to_FPRA_save_delta + 8);
- compiler->unwindSaveReg(REG_RA, genFuncletInfo.fiSP_to_FPRA_save_delta + 8);
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, SP_to_CalleeSaved_delta);
+ compiler->unwindSaveReg(REG_RA, SP_to_CalleeSaved_delta);
- maskSaveRegsInt &= ~(RBM_RA | RBM_FP); // We've saved these now
-
- genSaveCalleeSavedRegistersHelp(maskSaveRegsInt | maskSaveRegsFloat, genFuncletInfo.fiSP_to_PSP_slot_delta + 8,
- 0);
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, SP_to_CalleeSaved_delta + 8);
+ compiler->unwindSaveReg(REG_FP, SP_to_CalleeSaved_delta + 8);
}
- else if (genFuncletInfo.fiFrameType == 2)
+ else
{
- // fiFrameType constraints:
- assert(frameSize < -2048);
+ assert(frameSize < -2040);
- offset = -frameSize - genFuncletInfo.fiSP_to_FPRA_save_delta;
- int SP_delta = roundUp((UINT)offset, STACK_ALIGN);
- offset = SP_delta - offset;
-
- genStackPointerAdjustment(-SP_delta, REG_R21, nullptr, /* reportUnwindData */ true);
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset);
- compiler->unwindSaveReg(REG_FP, offset);
+ int SP_delta = frameSize + SP_to_CalleeSaved_delta;
+ genStackPointerAdjustment(SP_delta, REG_R21, nullptr, /* reportUnwindData */ true);
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset + 8);
- compiler->unwindSaveReg(REG_RA, offset + 8);
+ genSaveCalleeSavedRegistersHelp(maskSaveRegs, genFuncletInfo.fiCalleeSavedPadding, 0);
+ regsSavedSize += genFuncletInfo.fiCalleeSavedPadding;
- maskSaveRegsInt &= ~(RBM_RA | RBM_FP); // We've saved these now
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, regsSavedSize);
+ compiler->unwindSaveReg(REG_RA, regsSavedSize);
- offset = frameSize + SP_delta + genFuncletInfo.fiSP_to_PSP_slot_delta + 8;
- genSaveCalleeSavedRegistersHelp(maskSaveRegsInt | maskSaveRegsFloat, offset, 0);
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, regsSavedSize + 8);
+ compiler->unwindSaveReg(REG_FP, regsSavedSize + 8);
- genStackPointerAdjustment(frameSize + SP_delta, REG_R21, nullptr, /* reportUnwindData */ true);
- }
- else
- {
- unreached();
+ genStackPointerAdjustment(-SP_to_CalleeSaved_delta, REG_R21, nullptr, /* reportUnwindData */ true);
}
// This is the end of the OS-reported prolog for purposes of unwinding
@@ -1140,84 +1007,47 @@ void CodeGen::genFuncletEpilog()
ScopedSetVariable _setGeneratingEpilog(&compiler->compGeneratingEpilog, true);
- bool unwindStarted = false;
- int frameSize = genFuncletInfo.fiSpDelta1;
-
- if (!unwindStarted)
- {
- // We can delay this until we know we'll generate an unwindable instruction, if necessary.
- compiler->unwindBegEpilog();
- unwindStarted = true;
- }
-
- regMaskTP maskRestoreRegsFloat = genFuncletInfo.fiSaveRegs & RBM_ALLFLOAT;
- regMaskTP maskRestoreRegsInt = genFuncletInfo.fiSaveRegs & ~maskRestoreRegsFloat;
-
- // Funclets must always save RA and FP, since when we have funclets we must have an FP frame.
- assert((maskRestoreRegsInt & RBM_RA) != 0);
- assert((maskRestoreRegsInt & RBM_FP) != 0);
-
-#ifdef DEBUG
- if (compiler->opts.disAsm)
- {
- printf("DEBUG: CodeGen::genFuncletEpilog, frameType:%d\n\n", genFuncletInfo.fiFrameType);
- }
-#endif
-
- regMaskTP regsToRestoreMask = maskRestoreRegsInt | maskRestoreRegsFloat;
+ compiler->unwindBegEpilog();
+ int frameSize = genFuncletInfo.fiSpDelta;
assert(frameSize < 0);
- if (genFuncletInfo.fiFrameType == 1)
- {
- // fiFrameType constraints:
- assert(frameSize >= -2048);
- assert(genFuncletInfo.fiSP_to_FPRA_save_delta < 2040);
- regsToRestoreMask &= ~(RBM_RA | RBM_FP); // We restore FP/RA at the end
+ regMaskTP regsToRestoreMask = genFuncletInfo.fiSaveRegs & RBM_CALLEE_SAVED;
+ int regsRestoreSize = (compiler->compCalleeRegsPushed - 2) << 3;
- genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, genFuncletInfo.fiSP_to_PSP_slot_delta + 8, 0);
+ int SP_to_CalleeSaved_delta = genFuncletInfo.fiSP_to_CalleeSaved_delta;
+ if ((SP_to_CalleeSaved_delta + regsRestoreSize + genFuncletInfo.fiCalleeSavedPadding) <= 2040)
+ {
+ SP_to_CalleeSaved_delta += genFuncletInfo.fiCalleeSavedPadding;
+ genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, SP_to_CalleeSaved_delta, 0);
+ SP_to_CalleeSaved_delta += regsRestoreSize;
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE,
- genFuncletInfo.fiSP_to_FPRA_save_delta + 8);
- compiler->unwindSaveReg(REG_RA, genFuncletInfo.fiSP_to_FPRA_save_delta + 8);
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, SP_to_CalleeSaved_delta);
+ compiler->unwindSaveReg(REG_RA, SP_to_CalleeSaved_delta);
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, genFuncletInfo.fiSP_to_FPRA_save_delta);
- compiler->unwindSaveReg(REG_FP, genFuncletInfo.fiSP_to_FPRA_save_delta);
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, SP_to_CalleeSaved_delta + 8);
+ compiler->unwindSaveReg(REG_FP, SP_to_CalleeSaved_delta + 8);
- // generate daddiu SP,SP,imm
genStackPointerAdjustment(-frameSize, REG_R21, nullptr, /* reportUnwindData */ true);
}
- else if (genFuncletInfo.fiFrameType == 2)
+ else
{
- // fiFrameType constraints:
- assert(frameSize < -2048);
-
- int offset = -frameSize - genFuncletInfo.fiSP_to_FPRA_save_delta;
- int SP_delta = roundUp((UINT)offset, STACK_ALIGN);
- offset = SP_delta - offset;
-
- // first, generate daddiu SP,SP,imm
- genStackPointerAdjustment(-frameSize - SP_delta, REG_R21, nullptr, /* reportUnwindData */ true);
+ assert(frameSize < -2040);
- int offset2 = frameSize + SP_delta + genFuncletInfo.fiSP_to_PSP_slot_delta + 8;
- assert(offset2 < 2040); // can amend.
+ genStackPointerAdjustment(SP_to_CalleeSaved_delta, REG_R21, nullptr, /* reportUnwindData */ true);
- regsToRestoreMask &= ~(RBM_RA | RBM_FP); // We restore FP/RA at the end
- genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, offset2, 0);
+ genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, genFuncletInfo.fiCalleeSavedPadding, 0);
+ regsRestoreSize += genFuncletInfo.fiCalleeSavedPadding;
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset + 8);
- compiler->unwindSaveReg(REG_RA, offset + 8);
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, regsRestoreSize);
+ compiler->unwindSaveReg(REG_RA, regsRestoreSize);
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset);
- compiler->unwindSaveReg(REG_FP, offset);
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, regsRestoreSize + 8);
+ compiler->unwindSaveReg(REG_FP, regsRestoreSize + 8);
- // second, generate daddiu SP,SP,imm for remaine space.
- genStackPointerAdjustment(SP_delta, REG_R21, nullptr, /* reportUnwindData */ true);
- }
- else
- {
- unreached();
+ genStackPointerAdjustment(-frameSize - SP_to_CalleeSaved_delta, REG_R21, nullptr, /* reportUnwindData */ true);
}
+
GetEmitter()->emitIns_R_R_I(INS_jirl, emitActualTypeSize(TYP_I_IMPL), REG_R0, REG_RA, 0);
compiler->unwindReturn(REG_RA);
@@ -1246,61 +1076,62 @@ void CodeGen::genCaptureFuncletPrologEpilogInfo()
// The frame size and offsets must be finalized
assert(compiler->lvaDoneFrameLayout == Compiler::FINAL_FRAME_LAYOUT);
- genFuncletInfo.fiFunction_CallerSP_to_FP_delta = genCallerSPtoFPdelta();
-
regMaskTP rsMaskSaveRegs = regSet.rsMaskCalleeSaved;
assert((rsMaskSaveRegs & RBM_RA) != 0);
assert((rsMaskSaveRegs & RBM_FP) != 0);
unsigned PSPSize = (compiler->lvaPSPSym != BAD_VAR_NUM) ? 8 : 0;
- unsigned saveRegsCount = genCountBits(rsMaskSaveRegs);
- assert((saveRegsCount == compiler->compCalleeRegsPushed) || (saveRegsCount == compiler->compCalleeRegsPushed - 1));
+ // Because a method and funclets must have the same caller-relative PSPSym offset,
+ // if there is a PSPSym, we have to pad the funclet frame size for OSR.
+ //
+ unsigned osrPad = 0;
+ if (compiler->opts.IsOSR() && (PSPSize > 0))
+ {
+ osrPad = compiler->info.compPatchpointInfo->TotalFrameSize();
+
+ // OSR pad must be already aligned to stack size.
+ assert((osrPad % STACK_ALIGN) == 0);
+ }
+
+ genFuncletInfo.fiCalleeSavedPadding = 0;
+ genFuncletInfo.fiFunction_CallerSP_to_FP_delta = genCallerSPtoFPdelta() - osrPad;
- unsigned saveRegsPlusPSPSize =
- roundUp((UINT)genTotalFrameSize(), STACK_ALIGN) - compiler->compLclFrameSize + PSPSize;
+ unsigned regsSavedSize = genCountBits(rsMaskSaveRegs) << 3;
+ assert(genCountBits(rsMaskSaveRegs) == compiler->compCalleeRegsPushed);
- unsigned saveRegsPlusPSPSizeAligned = roundUp(saveRegsPlusPSPSize, STACK_ALIGN);
+ unsigned saveRegsPlusPSPSize = regsSavedSize + PSPSize;
assert(compiler->lvaOutgoingArgSpaceSize % REGSIZE_BYTES == 0);
unsigned outgoingArgSpaceAligned = roundUp(compiler->lvaOutgoingArgSpaceSize, STACK_ALIGN);
- unsigned maxFuncletFrameSizeAligned = saveRegsPlusPSPSizeAligned + outgoingArgSpaceAligned;
- assert((maxFuncletFrameSizeAligned % STACK_ALIGN) == 0);
-
- int SP_to_FPRA_save_delta = compiler->lvaOutgoingArgSpaceSize;
-
- unsigned funcletFrameSize = saveRegsPlusPSPSize + compiler->lvaOutgoingArgSpaceSize;
+ unsigned funcletFrameSize = osrPad + saveRegsPlusPSPSize + compiler->lvaOutgoingArgSpaceSize;
unsigned funcletFrameSizeAligned = roundUp(funcletFrameSize, STACK_ALIGN);
- assert(funcletFrameSizeAligned <= maxFuncletFrameSizeAligned);
- unsigned funcletFrameAlignmentPad = funcletFrameSizeAligned - funcletFrameSize;
- assert((funcletFrameAlignmentPad == 0) || (funcletFrameAlignmentPad == REGSIZE_BYTES));
-
- if (maxFuncletFrameSizeAligned <= (2048 - 8))
+ int SP_to_CalleeSaved_delta = compiler->lvaOutgoingArgSpaceSize;
+ if ((SP_to_CalleeSaved_delta + regsSavedSize) >= 2040)
{
- genFuncletInfo.fiFrameType = 1;
- saveRegsPlusPSPSize -= 2 * 8; // FP/RA
- }
- else
- {
- unsigned saveRegsPlusPSPAlignmentPad = saveRegsPlusPSPSizeAligned - saveRegsPlusPSPSize;
- assert((saveRegsPlusPSPAlignmentPad == 0) || (saveRegsPlusPSPAlignmentPad == REGSIZE_BYTES));
+ int offset = funcletFrameSizeAligned - SP_to_CalleeSaved_delta;
+ SP_to_CalleeSaved_delta = AlignUp((UINT)offset, STACK_ALIGN);
- genFuncletInfo.fiFrameType = 2;
- saveRegsPlusPSPSize -= 2 * 8; // FP/RA
+ genFuncletInfo.fiCalleeSavedPadding = SP_to_CalleeSaved_delta - offset;
}
- int CallerSP_to_PSP_slot_delta = -(int)saveRegsPlusPSPSize;
- genFuncletInfo.fiSpDelta1 = -(int)funcletFrameSizeAligned;
- int SP_to_PSP_slot_delta = funcletFrameSizeAligned - saveRegsPlusPSPSize;
+ if (compiler->lvaMonAcquired != BAD_VAR_NUM && !compiler->opts.IsOSR())
+ {
+ // We furthermore allocate the "monitor acquired" bool between PSP and
+ // the saved registers because this is part of the EnC header.
+ // Note that OSR methods reuse the monitor bool created by tier 0.
+ osrPad += compiler->lvaLclSize(compiler->lvaMonAcquired);
+ }
/* Now save it for future use */
- genFuncletInfo.fiSaveRegs = rsMaskSaveRegs;
- genFuncletInfo.fiSP_to_FPRA_save_delta = SP_to_FPRA_save_delta;
+ genFuncletInfo.fiSpDelta = -(int)funcletFrameSizeAligned;
+ genFuncletInfo.fiSaveRegs = rsMaskSaveRegs;
+ genFuncletInfo.fiSP_to_CalleeSaved_delta = SP_to_CalleeSaved_delta;
- genFuncletInfo.fiSP_to_PSP_slot_delta = SP_to_PSP_slot_delta;
- genFuncletInfo.fiCallerSP_to_PSP_slot_delta = CallerSP_to_PSP_slot_delta;
+ genFuncletInfo.fiSP_to_PSP_slot_delta = funcletFrameSizeAligned - osrPad - 8;
+ genFuncletInfo.fiCallerSP_to_PSP_slot_delta = -(int)osrPad - 8;
#ifdef DEBUG
if (verbose)
@@ -1310,23 +1141,22 @@ void CodeGen::genCaptureFuncletPrologEpilogInfo()
printf(" Save regs: ");
dspRegMask(genFuncletInfo.fiSaveRegs);
printf("\n");
- printf(" Function CallerSP-to-FP delta: %d\n", genFuncletInfo.fiFunction_CallerSP_to_FP_delta);
- printf(" SP to FP/RA save location delta: %d\n", genFuncletInfo.fiSP_to_FPRA_save_delta);
- printf(" Frame type: %d\n", genFuncletInfo.fiFrameType);
- printf(" SP delta 1: %d\n", genFuncletInfo.fiSpDelta1);
-
- if (compiler->lvaPSPSym != BAD_VAR_NUM)
+ if (compiler->opts.IsOSR())
{
- if (CallerSP_to_PSP_slot_delta !=
- compiler->lvaGetCallerSPRelativeOffset(compiler->lvaPSPSym)) // for debugging
- {
- printf("lvaGetCallerSPRelativeOffset(lvaPSPSym): %d\n",
- compiler->lvaGetCallerSPRelativeOffset(compiler->lvaPSPSym));
- }
+ printf(" OSR Pad: %d\n", osrPad);
}
+ printf(" Function CallerSP-to-FP delta: %d\n", genFuncletInfo.fiFunction_CallerSP_to_FP_delta);
+ printf(" SP to CalleeSaved location delta: %d\n", genFuncletInfo.fiSP_to_CalleeSaved_delta);
+ printf(" SP delta: %d\n", genFuncletInfo.fiSpDelta);
}
+ assert(genFuncletInfo.fiSP_to_CalleeSaved_delta >= 0);
- assert(genFuncletInfo.fiSP_to_FPRA_save_delta >= 0);
+ if (compiler->lvaPSPSym != BAD_VAR_NUM)
+ {
+ assert(genFuncletInfo.fiCallerSP_to_PSP_slot_delta ==
+ compiler->lvaGetCallerSPRelativeOffset(compiler->lvaPSPSym)); // same offset used in main function and
+ // funclet!
+ }
#endif // DEBUG
}
@@ -1511,12 +1341,17 @@ void CodeGen::genSetPSPSym(regNumber initReg, bool* pInitRegZeroed)
int SPtoCallerSPdelta = -genCallerSPtoInitialSPdelta();
+ if (compiler->opts.IsOSR())
+ {
+ SPtoCallerSPdelta += compiler->info.compPatchpointInfo->TotalFrameSize();
+ }
+
// We will just use the initReg since it is an available register
// and we are probably done using it anyway...
regNumber regTmp = initReg;
*pInitRegZeroed = false;
- genInstrWithConstant(INS_addi_d, EA_PTRSIZE, regTmp, REG_SPBASE, SPtoCallerSPdelta, REG_R21, false);
+ genInstrWithConstant(INS_addi_d, EA_PTRSIZE, regTmp, REG_SPBASE, SPtoCallerSPdelta, regTmp, false);
GetEmitter()->emitIns_S_R(INS_st_d, EA_PTRSIZE, regTmp, compiler->lvaPSPSym, 0);
}
@@ -1806,7 +1641,7 @@ void CodeGen::genSetRegToConst(regNumber targetReg, var_types targetType, GenTre
emitAttr size = emitActualTypeSize(tree);
double constValue = tree->AsDblCon()->DconValue();
- // Make sure we use "daddiu reg, zero, 0x00" only for positive zero (0.0)
+ // Make sure we use "addi.d reg, zero, 0x00" only for positive zero (0.0)
// and not for negative zero (-0.0)
if (*(__int64*)&constValue == 0)
{
@@ -2867,8 +2702,8 @@ void CodeGen::genCodeForInitBlkUnroll(GenTreeBlk* node)
// GC pointers.
// For this case we'll generate a sequence of loads/stores in the case of struct
// slots that don't contain GC pointers. The generated code will look like:
-// ld tempReg, 8(A5)
-// sd tempReg, 8(A6)
+// ld.d tempReg,A5,8
+// st.d tempReg,A6,8
//
// In the case of a GC-Pointer we'll call the ByRef write barrier helper
// who happens to use the same registers as the previous call to maintain
@@ -2876,14 +2711,14 @@ void CodeGen::genCodeForInitBlkUnroll(GenTreeBlk* node)
// bl CORINFO_HELP_ASSIGN_BYREF
//
// So finally an example would look like this:
-// ld tempReg, 8(A5)
-// sd tempReg, 8(A6)
+// ld.d tempReg,A5,8
+// st.d tempReg,A6,8
// bl CORINFO_HELP_ASSIGN_BYREF
-// ld tempReg, 8(A5)
-// sd tempReg, 8(A6)
+// ld.d tempReg,A5,8
+// st.d tempReg,A6,8
// bl CORINFO_HELP_ASSIGN_BYREF
-// ld tempReg, 8(A5)
-// sd tempReg, 8(A6)
+// ld.d tempReg,A5,8
+// st.d tempReg,A6,8
void CodeGen::genCodeForCpObj(GenTreeBlk* cpObjNode)
{
GenTree* dstAddr = cpObjNode->Addr();
@@ -2955,7 +2790,7 @@ void CodeGen::genCodeForCpObj(GenTreeBlk* cpObjNode)
if (dstOnStack)
{
unsigned i = 0;
- // Check if two or more remaining slots and use two ld/sd sequence
+ // Check if two or more remaining slots and use two load/store sequence
while (i < slots - 1)
{
emitAttr attr0 = emitTypeSize(layout->GetGCPtrType(i + 0));
@@ -2977,7 +2812,7 @@ void CodeGen::genCodeForCpObj(GenTreeBlk* cpObjNode)
i += 2;
}
- // Use a ld/sd sequence for the last remainder
+ // Use a load/store sequence for the last remainder
if (i < slots)
{
emitAttr attr0 = emitTypeSize(layout->GetGCPtrType(i + 0));
@@ -3004,7 +2839,7 @@ void CodeGen::genCodeForCpObj(GenTreeBlk* cpObjNode)
{
if (!layout->IsGCPtr(i))
{
- // Check if the next slot's type is also TYP_GC_NONE and use two ld/sd
+ // Check if the next slot's type is also TYP_GC_NONE and use two load/store
if ((i + 1 < slots) && !layout->IsGCPtr(i + 1))
{
if ((i + 2) == slots)
@@ -4078,29 +3913,6 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
assert(targetReg != REG_NA);
assert(tree->TypeGet() != TYP_VOID);
- if (op1->isContainedIntOrIImmed())
- {
- op1 = tree->gtOp2;
- op2 = tree->gtOp1;
- switch (tree->OperGet())
- {
- case GT_LT:
- tree->SetOper(GT_GT);
- break;
- case GT_LE:
- tree->SetOper(GT_GE);
- break;
- case GT_GT:
- tree->SetOper(GT_LT);
- break;
- case GT_GE:
- tree->SetOper(GT_LE);
- break;
- default:
- break;
- }
- }
-
assert(!op1->isContainedIntOrIImmed());
assert(tree->OperIs(GT_LT, GT_LE, GT_EQ, GT_NE, GT_GT, GT_GE));
@@ -4120,6 +3932,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
if (IsUnsigned)
{
imm = static_cast(imm);
+
emit->emitIns_R_R_I_I(INS_bstrpick_d, EA_8BYTE, tmpRegOp1, regOp1, 31, 0);
}
else
@@ -4355,19 +4168,18 @@ void CodeGen::genCodeForJumpCompare(GenTreeOpCC* tree)
{
case EA_4BYTE:
{
- regNumber tmpRegOp1 = REG_R21;
- assert(regOp1 != tmpRegOp1);
+ assert(regOp1 != REG_R21);
if (cond.IsUnsigned())
{
imm = static_cast(imm);
- emit->emitIns_R_R_I_I(INS_bstrpick_d, EA_8BYTE, tmpRegOp1, regOp1, 31, 0);
+ emit->emitIns_R_R_I_I(INS_bstrpick_d, EA_8BYTE, REG_R21, regOp1, 31, 0);
}
else
{
imm = static_cast(imm);
- emit->emitIns_R_R_I(INS_slli_w, EA_4BYTE, tmpRegOp1, regOp1, 0);
+ emit->emitIns_R_R_I(INS_slli_w, EA_4BYTE, REG_R21, regOp1, 0);
}
- regOp1 = tmpRegOp1;
+ regOp1 = REG_R21;
break;
}
case EA_8BYTE:
@@ -4389,6 +4201,22 @@ void CodeGen::genCodeForJumpCompare(GenTreeOpCC* tree)
emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
regs = (int)REG_RA << 5;
}
+ else
+ {
+ if (cmpSize == EA_4BYTE)
+ {
+ assert(regOp1 != REG_R21);
+ if (cond.IsUnsigned())
+ {
+ emit->emitIns_R_R_I_I(INS_bstrpick_d, EA_8BYTE, REG_R21, regOp1, 31, 0);
+ }
+ else
+ {
+ emit->emitIns_R_R_I(INS_slli_w, EA_8BYTE, REG_R21, regOp1, 0);
+ }
+ regOp1 = REG_R21;
+ }
+ }
switch (cond.GetCode())
{
@@ -4498,8 +4326,9 @@ void CodeGen::genCodeForJumpCompare(GenTreeOpCC* tree)
int CodeGenInterface::genSPtoFPdelta() const
{
assert(isFramePointerUsed());
+ assert(compiler->compCalleeRegsPushed >= 2);
- int delta = compiler->lvaOutgoingArgSpaceSize;
+ int delta = compiler->lvaOutgoingArgSpaceSize + (compiler->compCalleeRegsPushed << 3) - 8;
assert(delta >= 0);
return delta;
@@ -4538,9 +4367,7 @@ int CodeGenInterface::genTotalFrameSize() const
int CodeGenInterface::genCallerSPtoFPdelta() const
{
assert(isFramePointerUsed());
- int callerSPtoFPdelta;
-
- callerSPtoFPdelta = genCallerSPtoInitialSPdelta() + genSPtoFPdelta();
+ int callerSPtoFPdelta = genCallerSPtoInitialSPdelta() + genSPtoFPdelta();
assert(callerSPtoFPdelta <= 0);
return callerSPtoFPdelta;
@@ -4553,9 +4380,7 @@ int CodeGenInterface::genCallerSPtoFPdelta() const
int CodeGenInterface::genCallerSPtoInitialSPdelta() const
{
- int callerSPtoSPdelta = 0;
-
- callerSPtoSPdelta -= genTotalFrameSize();
+ int callerSPtoSPdelta = -genTotalFrameSize();
assert(callerSPtoSPdelta <= 0);
return callerSPtoSPdelta;
@@ -5248,6 +5073,12 @@ void CodeGen::genSetGSSecurityCookie(regNumber initReg, bool* pInitRegZeroed)
return;
}
+ if (compiler->opts.IsOSR() && compiler->info.compPatchpointInfo->HasSecurityCookie())
+ {
+ // Security cookie is on original frame and was initialized there.
+ return;
+ }
+
if (compiler->gsGlobalSecurityCookieAddr == nullptr)
{
noway_assert(compiler->gsGlobalSecurityCookieVal != 0);
@@ -7851,7 +7682,7 @@ inline void CodeGen::genJumpToThrowHlpBlk_la(
callType = emitter::EC_INDIR_R;
callTarget = REG_DEFAULT_HELPER_CALL_TARGET;
- // ssize_t imm = (4 + 1 + 1) << 2;// 4=li, 1=ld, 1=jirl.
+ // ssize_t imm = (4 + 1 + 1) << 2;// 4=li, 1=load, 1=jirl.
// instGen_Set_Reg_To_Imm(EA_PTR_DSP_RELOC, callTarget, (ssize_t)pAddr);
// emit->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, callTarget, callTarget, 0);
@@ -7959,7 +7790,66 @@ void CodeGen::genProfilingLeaveCallback(unsigned helper /*= CORINFO_HELP_PROF_FC
/*-----------------------------------------------------------------------------
*
- * Push/Pop any callee-saved registers we have used
+ * Push/Pop any callee-saved registers we have used
+ * For most frames, generatint liking:
+ * addi.d sp, sp, -totalFrameSize
+ * fst.d f24,sp,off
+ * ...
+ * fst.d f31,sp,off+7*8
+ *
+ * st.d s0,sp,off2
+ * st.d s1,sp,off2+8
+ * ...
+ * st.d s8,sp,off2+8*8
+ *
+ * st.d ra,sp,off3
+ * st.d fp,sp,off3+8
+ *
+ * Notes:
+ * 1. FP is always saved, and the first store is FP, RA.
+ * 2. General-purpose registers are 8 bytes, floating-point registers are 8 bytes.
+ * 3. For frames with varargs, not implemented completely and not tested !
+ * 4. We allocate the frame here; no further changes to SP are allowed (except in the body, for localloc).
+ *
+ * For functions with GS and localloc, we change the frame so the frame pointer and RA are saved at the top
+ * of the frame, just under the varargs registers (if any). Note that the funclet frames must follow the same
+ * rule, and both main frame and funclet frames (if any) must put PSPSym in the same offset from Caller-SP.
+ * Since this frame type is relatively rare, we force using it via stress modes, for additional coverage.
+ *
+ * The frames look like the following (simplified to only include components that matter for establishing the
+ * frames). See also Compiler::lvaAssignFrameOffsets().
+ *
+ *
+ * The LoongArch64's frame layout is liking:
+ *
+ * | |
+ * |-----------------------|
+ * | incoming arguments |
+ * +=======================+ <---- Caller's SP
+ * | Arguments Or | // if needed.
+ * | Varargs regs space | // Only for varargs functions; (varargs not implemented for LoongArch64)
+ * |-----------------------|
+ * | MonitorAcquired | // 8 bytes; for synchronized methods
+ * |-----------------------|
+ * | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
+ * |-----------------------|
+ * | locals, temps, etc. |
+ * |-----------------------|
+ * | possible GS cookie |
+ * |-----------------------|
+ * | Saved FP | // 8 bytes
+ * |-----------------------|
+ * | Saved RA | // 8 bytes
+ * |-----------------------|
+ * |Callee saved registers | // not including FP/RA; multiple of 8 bytes
+ * |-----------------------|
+ * | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
+ * |-----------------------| <---- Ambient SP
+ * | | |
+ * ~ | Stack grows ~
+ * | | downward |
+ * V
+ *
*/
void CodeGen::genPushCalleeSavedRegisters(regNumber initReg, bool* pInitRegZeroed)
{
@@ -7974,11 +7864,8 @@ void CodeGen::genPushCalleeSavedRegisters(regNumber initReg, bool* pInitRegZeroe
}
#endif
- // On LA we push the FP (frame-pointer) here along with all other callee saved registers
- if (isFramePointerUsed())
- {
- rsPushRegs |= RBM_FPBASE;
- }
+ // We always use the FP (frame-pointer).
+ assert(isFramePointerUsed());
//
// It may be possible to skip pushing/popping ra for leaf methods. However, such optimization would require
@@ -8000,122 +7887,22 @@ void CodeGen::genPushCalleeSavedRegisters(regNumber initReg, bool* pInitRegZeroe
// is not worth it.
//
- rsPushRegs |= RBM_RA; // We must save the return address (in the RA register).
- regSet.rsMaskCalleeSaved = rsPushRegs;
- regMaskTP maskSaveRegsFloat = rsPushRegs & RBM_ALLFLOAT;
- regMaskTP maskSaveRegsInt = rsPushRegs & ~maskSaveRegsFloat;
+ regSet.rsMaskCalleeSaved = rsPushRegs | RBM_FPBASE | RBM_RA;
#ifdef DEBUG
- if (compiler->compCalleeRegsPushed != genCountBits(rsPushRegs))
+ if (compiler->compCalleeRegsPushed != genCountBits(regSet.rsMaskCalleeSaved))
{
printf("Error: unexpected number of callee-saved registers to push. Expected: %d. Got: %d ",
- compiler->compCalleeRegsPushed, genCountBits(rsPushRegs));
- dspRegMask(rsPushRegs);
+ compiler->compCalleeRegsPushed, genCountBits(rsPushRegs | RBM_FPBASE | RBM_RA));
+ dspRegMask(rsPushRegs | RBM_FPBASE | RBM_RA);
printf("\n");
- assert(compiler->compCalleeRegsPushed == genCountBits(rsPushRegs));
+ assert(compiler->compCalleeRegsPushed == genCountBits(rsPushRegs | RBM_FPBASE | RBM_RA));
}
-#endif // DEBUG
-
- // See the document "LOONGARCH64 JIT Frame Layout" and/or "LOONGARCH64 Exception Data" for more details or
- // requirements and
- // options. Case numbers in comments here refer to this document. See also Compiler::lvaAssignFrameOffsets()
- // for pictures of the general frame layouts, and CodeGen::genFuncletProlog() implementations (per architecture)
- // for pictures of the funclet frame layouts.
- //
- // For most frames, generate, e.g.:
- // sdc1 f31, off+7*8(sp)
- // ...
- // sdc1 f24, off(sp)
- //
- // sd s7, off2+7*8(sp)
- // ...
- // sd s1, off2+8(sp)
- // sd s0, off2(sp)
- //
- // sd fp, 0(sp)
- // sd ra, 8(sp)
- //
- // Notes:
- // 1. FP is always saved, and the first store is FP, RA.
- // 2. General-purpose registers are 8 bytes, floating-point registers are 8 bytes, but SIMD/FP registers 16 bytes.
- // TODO-LOONGARCH64: supporting SIMD feature !
- // 3. For frames with varargs, not implemented completely and not tested !
- // 4. We allocate the frame here; no further changes to SP are allowed (except in the body, for localloc).
- //
- // For functions with GS and localloc, we change the frame so the frame pointer and RA are saved at the top
- // of the frame, just under the varargs registers (if any). Note that the funclet frames must follow the same
- // rule, and both main frame and funclet frames (if any) must put PSPSym in the same offset from Caller-SP.
- // Since this frame type is relatively rare, we force using it via stress modes, for additional coverage.
- //
- // The frames look like the following (simplified to only include components that matter for establishing the
- // frames). See also Compiler::lvaAssignFrameOffsets().
- //
- //
- // Frames with FP, RA saved at bottom of frame (above outgoing argument space):
- //
- // | |
- // |-----------------------|
- // | incoming arguments |
- // +=======================+ <---- Caller's SP
- // | Arguments Or | // if needed.
- // | Varargs regs space | // Only for varargs functions; (varargs not implemented for LoongArch64)
- // |-----------------------|
- // |Callee saved registers | // not including FP/RA; multiple of 8 bytes
- // |-----------------------|
- // | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
- // |-----------------------|
- // | locals, temps, etc. |
- // |-----------------------|
- // | possible GS cookie |
- // |-----------------------|
- // | Saved RA | // 8 bytes
- // |-----------------------|
- // | Saved FP | // 8 bytes
- // |-----------------------|
- // | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
- // |-----------------------| <---- Ambient SP
- // | | |
- // ~ | Stack grows ~
- // | | downward |
- // V
- //
- //
- // Frames with FP, RA saved at top of frame (note: above all callee-saved regs):
- //
- // | |
- // |-----------------------|
- // | incoming arguments |
- // +=======================+ <---- Caller's SP
- // | Arguments Or | // if needed.
- // | Varargs regs space | // Only for varargs functions; (varargs not implemented for LoongArch64)
- // |-----------------------|
- // | Saved RA | // 8 bytes
- // |-----------------------|
- // | Saved FP | // 8 bytes
- // |-----------------------|
- // |Callee saved registers | // not including FP/RA; multiple of 8 bytes
- // |-----------------------|
- // | PSP slot | // 8 bytes (omitted in NativeAOT ABI)
- // |-----------------------|
- // | locals, temps, etc. |
- // |-----------------------|
- // | possible GS cookie |
- // |-----------------------|
- // | Outgoing arg space | // multiple of 8 bytes; if required (i.e., #outsz != 0)
- // |-----------------------| <---- Ambient SP
- // | | |
- // ~ | Stack grows ~
- // | | downward |
- // V
- //
-
- int totalFrameSize = genTotalFrameSize();
- int offset; // This will be the starting place for saving the callee-saved registers, in increasing order.
-
-#ifdef DEBUG
if (verbose)
{
+ regMaskTP maskSaveRegsFloat = rsPushRegs & RBM_ALLFLOAT;
+ regMaskTP maskSaveRegsInt = rsPushRegs & ~maskSaveRegsFloat;
printf("Save float regs: ");
dspRegMask(maskSaveRegsFloat);
printf("\n");
@@ -8129,100 +7916,80 @@ void CodeGen::genPushCalleeSavedRegisters(regNumber initReg, bool* pInitRegZeroe
// generate based on various sizes.
int frameType = 0;
- // The amount to subtract from SP before starting to store the callee-saved registers. It might be folded into the
- // first save instruction as a "predecrement" amount, if possible.
+ // The amount to add from SP before starting to store the callee-saved registers.
int calleeSaveSPDelta = 0;
- // By default, we'll establish the frame pointer chain. (Note that currently frames without FP are NYI.)
- bool establishFramePointer = true;
+ // If we need to generate a GS cookie, we need to make sure the saved frame pointer and return address
+ // (FP and RA) are protected from buffer overrun by the GS cookie. If FP/RA are at the lowest addresses,
+ // then they are safe, since they are lower than any unsafe buffers. And the GS cookie we add will
+ // protect our caller's frame. If we have a localloc, however, that is dynamically placed lower than our
+ // saved FP/RA. In that case, we save FP/RA along with the rest of the callee-saved registers, above
+ // the GS cookie.
+ //
+ // After the frame is allocated, the frame pointer is established, pointing at the saved frame pointer to
+ // create a frame pointer chain.
+ //
- // If we do establish the frame pointer, what is the amount we add to SP to do so?
- unsigned offsetSpToSavedFp = 0;
+ // This will be the starting place for saving the callee-saved registers, in increasing order.
+ int offset = compiler->lvaOutgoingArgSpaceSize;
- if (isFramePointerUsed())
+ int totalFrameSize = genTotalFrameSize();
+ // The (totalFrameSize <= 2040) condition ensures the offsets of st.d/ld.d.
+ if (totalFrameSize <= 2040)
{
- // We need to save both FP and RA.
-
- assert((maskSaveRegsInt & RBM_FP) != 0);
- assert((maskSaveRegsInt & RBM_RA) != 0);
+ GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, -totalFrameSize);
+ compiler->unwindAllocStack(totalFrameSize);
- // If we need to generate a GS cookie, we need to make sure the saved frame pointer and return address
- // (FP and RA) are protected from buffer overrun by the GS cookie. If FP/RA are at the lowest addresses,
- // then they are safe, since they are lower than any unsafe buffers. And the GS cookie we add will
- // protect our caller's frame. If we have a localloc, however, that is dynamically placed lower than our
- // saved FP/RA. In that case, we save FP/RA along with the rest of the callee-saved registers, above
- // the GS cookie.
+ // Case #1.
//
- // After the frame is allocated, the frame pointer is established, pointing at the saved frame pointer to
- // create a frame pointer chain.
+ // Generate:
+ // addi.d sp, sp, -framesz
+ // st.d callee_saved_registers ### not including the fp and ra.
+ // st.d ra,sp,outsz
+ // st.d fp,sp,outsz+8
//
+ // After saving callee-saved registers, ra and fp, we establish the frame pointer with:
+ // addi.d fp, sp, (the offset of saving fp)
+ // We do this *after* saving callee-saved registers, so the prolog/epilog unwind codes mostly match.
- if (totalFrameSize < 2048)
- {
- GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, -totalFrameSize);
- compiler->unwindAllocStack(totalFrameSize);
+ JITDUMP("Frame type 1. #outsz=%d; #framesz=%d; LclFrameSize=%d\n", unsigned(compiler->lvaOutgoingArgSpaceSize),
+ totalFrameSize, compiler->compLclFrameSize);
- // Case #1.
- //
- // Generate:
- // daddiu sp, sp, -framesz
- // sd fp, outsz(sp)
- // sd ra, outsz+8(sp)
- //
- // The (totalFrameSize <= 2047) condition ensures the offsets of sd/ld.
- //
- // After saving callee-saved registers, we establish the frame pointer with:
- // daddiu fp, sp, offset-fp
- // We do this *after* saving callee-saved registers, so the prolog/epilog unwind codes mostly match.
-
- JITDUMP("Frame type 1. #outsz=%d; #framesz=%d; LclFrameSize=%d\n",
- unsigned(compiler->lvaOutgoingArgSpaceSize), totalFrameSize, compiler->compLclFrameSize);
-
- frameType = 1;
-
- offsetSpToSavedFp = compiler->lvaOutgoingArgSpaceSize;
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offsetSpToSavedFp);
- compiler->unwindSaveReg(REG_FP, offsetSpToSavedFp);
+ frameType = 1;
+ }
+ else
+ {
+ JITDUMP("Frame type 2. #outsz=%d; #framesz=%d; LclFrameSize=%d\n", unsigned(compiler->lvaOutgoingArgSpaceSize),
+ totalFrameSize, compiler->compLclFrameSize);
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offsetSpToSavedFp + 8);
- compiler->unwindSaveReg(REG_RA, offsetSpToSavedFp + 8);
+ frameType = 2;
- maskSaveRegsInt &= ~(RBM_FP | RBM_RA); // We've already saved FP/RA
+ if ((offset + (compiler->compCalleeRegsPushed << 3)) >= 2040)
+ {
+ offset = totalFrameSize - compiler->lvaOutgoingArgSpaceSize;
+ calleeSaveSPDelta = AlignUp((UINT)offset, STACK_ALIGN);
+ offset = calleeSaveSPDelta - offset;
- offset = compiler->compLclFrameSize + 2 * REGSIZE_BYTES; // FP/RA
+ genStackPointerAdjustment(-calleeSaveSPDelta, initReg, pInitRegZeroed, /* reportUnwindData */ true);
}
else
{
- JITDUMP("Frame type 2. #outsz=%d; #framesz=%d; LclFrameSize=%d\n",
- unsigned(compiler->lvaOutgoingArgSpaceSize), totalFrameSize, compiler->compLclFrameSize);
-
- frameType = 2;
-
- maskSaveRegsInt &= ~(RBM_FP | RBM_RA); // We've already saved FP/RA
-
- offset = totalFrameSize - compiler->compLclFrameSize - 2 * REGSIZE_BYTES;
- calleeSaveSPDelta = AlignUp((UINT)offset, STACK_ALIGN);
- offset = calleeSaveSPDelta - offset;
+ genStackPointerAdjustment(-totalFrameSize, initReg, pInitRegZeroed, /* reportUnwindData */ true);
}
}
- else
- {
- // No frame pointer (no chaining).
- assert((maskSaveRegsInt & RBM_FP) == 0);
- assert((maskSaveRegsInt & RBM_RA) != 0);
- // Note that there is no pre-indexed save_lrpair unwind code variant, so we can't allocate the frame using
- // 'sd' if we only have one callee-saved register plus RA to save.
+ JITDUMP(" offset=%d, calleeSaveSPDelta=%d\n", offset, calleeSaveSPDelta);
+ genSaveCalleeSavedRegistersHelp(rsPushRegs, offset, 0);
+ offset += (int)(genCountBits(rsPushRegs) << 3);
- NYI("Frame without frame pointer");
- offset = 0;
- }
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset);
+ compiler->unwindSaveReg(REG_RA, offset);
- assert(frameType != 0);
+ GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset + 8);
+ compiler->unwindSaveReg(REG_FP, offset + 8);
- JITDUMP(" offset=%d, calleeSaveSPDelta=%d\n", offset, calleeSaveSPDelta);
- genSaveCalleeSavedRegistersHelp(maskSaveRegsInt | maskSaveRegsFloat, offset, -calleeSaveSPDelta);
+ JITDUMP(" offsetSpToSavedFp=%d\n", offset + 8);
+ genEstablishFramePointer(offset + 8, /* reportUnwindData */ true);
// For varargs, home the incoming arg registers last. Note that there is nothing to unwind here,
// so we just report "NOP" unwind codes. If there's no more frame setup after this, we don't
@@ -8236,63 +8003,16 @@ void CodeGen::genPushCalleeSavedRegisters(regNumber initReg, bool* pInitRegZeroe
#ifdef DEBUG
if (compiler->opts.disAsm)
{
+ assert(frameType != 0);
printf("DEBUG: LOONGARCH64, frameType:%d\n\n", frameType);
}
#endif
- if (frameType == 1)
- {
- // offsetSpToSavedFp = genSPtoFPdelta();
- }
- else if (frameType == 2)
- {
- if (compiler->lvaOutgoingArgSpaceSize >= 2040)
- {
- offset = totalFrameSize - calleeSaveSPDelta - compiler->lvaOutgoingArgSpaceSize;
- calleeSaveSPDelta = AlignUp((UINT)offset, STACK_ALIGN);
- offset = calleeSaveSPDelta - offset;
- genStackPointerAdjustment(-calleeSaveSPDelta, initReg, pInitRegZeroed, /* reportUnwindData */ true);
-
- offsetSpToSavedFp = offset;
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset);
- compiler->unwindSaveReg(REG_FP, offset);
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset + 8);
- compiler->unwindSaveReg(REG_RA, offset + 8);
-
- genEstablishFramePointer(offset, /* reportUnwindData */ true);
-
- calleeSaveSPDelta = compiler->lvaOutgoingArgSpaceSize & ~0xf;
- genStackPointerAdjustment(-calleeSaveSPDelta, initReg, pInitRegZeroed, /* reportUnwindData */ true);
- }
- else
- {
- calleeSaveSPDelta = totalFrameSize - calleeSaveSPDelta;
- genStackPointerAdjustment(-calleeSaveSPDelta, initReg, pInitRegZeroed, /* reportUnwindData */ true);
-
- offset = compiler->lvaOutgoingArgSpaceSize;
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset);
- compiler->unwindSaveReg(REG_FP, offset);
-
- GetEmitter()->emitIns_R_R_I(INS_st_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset + 8);
- compiler->unwindSaveReg(REG_RA, offset + 8);
-
- genEstablishFramePointer(offset, /* reportUnwindData */ true);
- }
-
- establishFramePointer = false;
- }
- else
- {
- unreached();
- }
-
- if (establishFramePointer)
+ if (calleeSaveSPDelta != 0)
{
- JITDUMP(" offsetSpToSavedFp=%d\n", offsetSpToSavedFp);
- genEstablishFramePointer(offsetSpToSavedFp, /* reportUnwindData */ true);
+ assert(frameType == 2);
+ calleeSaveSPDelta = totalFrameSize - calleeSaveSPDelta;
+ genStackPointerAdjustment(-calleeSaveSPDelta, initReg, pInitRegZeroed, /* reportUnwindData */ true);
}
}
@@ -8300,147 +8020,108 @@ void CodeGen::genPopCalleeSavedRegisters(bool jmpEpilog)
{
assert(compiler->compGeneratingEpilog);
- regMaskTP rsRestoreRegs = regSet.rsGetModifiedRegsMask() & RBM_CALLEE_SAVED;
+ regMaskTP regsToRestoreMask = regSet.rsGetModifiedRegsMask() & RBM_CALLEE_SAVED;
- if (isFramePointerUsed())
- {
- rsRestoreRegs |= RBM_FPBASE;
- }
-
- rsRestoreRegs |= RBM_RA; // We must save/restore the return address.
+ assert(isFramePointerUsed());
- regMaskTP regsToRestoreMask = rsRestoreRegs;
+ // This will be the starting place for restoring the callee-saved registers, in decreasing order.
+ int calleeSaveSPOffset = 0;
+ int remainingSPSize = 0;
int totalFrameSize = genTotalFrameSize();
+ if (totalFrameSize <= 2040)
+ {
+ JITDUMP("Frame type 1. #outsz=%d; #framesz=%d; localloc? %s\n", unsigned(compiler->lvaOutgoingArgSpaceSize),
+ totalFrameSize, dspBool(compiler->compLocallocUsed));
- int calleeSaveSPOffset = 0; // This will be the starting place for restoring
- // the callee-saved registers, in decreasing order.
- int frameType = 0; // An indicator of what type of frame we are popping.
- int calleeSaveSPDelta = 0; // Amount to add to SP after callee-saved registers have been restored.
+ if (compiler->compLocallocUsed)
+ {
+ int SPtoFPdelta = (compiler->compCalleeRegsPushed << 3) - 8 + compiler->lvaOutgoingArgSpaceSize;
- if (isFramePointerUsed())
+ // Restore sp from fp
+ GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_FPBASE, -SPtoFPdelta);
+ compiler->unwindSetFrameReg(REG_FPBASE, SPtoFPdelta);
+ }
+ calleeSaveSPOffset = compiler->lvaOutgoingArgSpaceSize;
+ remainingSPSize = totalFrameSize;
+ }
+ else
{
- if (totalFrameSize <= 2047)
+ JITDUMP("Frame type 2. #outsz=%d; #framesz=%d; #calleeSaveRegsPushed:%d; "
+ "localloc? %s\n",
+ unsigned(compiler->lvaOutgoingArgSpaceSize), totalFrameSize, compiler->compCalleeRegsPushed,
+ dspBool(compiler->compLocallocUsed));
+
+ if ((compiler->lvaOutgoingArgSpaceSize + (compiler->compCalleeRegsPushed << 3)) > 2047)
{
+ calleeSaveSPOffset = compiler->lvaOutgoingArgSpaceSize & -16;
if (compiler->compLocallocUsed)
{
- int SPtoFPdelta = genSPtoFPdelta();
+ int SPtoFPdelta = (compiler->compCalleeRegsPushed << 3) - 8;
+
// Restore sp from fp
GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_FPBASE, -SPtoFPdelta);
compiler->unwindSetFrameReg(REG_FPBASE, SPtoFPdelta);
}
-
- JITDUMP("Frame type 1(save FP/RA at bottom). #outsz=%d; #framesz=%d; localloc? %s\n",
- unsigned(compiler->lvaOutgoingArgSpaceSize), totalFrameSize, dspBool(compiler->compLocallocUsed));
-
- frameType = 1;
-
- regsToRestoreMask &= ~(RBM_FP | RBM_RA); // We'll restore FP/RA at the end.
-
- calleeSaveSPOffset = compiler->compLclFrameSize + 2 * REGSIZE_BYTES;
+ else
+ {
+ genStackPointerAdjustment(calleeSaveSPOffset, REG_RA, nullptr, /* reportUnwindData */ true);
+ }
+ remainingSPSize = totalFrameSize - calleeSaveSPOffset;
+ calleeSaveSPOffset = compiler->lvaOutgoingArgSpaceSize - calleeSaveSPOffset;
}
else
{
- JITDUMP("Frame type 2(save FP/RA at bottom). #outsz=%d; #framesz=%d; #calleeSaveRegsPushed:%d; "
- "localloc? %s\n",
- unsigned(compiler->lvaOutgoingArgSpaceSize), totalFrameSize, compiler->compCalleeRegsPushed,
- dspBool(compiler->compLocallocUsed));
-
- frameType = 2;
-
- int outSzAligned;
- if (compiler->lvaOutgoingArgSpaceSize >= 2040)
+ if (compiler->compLocallocUsed)
{
- int offset = totalFrameSize - compiler->compLclFrameSize - 2 * REGSIZE_BYTES;
- calleeSaveSPDelta = AlignUp((UINT)offset, STACK_ALIGN);
- calleeSaveSPOffset = calleeSaveSPDelta - offset;
-
- int offset2 = totalFrameSize - calleeSaveSPDelta - compiler->lvaOutgoingArgSpaceSize;
- calleeSaveSPDelta = AlignUp((UINT)offset2, STACK_ALIGN);
- offset2 = calleeSaveSPDelta - offset2;
-
- if (compiler->compLocallocUsed)
- {
- // Restore sp from fp
- GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_FPBASE, -offset2);
- compiler->unwindSetFrameReg(REG_FPBASE, offset2);
- }
- else
- {
- outSzAligned = compiler->lvaOutgoingArgSpaceSize & ~0xf;
- genStackPointerAdjustment(outSzAligned, REG_R21, nullptr, /* reportUnwindData */ true);
- }
-
- regsToRestoreMask &= ~(RBM_FP | RBM_RA); // We'll restore FP/RA at the end.
-
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset2 + 8);
- compiler->unwindSaveReg(REG_RA, offset2 + 8);
+ int SPtoFPdelta = (compiler->compCalleeRegsPushed << 3) - 8 + compiler->lvaOutgoingArgSpaceSize;
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset2);
- compiler->unwindSaveReg(REG_FP, offset2);
-
- genStackPointerAdjustment(calleeSaveSPDelta, REG_R21, nullptr, /* reportUnwindData */ true);
-
- calleeSaveSPDelta = totalFrameSize - compiler->compLclFrameSize - 2 * REGSIZE_BYTES;
- calleeSaveSPDelta = AlignUp((UINT)calleeSaveSPDelta, STACK_ALIGN);
+ // Restore sp from fp
+ GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_FPBASE, -SPtoFPdelta);
+ compiler->unwindSetFrameReg(REG_FPBASE, SPtoFPdelta);
}
- else
- {
- int offset2 = compiler->lvaOutgoingArgSpaceSize;
- if (compiler->compLocallocUsed)
- {
- // Restore sp from fp
- GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_FPBASE, -offset2);
- compiler->unwindSetFrameReg(REG_FPBASE, offset2);
- }
-
- regsToRestoreMask &= ~(RBM_FP | RBM_RA); // We'll restore FP/RA at the end.
+ calleeSaveSPOffset = compiler->lvaOutgoingArgSpaceSize;
+ remainingSPSize = totalFrameSize;
+ }
+ }
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, offset2 + 8);
- compiler->unwindSaveReg(REG_RA, offset2 + 8);
+ JITDUMP(" calleeSaveSPOffset=%d\n", calleeSaveSPOffset);
+ genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, calleeSaveSPOffset, 0);
+ calleeSaveSPOffset += (compiler->compCalleeRegsPushed - 2) << 3;
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, offset2);
- compiler->unwindSaveReg(REG_FP, offset2);
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, calleeSaveSPOffset);
+ compiler->unwindSaveReg(REG_RA, calleeSaveSPOffset);
- calleeSaveSPOffset = totalFrameSize - compiler->compLclFrameSize - 2 * REGSIZE_BYTES;
- calleeSaveSPDelta = AlignUp((UINT)calleeSaveSPOffset, STACK_ALIGN);
- calleeSaveSPOffset = calleeSaveSPDelta - calleeSaveSPOffset;
+ GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, calleeSaveSPOffset + 8);
+ compiler->unwindSaveReg(REG_FP, calleeSaveSPOffset + 8);
- genStackPointerAdjustment(totalFrameSize - calleeSaveSPDelta, REG_R21, nullptr,
- /* reportUnwindData */ true);
- }
- }
+ if (emitter::isValidUimm11(remainingSPSize))
+ {
+ GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, remainingSPSize);
}
else
{
- // No frame pointer (no chaining).
- NYI("Frame without frame pointer");
- calleeSaveSPOffset = 0;
+ GetEmitter()->emitIns_I_la(EA_PTRSIZE, REG_R21, remainingSPSize);
+ GetEmitter()->emitIns_R_R_R(INS_add_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, REG_R21);
}
+ compiler->unwindAllocStack(remainingSPSize);
- JITDUMP(" calleeSaveSPOffset=%d, calleeSaveSPDelta=%d\n", calleeSaveSPOffset, calleeSaveSPDelta);
- genRestoreCalleeSavedRegistersHelp(regsToRestoreMask, calleeSaveSPOffset, calleeSaveSPDelta);
-
- if (frameType == 1)
+ // For OSR, we must also adjust the SP to remove the Tier0 frame.
+ if (compiler->opts.IsOSR())
{
- calleeSaveSPOffset = compiler->lvaOutgoingArgSpaceSize;
-
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_RA, REG_SPBASE, calleeSaveSPOffset + 8);
- compiler->unwindSaveReg(REG_RA, calleeSaveSPOffset + 8);
+ const int tier0FrameSize = compiler->info.compPatchpointInfo->TotalFrameSize();
+ JITDUMP("Extra SP adjust for OSR to pop off Tier0 frame: %d bytes\n", tier0FrameSize);
- GetEmitter()->emitIns_R_R_I(INS_ld_d, EA_PTRSIZE, REG_FP, REG_SPBASE, calleeSaveSPOffset);
- compiler->unwindSaveReg(REG_FP, calleeSaveSPOffset);
-
- GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, totalFrameSize);
- compiler->unwindAllocStack(totalFrameSize);
- }
- else if (frameType == 2)
- {
- // had done.
- }
- else
- {
- unreached();
+ if (emitter::isValidUimm11(tier0FrameSize))
+ {
+ GetEmitter()->emitIns_R_R_I(INS_addi_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, tier0FrameSize);
+ }
+ else
+ {
+ GetEmitter()->emitIns_I_la(EA_PTRSIZE, REG_R21, tier0FrameSize);
+ GetEmitter()->emitIns_R_R_R(INS_add_d, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, REG_R21);
+ }
+ compiler->unwindAllocStack(tier0FrameSize);
}
}
diff --git a/src/coreclr/jit/codegenriscv64.cpp b/src/coreclr/jit/codegenriscv64.cpp
index 79ac3549c32841..e85a7336ccd014 100644
--- a/src/coreclr/jit/codegenriscv64.cpp
+++ b/src/coreclr/jit/codegenriscv64.cpp
@@ -76,7 +76,7 @@ bool CodeGen::genInstrWithConstant(instruction ins,
// first we load the immediate into tmpReg
assert(!EA_IS_RELOC(size));
- GetEmitter()->emitIns_I_la(size, tmpReg, imm);
+ GetEmitter()->emitLoadImmediate(size, tmpReg, imm);
regSet.verifyRegUsed(tmpReg);
// when we are in an unwind code region
@@ -1230,7 +1230,7 @@ void CodeGen::instGen_Set_Reg_To_Imm(emitAttr size,
}
else
{
- emit->emitIns_I_la(size, reg, imm);
+ emit->emitLoadImmediate(size, reg, imm);
}
regSet.verifyRegUsed(reg);
@@ -1308,7 +1308,25 @@ void CodeGen::genSetRegToConst(regNumber targetReg, var_types targetType, GenTre
// Produce code for a GT_INC_SATURATE node.
void CodeGen::genCodeForIncSaturate(GenTree* tree)
{
- NYI_RISCV64("genCodeForIncSaturate-----unimplemented/unused on RISCV64 yet----");
+ regNumber targetReg = tree->GetRegNum();
+
+ // The arithmetic node must be sitting in a register (since it's not contained)
+ assert(!tree->isContained());
+ // The dst can only be a register.
+ assert(targetReg != REG_NA);
+
+ GenTree* operand = tree->gtGetOp1();
+ assert(!operand->isContained());
+ // The src must be a register.
+ regNumber operandReg = genConsumeReg(operand);
+ emitAttr attr = emitActualTypeSize(tree);
+
+ GetEmitter()->emitIns_R_R_I(INS_addi, attr, targetReg, operandReg, 1);
+ // bne targetReg, zero, 2 * 4
+ GetEmitter()->emitIns_R_R_I(INS_bne, attr, targetReg, REG_R0, 8);
+ GetEmitter()->emitIns_R_R_I(INS_xori, attr, targetReg, targetReg, -1);
+
+ genProduceReg(tree);
}
// Generate code to get the high N bits of a N*N=2N bit multiplication result
@@ -1556,7 +1574,7 @@ void CodeGen::genCodeForStoreLclVar(GenTreeLclVar* lclNode)
else if (data->IsIntegralConst())
{
ssize_t imm = data->AsIntConCommon()->IconValue();
- emit->emitIns_I_la(EA_PTRSIZE, rsGetRsvdReg(), imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, rsGetRsvdReg(), imm);
dataReg = rsGetRsvdReg();
}
else
@@ -1792,7 +1810,7 @@ void CodeGen::genLclHeap(GenTree* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, rsGetRsvdReg(), amount);
+ emit->emitLoadImmediate(EA_PTRSIZE, rsGetRsvdReg(), amount);
emit->emitIns_R_R_R(INS_sub, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, rsGetRsvdReg());
}
@@ -2077,7 +2095,7 @@ void CodeGen::genCodeForDivMod(GenTreeOp* tree)
{
ssize_t intConst = (int)(divisorOp->AsIntCon()->gtIconVal);
divisorReg = rsGetRsvdReg();
- emit->emitIns_I_la(EA_PTRSIZE, divisorReg, intConst);
+ emit->emitLoadImmediate(EA_PTRSIZE, divisorReg, intConst);
}
// Only for commutative operations do we check src1 and allow it to be a contained immediate
else if (tree->OperIsCommutative())
@@ -2091,7 +2109,7 @@ void CodeGen::genCodeForDivMod(GenTreeOp* tree)
assert(!divisorOp->isContainedIntOrIImmed());
ssize_t intConst = (int)(src1->AsIntCon()->gtIconVal);
Reg1 = rsGetRsvdReg();
- emit->emitIns_I_la(EA_PTRSIZE, Reg1, intConst);
+ emit->emitLoadImmediate(EA_PTRSIZE, Reg1, intConst);
}
}
else
@@ -2965,7 +2983,7 @@ void CodeGen::genCodeForReturnTrap(GenTreeOp* tree)
else
{
// TODO-RISCV64: maybe optimize further.
- GetEmitter()->emitIns_I_la(EA_PTRSIZE, callTarget, (ssize_t)pAddr);
+ GetEmitter()->emitLoadImmediate(EA_PTRSIZE, callTarget, (ssize_t)pAddr);
GetEmitter()->emitIns_R_R_I(INS_ld, EA_PTRSIZE, callTarget, callTarget, 0);
}
regSet.verifyRegUsed(callTarget);
@@ -3507,7 +3525,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
emit->emitIns_R_R_R(IsUnsigned ? INS_sltu : INS_slt, EA_PTRSIZE, targetReg, regOp1, REG_RA);
}
}
@@ -3523,7 +3541,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm + 1);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm + 1);
emit->emitIns_R_R_R(IsUnsigned ? INS_sltu : INS_slt, EA_PTRSIZE, targetReg, regOp1, REG_RA);
}
}
@@ -3541,7 +3559,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
emit->emitIns_R_R_R(IsUnsigned ? INS_sltu : INS_slt, EA_PTRSIZE, targetReg, REG_RA, regOp1);
}
}
@@ -3557,7 +3575,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
emit->emitIns_R_R_R(IsUnsigned ? INS_sltu : INS_slt, EA_PTRSIZE, targetReg, regOp1, REG_RA);
}
emit->emitIns_R_R_I(INS_xori, EA_PTRSIZE, targetReg, targetReg, 1);
@@ -3575,7 +3593,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
emit->emitIns_R_R_R(INS_xor, EA_PTRSIZE, targetReg, regOp1, REG_RA);
emit->emitIns_R_R_R(INS_sltu, EA_PTRSIZE, targetReg, REG_R0, targetReg);
}
@@ -3593,7 +3611,7 @@ void CodeGen::genCodeForCompare(GenTreeOp* tree)
}
else
{
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
emit->emitIns_R_R_R(INS_xor, EA_PTRSIZE, targetReg, regOp1, REG_RA);
emit->emitIns_R_R_I(INS_sltiu, EA_PTRSIZE, targetReg, targetReg, 1);
}
@@ -3711,28 +3729,32 @@ void CodeGen::genCodeForJumpCompare(GenTreeOpCC* tree)
switch (cmpSize)
{
case EA_4BYTE:
+ {
+ regNumber tmpRegOp1 = rsGetRsvdReg();
+ assert(regOp1 != tmpRegOp1);
if (cond.IsUnsigned())
{
imm = static_cast(imm);
- regNumber tmpRegOp1 = rsGetRsvdReg();
assert(regOp1 != tmpRegOp1);
emit->emitIns_R_R_I(INS_slli, EA_8BYTE, tmpRegOp1, regOp1, 32);
emit->emitIns_R_R_I(INS_srli, EA_8BYTE, tmpRegOp1, tmpRegOp1, 32);
- regOp1 = tmpRegOp1;
}
else
{
imm = static_cast(imm);
+ emit->emitIns_R_R_I(INS_addiw, EA_8BYTE, tmpRegOp1, regOp1, 0);
}
+ regOp1 = tmpRegOp1;
break;
+ }
case EA_8BYTE:
break;
default:
unreached();
}
- emit->emitIns_I_la(EA_PTRSIZE, REG_RA, imm);
+ emit->emitLoadImmediate(EA_PTRSIZE, REG_RA, imm);
regs = (int)REG_RA << 5;
}
@@ -4280,7 +4302,7 @@ void CodeGen::genStackPointerConstantAdjustment(ssize_t spDelta, regNumber regTm
}
else
{
- GetEmitter()->emitIns_I_la(EA_PTRSIZE, regTmp, spDelta);
+ GetEmitter()->emitLoadImmediate(EA_PTRSIZE, regTmp, spDelta);
GetEmitter()->emitIns_R_R_R(INS_add, EA_PTRSIZE, REG_SPBASE, REG_SPBASE, regTmp);
}
}
@@ -4748,7 +4770,7 @@ void CodeGen::genSetGSSecurityCookie(regNumber initReg, bool* pInitRegZeroed)
}
else
{
- GetEmitter()->emitIns_I_la(EA_PTRSIZE, initReg, ((size_t)compiler->gsGlobalSecurityCookieAddr));
+ GetEmitter()->emitLoadImmediate(EA_PTRSIZE, initReg, ((size_t)compiler->gsGlobalSecurityCookieAddr));
GetEmitter()->emitIns_R_R_I(INS_ld, EA_PTRSIZE, initReg, initReg, 0);
}
regSet.verifyRegUsed(initReg);
@@ -5674,7 +5696,7 @@ void CodeGen::genCodeForIndexAddr(GenTreeIndexAddr* node)
}
else
{
- GetEmitter()->emitIns_I_la(EA_PTRSIZE, rsGetRsvdReg(), scale);
+ GetEmitter()->emitLoadImmediate(EA_PTRSIZE, rsGetRsvdReg(), scale);
instruction ins;
instruction ins2;
@@ -6607,10 +6629,10 @@ void CodeGen::genIntCastOverflowCheck(GenTreeCast* cast, const GenIntCastDesc& d
{
const regNumber tempReg = rsGetRsvdReg();
assert(tempReg != reg);
- GetEmitter()->emitIns_I_la(EA_8BYTE, tempReg, INT32_MAX);
+ GetEmitter()->emitLoadImmediate(EA_8BYTE, tempReg, INT32_MAX);
genJumpToThrowHlpBlk_la(SCK_OVERFLOW, INS_blt, tempReg, nullptr, reg);
- GetEmitter()->emitIns_I_la(EA_8BYTE, tempReg, INT32_MIN);
+ GetEmitter()->emitLoadImmediate(EA_8BYTE, tempReg, INT32_MIN);
genJumpToThrowHlpBlk_la(SCK_OVERFLOW, INS_blt, reg, nullptr, tempReg);
}
break;
@@ -6625,7 +6647,7 @@ void CodeGen::genIntCastOverflowCheck(GenTreeCast* cast, const GenIntCastDesc& d
if (castMaxValue > 2047)
{
assert((castMaxValue == 32767) || (castMaxValue == 65535));
- GetEmitter()->emitIns_I_la(EA_ATTR(desc.CheckSrcSize()), rsGetRsvdReg(), castMaxValue + 1);
+ GetEmitter()->emitLoadImmediate(EA_ATTR(desc.CheckSrcSize()), rsGetRsvdReg(), castMaxValue + 1);
ins = castMinValue == 0 ? INS_bgeu : INS_bge;
genJumpToThrowHlpBlk_la(SCK_OVERFLOW, ins, reg, nullptr, rsGetRsvdReg());
}
@@ -6646,7 +6668,7 @@ void CodeGen::genIntCastOverflowCheck(GenTreeCast* cast, const GenIntCastDesc& d
}
else
{
- GetEmitter()->emitIns_I_la(EA_8BYTE, rsGetRsvdReg(), castMinValue);
+ GetEmitter()->emitLoadImmediate(EA_8BYTE, rsGetRsvdReg(), castMinValue);
GetEmitter()->emitIns_R_R_R(INS_slt, EA_ATTR(desc.CheckSrcSize()), rsGetRsvdReg(), reg,
rsGetRsvdReg());
}
@@ -6985,7 +7007,7 @@ void CodeGen::genLeaInstruction(GenTreeAddrMode* lea)
regNumber tmpReg = lea->GetSingleTempReg();
// First load tmpReg with the large offset constant
- emit->emitIns_I_la(EA_PTRSIZE, tmpReg, offset);
+ emit->emitLoadImmediate(EA_PTRSIZE, tmpReg, offset);
// Then compute target reg from [memBase + tmpReg]
emit->emitIns_R_R_R(INS_add, size, lea->GetRegNum(), memBase->GetRegNum(), tmpReg);
@@ -7843,7 +7865,7 @@ void CodeGen::genFnPrologCalleeRegArgs()
tmpReg = REG_RA;
// Prepare tmpReg to possible future use
- GetEmitter()->emitIns_I_la(EA_PTRSIZE, tmpReg, baseOffset);
+ GetEmitter()->emitLoadImmediate(EA_PTRSIZE, tmpReg, baseOffset);
GetEmitter()->emitIns_R_R_R(INS_add, EA_PTRSIZE, tmpReg, tmpReg, FPbased ? REG_FPBASE : REG_SPBASE);
GetEmitter()->emitIns_S_R_R(ins_Store(storeType), size, srcRegNum, tmpReg, varNum, 0);
}
@@ -7903,7 +7925,7 @@ void CodeGen::genFnPrologCalleeRegArgs()
else
{
assert(!EA_IS_RELOC(size));
- GetEmitter()->emitIns_I_la(size, REG_SCRATCH, genTotalFrameSize());
+ GetEmitter()->emitLoadImmediate(size, REG_SCRATCH, genTotalFrameSize());
GetEmitter()->emitIns_R_R_R(INS_add, size, REG_SCRATCH, REG_SCRATCH, REG_SPBASE);
GetEmitter()->emitIns_R_R_I(INS_ld, size, REG_SCRATCH, REG_SCRATCH, 0);
}
diff --git a/src/coreclr/jit/codegenxarch.cpp b/src/coreclr/jit/codegenxarch.cpp
index 3d6291e7c11c36..6d7a973ad07520 100644
--- a/src/coreclr/jit/codegenxarch.cpp
+++ b/src/coreclr/jit/codegenxarch.cpp
@@ -3401,15 +3401,15 @@ void CodeGen::genCodeForCpBlkUnroll(GenTreeBlk* node)
assert(srcOffset < (INT32_MAX - static_cast(size)));
assert(dstOffset < (INT32_MAX - static_cast(size)));
- if (size >= XMM_REGSIZE_BYTES)
+ // Get the largest SIMD register available if the size is large enough
+ unsigned regSize = compiler->roundDownSIMDSize(size);
+
+ if ((size >= regSize) && (regSize > 0))
{
regNumber tempReg = node->GetSingleTempReg(RBM_ALLFLOAT);
instruction simdMov = simdUnalignedMovIns();
- // Get the largest SIMD register available if the size is large enough
- unsigned regSize = compiler->roundDownSIMDSize(size);
-
auto emitSimdMovs = [&]() {
if (srcLclNum != BAD_VAR_NUM)
{
diff --git a/src/coreclr/jit/compiler.cpp b/src/coreclr/jit/compiler.cpp
index c0bfbb7a37203f..df7e672d955ea1 100644
--- a/src/coreclr/jit/compiler.cpp
+++ b/src/coreclr/jit/compiler.cpp
@@ -1329,7 +1329,7 @@ void Compiler::compStartup()
emitter::emitInit();
// Static vars of ValueNumStore
- ValueNumStore::InitValueNumStoreStatics();
+ ValueNumStore::ValidateValueNumStoreStatics();
compDisplayStaticSizes(jitstdout);
}
@@ -2285,15 +2285,16 @@ void Compiler::compSetProcessor()
// the overall JIT implementation, we currently require the entire set of ISAs to be
// supported and disable AVX512 support otherwise.
- if (instructionSetFlags.HasInstructionSet(InstructionSet_AVX512BW_VL) &&
- instructionSetFlags.HasInstructionSet(InstructionSet_AVX512CD_VL) &&
- instructionSetFlags.HasInstructionSet(InstructionSet_AVX512DQ_VL))
+ if (instructionSetFlags.HasInstructionSet(InstructionSet_AVX512F))
{
+ assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512F));
+ assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512F_VL));
assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512BW));
+ assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512BW_VL));
assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512CD));
+ assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512CD_VL));
assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512DQ));
- assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512F));
- assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512F_VL));
+ assert(instructionSetFlags.HasInstructionSet(InstructionSet_AVX512DQ_VL));
instructionSetFlags.AddInstructionSet(InstructionSet_Vector512);
@@ -2308,32 +2309,6 @@ void Compiler::compSetProcessor()
preferredVectorByteLength = 256 / 8;
}
}
- else
- {
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512F);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512F_VL);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512BW);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512BW_VL);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512CD);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512CD_VL);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512DQ);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512VBMI);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL);
-
-#ifdef TARGET_AMD64
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512F_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512F_VL_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512BW_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512BW_VL_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512CD_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512CD_VL_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512DQ_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512DQ_VL_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512VBMI_X64);
- instructionSetFlags.RemoveInstructionSet(InstructionSet_AVX512VBMI_VL_X64);
-#endif // TARGET_AMD64
- }
opts.preferredVectorByteLength = preferredVectorByteLength;
#elif defined(TARGET_ARM64)
@@ -5580,7 +5555,7 @@ void Compiler::generatePatchpointInfo()
//
const int totalFrameSize = codeGen->genTotalFrameSize() + TARGET_POINTER_SIZE;
const int offsetAdjust = 0;
-#elif defined(TARGET_ARM64)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// SP is not manipulated by calls so no frame size adjustment needed.
// Local Offsets may need adjusting, if FP is at bottom of frame.
//
@@ -6830,8 +6805,8 @@ int Compiler::compCompileHelper(CORINFO_MODULE_HANDLE classPtr,
{
frameSizeUpdate = 8;
}
-#elif defined(TARGET_ARM64)
- if ((totalFrameSize % 16) != 0)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
+ if ((totalFrameSize & 0xf) != 0)
{
frameSizeUpdate = 8;
}
diff --git a/src/coreclr/jit/compiler.h b/src/coreclr/jit/compiler.h
index c363ad7d9c7eed..65379c6c8240fb 100644
--- a/src/coreclr/jit/compiler.h
+++ b/src/coreclr/jit/compiler.h
@@ -7128,9 +7128,10 @@ class Compiler
void pickGDV(GenTreeCall* call,
IL_OFFSET ilOffset,
bool isInterface,
- CORINFO_CLASS_HANDLE* classGuess,
- CORINFO_METHOD_HANDLE* methodGuess,
- unsigned* likelihood);
+ CORINFO_CLASS_HANDLE* classGuesses,
+ CORINFO_METHOD_HANDLE* methodGuesses,
+ int* candidatesCount,
+ unsigned* likelihoods);
void considerGuardedDevirtualization(GenTreeCall* call,
IL_OFFSET ilOffset,
@@ -8653,7 +8654,7 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
// Get the number of bytes in a System.Numeric.Vector for the current compilation.
// Note - cannot be used for System.Runtime.Intrinsic
- unsigned getVectorTByteLength()
+ uint32_t getVectorTByteLength()
{
// We need to report the ISA dependency to the VM so that scenarios
// such as R2R work correctly for larger vector sizes, so we always
@@ -8661,17 +8662,31 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
CLANG_FORMAT_COMMENT_ANCHOR;
#if defined(TARGET_XARCH)
- if (compExactlyDependsOn(InstructionSet_AVX2))
+ // TODO-XArch: Add support for 512-bit Vector
+ assert(!compIsaSupportedDebugOnly(InstructionSet_VectorT512));
+
+ if (compExactlyDependsOn(InstructionSet_VectorT256))
{
- // TODO-XArch-AVX512 : Return ZMM_REGSIZE_BYTES once Vector supports AVX512.
+ assert(!compIsaSupportedDebugOnly(InstructionSet_VectorT128));
return YMM_REGSIZE_BYTES;
}
- else
+ else if (compExactlyDependsOn(InstructionSet_VectorT128))
{
return XMM_REGSIZE_BYTES;
}
+ else
+ {
+ return 0;
+ }
#elif defined(TARGET_ARM64)
- return FP_REGSIZE_BYTES;
+ if (compExactlyDependsOn(InstructionSet_VectorT128))
+ {
+ return FP_REGSIZE_BYTES;
+ }
+ else
+ {
+ return 0;
+ }
#else
assert(!"getVectorTByteLength() unimplemented on target arch");
unreached();
@@ -8690,23 +8705,33 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uint32_t getMaxVectorByteLength() const
{
#if defined(FEATURE_HW_INTRINSICS) && defined(TARGET_XARCH)
- if (compOpportunisticallyDependsOn(InstructionSet_AVX))
+ if (compOpportunisticallyDependsOn(InstructionSet_AVX512F))
{
- if (compOpportunisticallyDependsOn(InstructionSet_AVX512F))
- {
- return ZMM_REGSIZE_BYTES;
- }
- else
- {
- return YMM_REGSIZE_BYTES;
- }
+ return ZMM_REGSIZE_BYTES;
}
- else
+ else if (compOpportunisticallyDependsOn(InstructionSet_AVX))
+ {
+ return YMM_REGSIZE_BYTES;
+ }
+ else if (compOpportunisticallyDependsOn(InstructionSet_SSE))
{
return XMM_REGSIZE_BYTES;
}
+ else
+ {
+ assert((JitConfig.EnableHWIntrinsic() == 0) || (JitConfig.EnableSSE() == 0));
+ return 0;
+ }
#elif defined(TARGET_ARM64)
- return FP_REGSIZE_BYTES;
+ if (compOpportunisticallyDependsOn(InstructionSet_AdvSimd))
+ {
+ return FP_REGSIZE_BYTES;
+ }
+ else
+ {
+ assert((JitConfig.EnableHWIntrinsic() == 0) || (JitConfig.EnableArm64AdvSimd() == 0));
+ return 0;
+ }
#else
assert(!"getMaxVectorByteLength() unimplemented on target arch");
unreached();
diff --git a/src/coreclr/jit/compiler.hpp b/src/coreclr/jit/compiler.hpp
index b22bc7c519b0c9..c79c5edbeea1b5 100644
--- a/src/coreclr/jit/compiler.hpp
+++ b/src/coreclr/jit/compiler.hpp
@@ -430,36 +430,34 @@ static BasicBlockVisit VisitSuccessorEHSuccessors(Compiler* comp, BasicBlock* bl
}
//------------------------------------------------------------------------------
-// VisitAllSuccsInternal: Internal helper function to visit all successors
-// (including EH successors) of a block.
+// VisitAllSuccs: Visit all successors (including EH successors) of this block.
//
// Arguments:
// comp - Compiler instance
-// block - The block
// func - Callback
//
// Returns:
// Whether or not the visiting was aborted.
//
template
-static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block, TFunc func)
+BasicBlockVisit BasicBlock::VisitAllSuccs(Compiler* comp, TFunc func)
{
- switch (block->bbJumpKind)
+ switch (bbJumpKind)
{
case BBJ_EHFILTERRET:
- RETURN_ON_ABORT(func(block->bbJumpDest));
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbJumpDest, func));
+ RETURN_ON_ABORT(func(bbJumpDest));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbJumpDest, func));
break;
case BBJ_EHFINALLYRET:
{
- EHblkDsc* ehDsc = comp->ehGetDsc(block->getHndIndex());
+ EHblkDsc* ehDsc = comp->ehGetDsc(getHndIndex());
assert(ehDsc->HasFinallyHandler());
BasicBlock* begBlk;
BasicBlock* endBlk;
- comp->ehGetCallFinallyBlockRange(block->getHndIndex(), &begBlk, &endBlk);
+ comp->ehGetCallFinallyBlockRange(getHndIndex(), &begBlk, &endBlk);
BasicBlock* finBeg = ehDsc->ebdHndBeg;
@@ -475,7 +473,7 @@ static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block,
RETURN_ON_ABORT(func(bcall->bbNext));
}
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
for (BasicBlock* bcall = begBlk; bcall != endBlk; bcall = bcall->bbNext)
{
@@ -485,7 +483,7 @@ static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block,
}
assert(bcall->isBBCallAlwaysPair());
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, bcall->bbNext, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bcall->bbNext, func));
}
break;
@@ -494,61 +492,61 @@ static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block,
case BBJ_CALLFINALLY:
case BBJ_EHCATCHRET:
case BBJ_LEAVE:
- RETURN_ON_ABORT(func(block->bbJumpDest));
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbJumpDest, func));
+ RETURN_ON_ABORT(func(bbJumpDest));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbJumpDest, func));
break;
case BBJ_ALWAYS:
- RETURN_ON_ABORT(func(block->bbJumpDest));
+ RETURN_ON_ABORT(func(bbJumpDest));
- // If "block" is a "leave helper" block (the empty BBJ_ALWAYS block
+ // If this is a "leave helper" block (the empty BBJ_ALWAYS block
// that pairs with a preceding BBJ_CALLFINALLY block to implement a
// "leave" IL instruction), then no exceptions can occur within it
// and we skip its normal EH successors.
- if (!block->isBBCallAlwaysPairTail())
+ if (!isBBCallAlwaysPairTail())
{
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
}
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbJumpDest, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbJumpDest, func));
break;
case BBJ_NONE:
- RETURN_ON_ABORT(func(block->bbNext));
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbNext, func));
+ RETURN_ON_ABORT(func(bbNext));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbNext, func));
break;
case BBJ_COND:
- RETURN_ON_ABORT(func(block->bbNext));
+ RETURN_ON_ABORT(func(bbNext));
- if (block->bbJumpDest != block->bbNext)
+ if (bbJumpDest != bbNext)
{
- RETURN_ON_ABORT(func(block->bbJumpDest));
+ RETURN_ON_ABORT(func(bbJumpDest));
}
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbNext, func));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbNext, func));
- if (block->bbJumpDest != block->bbNext)
+ if (bbJumpDest != bbNext)
{
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, block->bbJumpDest, func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, bbJumpDest, func));
}
break;
case BBJ_SWITCH:
{
- Compiler::SwitchUniqueSuccSet sd = comp->GetDescriptorForSwitch(block);
+ Compiler::SwitchUniqueSuccSet sd = comp->GetDescriptorForSwitch(this);
for (unsigned i = 0; i < sd.numDistinctSuccs; i++)
{
RETURN_ON_ABORT(func(sd.nonDuplicates[i]));
}
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
for (unsigned i = 0; i < sd.numDistinctSuccs; i++)
{
- RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, block, sd.nonDuplicates[i], func));
+ RETURN_ON_ABORT(VisitSuccessorEHSuccessors(comp, this, sd.nonDuplicates[i], func));
}
break;
@@ -557,7 +555,7 @@ static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block,
case BBJ_THROW:
case BBJ_RETURN:
case BBJ_EHFAULTRET:
- RETURN_ON_ABORT(VisitEHSuccessors(comp, block, func));
+ RETURN_ON_ABORT(VisitEHSuccessors(comp, this, func));
break;
default:
@@ -567,51 +565,6 @@ static BasicBlockVisit VisitAllSuccsInternal(Compiler* comp, BasicBlock* block,
return BasicBlockVisit::Continue;
}
-//------------------------------------------------------------------------------
-// VisitAllSuccs: Visit all successors (including EH successors) of this block.
-//
-// Arguments:
-// comp - Compiler instance
-// func - Callback
-//
-// Returns:
-// Whether or not the visiting was aborted.
-//
-template
-BasicBlockVisit BasicBlock::VisitAllSuccs(Compiler* comp, TFunc func)
-{
-#ifdef DEBUG
- // Compare callback-based successors with old iterator based successors.
- BasicBlock* succs[64];
- unsigned index = 0;
-
- VisitAllSuccsInternal(comp, this, [&succs, &index](BasicBlock* succ) {
- if (index < ArrLen(succs))
- {
- succs[index] = succ;
- }
-
- index++;
- return BasicBlockVisit::Continue;
- });
-
- unsigned index2 = 0;
- for (BasicBlock* succ : GetAllSuccs(comp))
- {
- if (index2 < ArrLen(succs))
- {
- assert(succs[index2] == succ);
- }
-
- index2++;
- }
-
- assert(index == index2);
-#endif
-
- return VisitAllSuccsInternal(comp, this, func);
-}
-
#undef RETURN_ON_ABORT
#if defined(FEATURE_EH_FUNCLETS)
diff --git a/src/coreclr/jit/emit.h b/src/coreclr/jit/emit.h
index 1d49eb69070061..9cab8e6fcea2ff 100644
--- a/src/coreclr/jit/emit.h
+++ b/src/coreclr/jit/emit.h
@@ -781,6 +781,9 @@ class emitter
unsigned _idCallRegPtr : 1; // IL indirect calls: addr in reg
unsigned _idCallAddr : 1; // IL indirect calls: can make a direct call to iiaAddr
unsigned _idNoGC : 1; // Some helpers don't get recorded in GC tables
+#if defined(TARGET_XARCH)
+ unsigned _idEvexbContext : 1; // does EVEX.b need to be set.
+#endif // TARGET_XARCH
#ifdef TARGET_ARM64
opSize _idOpSize : 3; // operand size: 0=1 , 1=2 , 2=4 , 3=8, 4=16
@@ -814,8 +817,8 @@ class emitter
////////////////////////////////////////////////////////////////////////
// Space taken up to here:
- // x86: 46 bits
- // amd64: 46 bits
+ // x86: 47 bits
+ // amd64: 47 bits
// arm: 48 bits
// arm64: 50 bits
// loongarch64: 46 bits
@@ -830,8 +833,10 @@ class emitter
#define ID_EXTRA_BITFIELD_BITS (16)
#elif defined(TARGET_ARM64)
#define ID_EXTRA_BITFIELD_BITS (18)
-#elif defined(TARGET_XARCH) || defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
+#elif defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
#define ID_EXTRA_BITFIELD_BITS (14)
+#elif defined(TARGET_XARCH)
+#define ID_EXTRA_BITFIELD_BITS (15)
#else
#error Unsupported or unset target architecture
#endif
@@ -866,8 +871,8 @@ class emitter
////////////////////////////////////////////////////////////////////////
// Space taken up to here (with/without prev offset, assuming host==target):
- // x86: 52/48 bits
- // amd64: 53/48 bits
+ // x86: 53/49 bits
+ // amd64: 54/49 bits
// arm: 54/50 bits
// arm64: 57/52 bits
// loongarch64: 53/48 bits
@@ -1136,7 +1141,7 @@ class emitter
void idCodeSize(unsigned sz)
{
// RISCV64's instrDesc is not always meaning only one instruction.
- // e.g. the `emitter::emitIns_I_la` for emitting the immediates.
+ // e.g. the `emitter::emitLoadImmediate` for emitting the immediates.
assert(sz <= 32);
_idCodeSize = sz;
}
@@ -1529,6 +1534,19 @@ class emitter
_idNoGC = val;
}
+#ifdef TARGET_XARCH
+ bool idIsEvexbContext() const
+ {
+ return _idEvexbContext != 0;
+ }
+ void idSetEvexbContext()
+ {
+ assert(_idEvexbContext == 0);
+ _idEvexbContext = 1;
+ assert(_idEvexbContext == 1);
+ }
+#endif
+
#ifdef TARGET_ARMARCH
bool idIsLclVar() const
{
@@ -3655,9 +3673,25 @@ inline unsigned emitter::emitGetInsCIargs(instrDesc* id)
//
emitAttr emitter::emitGetMemOpSize(instrDesc* id) const
{
+
emitAttr defaultSize = id->idOpSize();
instruction ins = id->idIns();
+ if (id->idIsEvexbContext())
+ {
+ // should have the assumption that Evex.b now stands for the embedded broadcast context.
+ // reference: Section 2.7.5 in Intel 64 and ia-32 architectures software developer's manual volume 2.
+ ssize_t inputSize = GetInputSizeInBytes(id);
+ switch (inputSize)
+ {
+ case 4:
+ return EA_4BYTE;
+ case 8:
+ return EA_8BYTE;
+ default:
+ unreached();
+ }
+ }
switch (ins)
{
case INS_pextrb:
diff --git a/src/coreclr/jit/emitriscv64.cpp b/src/coreclr/jit/emitriscv64.cpp
index 40a9e7c4647f36..c4cf931d2186a5 100644
--- a/src/coreclr/jit/emitriscv64.cpp
+++ b/src/coreclr/jit/emitriscv64.cpp
@@ -1117,60 +1117,64 @@ void emitter::emitIns_J_cond_la(instruction ins, BasicBlock* dst, regNumber reg1
appendToCurIG(id);
}
-void emitter::emitIns_I_la(emitAttr size, regNumber reg, ssize_t imm)
+/*****************************************************************************
+ *
+ * Emits load of 64-bit constant to register.
+ *
+ */
+void emitter::emitLoadImmediate(emitAttr size, regNumber reg, ssize_t imm)
{
+ // In the worst case a sequence of 8 instructions will be used:
+ // LUI + ADDIW + SLLI + ADDI + SLLI + ADDI + SLLI + ADDI
+ //
+ // First 2 instructions (LUI + ADDIW) load up to 31 bit. And followed
+ // sequence of (SLLI + ADDI) is used for loading remaining part by batches
+ // of up to 11 bits.
+ //
+ // Note that LUI, ADDI/W use sign extension so that's why we have to work
+ // with 31 and 11 bit batches there.
+
assert(!EA_IS_RELOC(size));
assert(isGeneralRegister(reg));
- // TODO-CQ-RISCV: at least for imm=-2*1024*1024*1024 (and similar ones) code can be simplified to "lui rd, 0x80000"
+ if (isValidSimm12(imm))
+ {
+ emitIns_R_R_I(INS_addi, size, reg, REG_R0, imm & 0xFFF);
+ return;
+ }
+
+ // TODO-RISCV64: maybe optimized via emitDataConst(), check #86790
- if (0 == ((imm + 0x800) >> 31))
+ UINT32 msb = BitOperations::BitScanReverse((uint64_t)imm);
+ UINT32 high31;
+ if (msb > 30)
{
- if (((imm + 0x800) >> 12) != 0)
- {
- emitIns_R_I(INS_lui, size, reg, ((imm + 0x800) >> 12));
- if ((imm & 0xFFF) != 0)
- {
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, reg, reg, imm & 0xFFF);
- }
- }
- else
- {
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, reg, REG_R0, imm & 0xFFF);
- }
+ high31 = (imm >> (msb - 30)) & 0x7FffFFff;
}
else
{
- UINT32 high = (imm >> 33) & 0x7fffffff;
- regNumber highReg = reg;
- if (((high + 0x800) >> 12) != 0)
- {
- emitIns_R_I(INS_lui, size, highReg, ((high + 0x800) >> 12));
- if ((high & 0xFFF) != 0)
- {
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, highReg, highReg, high & 0xFFF);
- }
- }
- else if ((high & 0xFFF) != 0)
- {
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, highReg, REG_R0, high & 0xFFF);
- }
- else
- {
- highReg = REG_R0;
- }
- UINT64 low = imm & 0x1ffffffff;
- if (highReg != REG_R0)
- {
- emitIns_R_R_I(size == EA_4BYTE ? INS_slliw : INS_slli, size, highReg, highReg, 11);
- }
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, reg, highReg, (low >> 22) & 0x7FF);
+ high31 = imm & 0x7FffFFff;
+ }
- emitIns_R_R_I(size == EA_4BYTE ? INS_slliw : INS_slli, size, reg, reg, 11);
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, reg, reg, (low >> 11) & 0x7FF);
+ // Since ADDIW use sign extension fo immediate
+ // we have to adjust higher 19 bit loaded by LUI
+ // for case when low part is bigger than 0x800.
+ UINT32 high19 = (high31 + 0x800) >> 12;
- emitIns_R_R_I(size == EA_4BYTE ? INS_slliw : INS_slli, size, reg, reg, 11);
- emitIns_R_R_I(size == EA_4BYTE ? INS_addiw : INS_addi, size, reg, reg, low & 0x7FF);
+ emitIns_R_I(INS_lui, size, reg, high19);
+ emitIns_R_R_I(INS_addiw, size, reg, reg, high31 & 0xFFF);
+
+ // And load remaining part part by batches of 11 bits size.
+ INT32 remainingShift = msb - 30;
+ while (remainingShift > 0)
+ {
+ UINT32 shift = remainingShift >= 11 ? 11 : remainingShift % 11;
+ emitIns_R_R_I(INS_slli, size, reg, reg, shift);
+
+ UINT32 mask = 0x7ff >> (11 - shift);
+ ssize_t low11 = (imm >> (remainingShift - shift)) & mask;
+ emitIns_R_R_I(INS_addi, size, reg, reg, low11);
+ remainingShift = remainingShift - shift;
}
}
@@ -3771,8 +3775,8 @@ void emitter::emitInsLoadStoreOp(instruction ins, emitAttr attr, regNumber dataR
else // large offset
{
// First load/store tmpReg with the large offset constant
- emitIns_I_la(EA_PTRSIZE, tmpReg,
- offset); // codeGen->instGen_Set_Reg_To_Imm(EA_PTRSIZE, tmpReg, offset);
+ emitLoadImmediate(EA_PTRSIZE, tmpReg,
+ offset); // codeGen->instGen_Set_Reg_To_Imm(EA_PTRSIZE, tmpReg, offset);
// Then add the base register
// rd = rd + base
emitIns_R_R_R(INS_add, addType, tmpReg, tmpReg, memBase->GetRegNum());
@@ -3899,7 +3903,7 @@ void emitter::emitInsLoadStoreOp(instruction ins, emitAttr attr, regNumber dataR
regNumber tmpReg = indir->GetSingleTempReg();
// First load/store tmpReg with the large offset constant
- emitIns_I_la(EA_PTRSIZE, tmpReg, offset);
+ emitLoadImmediate(EA_PTRSIZE, tmpReg, offset);
// codeGen->instGen_Set_Reg_To_Imm(EA_PTRSIZE, tmpReg, offset);
// Then load/store dataReg from/to [memBase + tmpReg]
@@ -4160,7 +4164,7 @@ regNumber emitter::emitInsTernary(instruction ins, emitAttr attr, GenTree* dst,
}
else
{
- emitIns_R_R_R(INS_mulhu, attr, codeGen->rsGetRsvdReg(), src1->GetRegNum(), src2->GetRegNum());
+ emitIns_R_R_R(INS_mulh, attr, codeGen->rsGetRsvdReg(), src1->GetRegNum(), src2->GetRegNum());
}
}
}
diff --git a/src/coreclr/jit/emitriscv64.h b/src/coreclr/jit/emitriscv64.h
index f9eb23be2f8de8..a572ee9d2cf8c9 100644
--- a/src/coreclr/jit/emitriscv64.h
+++ b/src/coreclr/jit/emitriscv64.h
@@ -32,7 +32,8 @@ void emitDisInsName(code_t code, const BYTE* addr, instrDesc* id);
#endif // DEBUG
void emitIns_J_cond_la(instruction ins, BasicBlock* dst, regNumber reg1 = REG_R0, regNumber reg2 = REG_R0);
-void emitIns_I_la(emitAttr attr, regNumber reg, ssize_t imm);
+
+void emitLoadImmediate(emitAttr attr, regNumber reg, ssize_t imm);
/************************************************************************/
/* Private members that deal with target-dependent instr. descriptors */
diff --git a/src/coreclr/jit/emitxarch.cpp b/src/coreclr/jit/emitxarch.cpp
index db096a21f09699..e1586dcf9ff1b9 100644
--- a/src/coreclr/jit/emitxarch.cpp
+++ b/src/coreclr/jit/emitxarch.cpp
@@ -1231,9 +1231,10 @@ bool emitter::TakesEvexPrefix(const instrDesc* id) const
#define DEFAULT_BYTE_EVEX_PREFIX_MASK 0xFFFFFFFF00000000ULL
#define LBIT_IN_BYTE_EVEX_PREFIX 0x0000002000000000ULL
#define LPRIMEBIT_IN_BYTE_EVEX_PREFIX 0x0000004000000000ULL
+#define EVEX_B_BIT 0x0000001000000000ULL
//------------------------------------------------------------------------
-// AddEvexPrefix: Add default EVEX perfix with only LL' bits set.
+// AddEvexPrefix: Add default EVEX prefix with only LL' bits set.
//
// Arguments:
// ins -- processor instruction to check.
@@ -1268,6 +1269,22 @@ emitter::code_t emitter::AddEvexPrefix(instruction ins, code_t code, emitAttr at
return code;
}
+//------------------------------------------------------------------------
+// AddEvexPrefix: set Evex.b bit if EvexbContext is set in instruction descritor.
+//
+// Arguments:
+// code -- opcode bits.
+//
+// Return Value:
+// encoded code with Evex.b set if needed.
+//
+emitter::code_t emitter::AddEvexbBit(code_t code)
+{
+ assert(hasEvexPrefix(code));
+ code |= EVEX_B_BIT;
+ return code;
+}
+
// Returns true if this instruction requires a VEX prefix
// All AVX instructions require a VEX prefix
bool emitter::TakesVexPrefix(instruction ins) const
@@ -6667,7 +6684,8 @@ void emitter::emitIns_R_S_I(instruction ins, emitAttr attr, regNumber reg1, int
emitCurIGsize += sz;
}
-void emitter::emitIns_R_R_A(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, GenTreeIndir* indir)
+void emitter::emitIns_R_R_A(
+ instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, GenTreeIndir* indir, insOpts instOptions)
{
assert(IsAvx512OrPriorInstruction(ins));
assert(IsThreeOperandAVXInstruction(ins));
@@ -6678,6 +6696,11 @@ void emitter::emitIns_R_R_A(instruction ins, emitAttr attr, regNumber reg1, regN
id->idIns(ins);
id->idReg1(reg1);
id->idReg2(reg2);
+ if (instOptions == INS_OPTS_EVEX_b)
+ {
+ assert(UseEvexEncoding());
+ id->idSetEvexbContext();
+ }
emitHandleMemOp(indir, id, (ins == INS_mulx) ? IF_RWR_RWR_ARD : emitInsModeFormat(ins, IF_RRD_RRD_ARD), ins);
@@ -6778,8 +6801,13 @@ void emitter::emitIns_R_AR_R(instruction ins,
emitCurIGsize += sz;
}
-void emitter::emitIns_R_R_C(
- instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, CORINFO_FIELD_HANDLE fldHnd, int offs)
+void emitter::emitIns_R_R_C(instruction ins,
+ emitAttr attr,
+ regNumber reg1,
+ regNumber reg2,
+ CORINFO_FIELD_HANDLE fldHnd,
+ int offs,
+ insOpts instOptions)
{
assert(IsAvx512OrPriorInstruction(ins));
assert(IsThreeOperandAVXInstruction(ins));
@@ -6797,6 +6825,11 @@ void emitter::emitIns_R_R_C(
id->idReg1(reg1);
id->idReg2(reg2);
id->idAddr()->iiaFieldHnd = fldHnd;
+ if (instOptions == INS_OPTS_EVEX_b)
+ {
+ assert(UseEvexEncoding());
+ id->idSetEvexbContext();
+ }
UNATIVE_OFFSET sz = emitInsSizeCV(id, insCodeRM(ins));
id->idCodeSize(sz);
@@ -6829,7 +6862,8 @@ void emitter::emitIns_R_R_R(instruction ins, emitAttr attr, regNumber targetReg,
emitCurIGsize += sz;
}
-void emitter::emitIns_R_R_S(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, int varx, int offs)
+void emitter::emitIns_R_R_S(
+ instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, int varx, int offs, insOpts instOptions)
{
assert(IsAvx512OrPriorInstruction(ins));
assert(IsThreeOperandAVXInstruction(ins));
@@ -6842,6 +6876,11 @@ void emitter::emitIns_R_R_S(instruction ins, emitAttr attr, regNumber reg1, regN
id->idReg2(reg2);
id->idAddr()->iiaLclVar.initLclVarAddr(varx, offs);
+ if (instOptions == INS_OPTS_EVEX_b)
+ {
+ assert(UseEvexEncoding());
+ id->idSetEvexbContext();
+ }
#ifdef DEBUG
id->idDebugOnlyInfo()->idVarRefOffs = emitVarRefOffs;
#endif
@@ -8134,14 +8173,15 @@ void emitter::emitIns_SIMD_R_R_I(instruction ins, emitAttr attr, regNumber targe
// indir -- The GenTreeIndir used for the memory address
//
void emitter::emitIns_SIMD_R_R_A(
- instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, GenTreeIndir* indir)
+ instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, GenTreeIndir* indir, insOpts instOptions)
{
if (UseSimdEncoding())
{
- emitIns_R_R_A(ins, attr, targetReg, op1Reg, indir);
+ emitIns_R_R_A(ins, attr, targetReg, op1Reg, indir, instOptions);
}
else
{
+ assert(instOptions == INS_OPTS_NONE);
emitIns_Mov(INS_movaps, attr, targetReg, op1Reg, /* canSkip */ true);
emitIns_R_A(ins, attr, targetReg, indir);
}
@@ -8159,15 +8199,21 @@ void emitter::emitIns_SIMD_R_R_A(
// fldHnd -- The CORINFO_FIELD_HANDLE used for the memory address
// offs -- The offset added to the memory address from fldHnd
//
-void emitter::emitIns_SIMD_R_R_C(
- instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, CORINFO_FIELD_HANDLE fldHnd, int offs)
+void emitter::emitIns_SIMD_R_R_C(instruction ins,
+ emitAttr attr,
+ regNumber targetReg,
+ regNumber op1Reg,
+ CORINFO_FIELD_HANDLE fldHnd,
+ int offs,
+ insOpts instOptions)
{
if (UseSimdEncoding())
{
- emitIns_R_R_C(ins, attr, targetReg, op1Reg, fldHnd, offs);
+ emitIns_R_R_C(ins, attr, targetReg, op1Reg, fldHnd, offs, instOptions);
}
else
{
+ assert(instOptions == INS_OPTS_NONE);
emitIns_Mov(INS_movaps, attr, targetReg, op1Reg, /* canSkip */ true);
emitIns_R_C(ins, attr, targetReg, fldHnd, offs);
}
@@ -8222,14 +8268,15 @@ void emitter::emitIns_SIMD_R_R_R(
// offs -- The offset added to the memory address from varx
//
void emitter::emitIns_SIMD_R_R_S(
- instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, int varx, int offs)
+ instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, int varx, int offs, insOpts instOptions)
{
if (UseSimdEncoding())
{
- emitIns_R_R_S(ins, attr, targetReg, op1Reg, varx, offs);
+ emitIns_R_R_S(ins, attr, targetReg, op1Reg, varx, offs, instOptions);
}
else
{
+ assert(instOptions == INS_OPTS_NONE);
emitIns_Mov(INS_movaps, attr, targetReg, op1Reg, /* canSkip */ true);
emitIns_R_S(ins, attr, targetReg, varx, offs);
}
@@ -15717,7 +15764,7 @@ BYTE* emitter::emitOutputLJ(insGroup* ig, BYTE* dst, instrDesc* i)
// Return Value:
// size in bytes.
//
-ssize_t emitter::GetInputSizeInBytes(instrDesc* id)
+ssize_t emitter::GetInputSizeInBytes(instrDesc* id) const
{
insFlags inputSize = static_cast((CodeGenInterface::instInfo[id->idIns()] & Input_Mask));
@@ -18809,6 +18856,7 @@ emitter::insExecutionCharacteristics emitter::getInsExecutionCharacteristics(ins
case INS_pcmpgtq:
case INS_psadbw:
case INS_vdbpsadbw:
+ case INS_vpcmpgtq:
case INS_vpermps:
case INS_vpermpd:
case INS_vpermpd_reg:
@@ -19243,10 +19291,6 @@ emitter::insExecutionCharacteristics emitter::getInsExecutionCharacteristics(ins
break;
}
- case INS_kmovb_msk:
- case INS_kmovw_msk:
- case INS_kmovd_msk:
- case INS_kmovq_msk:
case INS_kmovb_gpr:
case INS_kmovw_gpr:
case INS_kmovd_gpr:
@@ -19257,6 +19301,16 @@ emitter::insExecutionCharacteristics emitter::getInsExecutionCharacteristics(ins
break;
}
+ case INS_kmovb_msk:
+ case INS_kmovw_msk:
+ case INS_kmovd_msk:
+ case INS_kmovq_msk:
+ {
+ result.insLatency += PERFSCORE_LATENCY_1C;
+ result.insThroughput = PERFSCORE_THROUGHPUT_1C;
+ break;
+ }
+
case INS_vpcmpb:
case INS_vpcmpw:
case INS_vpcmpd:
@@ -19286,13 +19340,72 @@ emitter::insExecutionCharacteristics emitter::getInsExecutionCharacteristics(ins
break;
}
+ case INS_kandb:
+ case INS_kandd:
+ case INS_kandq:
+ case INS_kandw:
+ case INS_kandnb:
+ case INS_kandnd:
+ case INS_kandnq:
+ case INS_kandnw:
+ case INS_knotb:
+ case INS_knotd:
+ case INS_knotq:
+ case INS_knotw:
+ case INS_korb:
+ case INS_kord:
+ case INS_korq:
+ case INS_korw:
+ case INS_kxnorb:
+ case INS_kxnord:
+ case INS_kxnorq:
+ case INS_kxnorw:
+ case INS_kxorb:
+ case INS_kxord:
+ case INS_kxorq:
+ case INS_kxorw:
+ {
+ result.insLatency += PERFSCORE_LATENCY_1C;
+ result.insThroughput = PERFSCORE_THROUGHPUT_1C;
+ break;
+ }
+
case INS_kortestb:
- case INS_kortestw:
case INS_kortestd:
case INS_kortestq:
+ case INS_kortestw:
+ case INS_ktestb:
+ case INS_ktestd:
+ case INS_ktestq:
+ case INS_ktestw:
{
+ // Keep these in a separate group as there isn't a documented latency
+ // Similar instructions have a 1 cycle latency, however
+
result.insLatency += PERFSCORE_LATENCY_1C;
result.insThroughput = PERFSCORE_THROUGHPUT_1C;
+
+ break;
+ }
+
+ case INS_kaddb:
+ case INS_kaddd:
+ case INS_kaddq:
+ case INS_kaddw:
+ case INS_kshiftlb:
+ case INS_kshiftld:
+ case INS_kshiftlq:
+ case INS_kshiftlw:
+ case INS_kshiftrb:
+ case INS_kshiftrd:
+ case INS_kshiftrq:
+ case INS_kshiftrw:
+ case INS_kunpckbw:
+ case INS_kunpckdq:
+ case INS_kunpckwd:
+ {
+ result.insLatency += PERFSCORE_LATENCY_4C;
+ result.insThroughput = PERFSCORE_THROUGHPUT_1C;
break;
}
diff --git a/src/coreclr/jit/emitxarch.h b/src/coreclr/jit/emitxarch.h
index d70f2b6d92a409..082686fda18439 100644
--- a/src/coreclr/jit/emitxarch.h
+++ b/src/coreclr/jit/emitxarch.h
@@ -294,6 +294,7 @@ bool hasEvexPrefix(code_t code)
return (code & EVEX_PREFIX_MASK) == EVEX_PREFIX_CODE;
}
code_t AddEvexPrefix(instruction ins, code_t code, emitAttr attr);
+code_t AddEvexbBit(code_t code);
//------------------------------------------------------------------------
// AddSimdPrefixIfNeeded: Add the correct SIMD prefix if required.
@@ -314,6 +315,10 @@ code_t AddSimdPrefixIfNeeded(const instrDesc* id, code_t code, emitAttr size)
if (TakesEvexPrefix(id))
{
code = AddEvexPrefix(ins, code, size);
+ if (id->idIsEvexbContext())
+ {
+ code = AddEvexbBit(code);
+ }
}
else if (TakesVexPrefix(ins))
{
@@ -385,7 +390,7 @@ bool codeEvexMigrationCheck(code_t code)
return hasEvexPrefix(code);
}
-ssize_t GetInputSizeInBytes(instrDesc* id);
+ssize_t GetInputSizeInBytes(instrDesc* id) const;
bool containsAVXInstruction = false;
bool ContainsAVX()
@@ -572,7 +577,12 @@ void emitIns_R_C_I(instruction ins, emitAttr attr, regNumber reg1, CORINFO_FIELD
void emitIns_R_S_I(instruction ins, emitAttr attr, regNumber reg1, int varx, int offs, int ival);
-void emitIns_R_R_A(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, GenTreeIndir* indir);
+void emitIns_R_R_A(instruction ins,
+ emitAttr attr,
+ regNumber reg1,
+ regNumber reg2,
+ GenTreeIndir* indir,
+ insOpts instOptions = INS_OPTS_NONE);
void emitIns_R_R_AR(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, regNumber base, int offs);
@@ -585,10 +595,21 @@ void emitIns_R_AR_R(instruction ins,
int scale,
int offs);
-void emitIns_R_R_C(
- instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, CORINFO_FIELD_HANDLE fldHnd, int offs);
-
-void emitIns_R_R_S(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, int varx, int offs);
+void emitIns_R_R_C(instruction ins,
+ emitAttr attr,
+ regNumber reg1,
+ regNumber reg2,
+ CORINFO_FIELD_HANDLE fldHnd,
+ int offs,
+ insOpts instOptions = INS_OPTS_NONE);
+
+void emitIns_R_R_S(instruction ins,
+ emitAttr attr,
+ regNumber reg1,
+ regNumber reg2,
+ int varx,
+ int offs,
+ insOpts instOptions = INS_OPTS_NONE);
void emitIns_R_R_R(instruction ins, emitAttr attr, regNumber reg1, regNumber reg2, regNumber reg3);
@@ -691,11 +712,27 @@ void emitIns_AX_R(instruction ins, emitAttr attr, regNumber ireg, regNumber reg,
void emitIns_SIMD_R_R_I(instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, int ival);
-void emitIns_SIMD_R_R_A(instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, GenTreeIndir* indir);
-void emitIns_SIMD_R_R_C(
- instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, CORINFO_FIELD_HANDLE fldHnd, int offs);
+void emitIns_SIMD_R_R_A(instruction ins,
+ emitAttr attr,
+ regNumber targetReg,
+ regNumber op1Reg,
+ GenTreeIndir* indir,
+ insOpts instOptions = INS_OPTS_NONE);
+void emitIns_SIMD_R_R_C(instruction ins,
+ emitAttr attr,
+ regNumber targetReg,
+ regNumber op1Reg,
+ CORINFO_FIELD_HANDLE fldHnd,
+ int offs,
+ insOpts instOptions = INS_OPTS_NONE);
void emitIns_SIMD_R_R_R(instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, regNumber op2Reg);
-void emitIns_SIMD_R_R_S(instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, int varx, int offs);
+void emitIns_SIMD_R_R_S(instruction ins,
+ emitAttr attr,
+ regNumber targetReg,
+ regNumber op1Reg,
+ int varx,
+ int offs,
+ insOpts instOptions = INS_OPTS_NONE);
void emitIns_SIMD_R_R_A_I(
instruction ins, emitAttr attr, regNumber targetReg, regNumber op1Reg, GenTreeIndir* indir, int ival);
@@ -840,7 +877,6 @@ inline bool emitIsUncondJump(instrDesc* jmp)
//------------------------------------------------------------------------
// HasEmbeddedBroadcast: Do we consider embedded broadcast while encoding.
-// TODO-XArch-AVX512: Add eventual check on the instrDesc
//
// Arguments:
// id - Instruction descriptor.
@@ -850,7 +886,7 @@ inline bool emitIsUncondJump(instrDesc* jmp)
//
inline bool HasEmbeddedBroadcast(const instrDesc* id) const
{
- return false;
+ return id->idIsEvexbContext();
}
inline bool HasHighSIMDReg(const instrDesc* id) const;
diff --git a/src/coreclr/jit/gcencode.cpp b/src/coreclr/jit/gcencode.cpp
index eaa5a584261a37..5cf1eece3e47dc 100644
--- a/src/coreclr/jit/gcencode.cpp
+++ b/src/coreclr/jit/gcencode.cpp
@@ -3887,7 +3887,7 @@ void GCInfo::gcInfoBlockHdrSave(GcInfoEncoder* gcInfoEncoder, unsigned methodSiz
//
const int osrOffset = ppInfo->GenericContextArgOffset() - 2 * REGSIZE_BYTES;
assert(offset == osrOffset);
-#elif defined(TARGET_ARM64)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// PP info has virtual offset. This is also the caller SP offset.
//
const int osrOffset = ppInfo->GenericContextArgOffset();
@@ -3930,7 +3930,7 @@ void GCInfo::gcInfoBlockHdrSave(GcInfoEncoder* gcInfoEncoder, unsigned methodSiz
//
const int osrOffset = ppInfo->KeptAliveThisOffset() - 2 * REGSIZE_BYTES;
assert(offset == osrOffset);
-#elif defined(TARGET_ARM64)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// PP info has virtual offset. This is also the caller SP offset.
//
const int osrOffset = ppInfo->KeptAliveThisOffset();
diff --git a/src/coreclr/jit/gentree.cpp b/src/coreclr/jit/gentree.cpp
index 833278adf22292..43a2556313adab 100644
--- a/src/coreclr/jit/gentree.cpp
+++ b/src/coreclr/jit/gentree.cpp
@@ -21,13 +21,13 @@ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
/*****************************************************************************/
const unsigned char GenTree::gtOperKindTable[] = {
-#define GTNODE(en, st, cm, ok) ((ok)>K_MASK) + GTK_COMMUTE *cm,
+#define GTNODE(en, st, cm, ivn, ok) ((ok)>K_MASK) + GTK_COMMUTE *cm,
#include "gtlist.h"
};
#ifdef DEBUG
const GenTreeDebugOperKind GenTree::gtDebugOperKindTable[] = {
-#define GTNODE(en, st, cm, ok) static_cast((ok)&DBK_MASK),
+#define GTNODE(en, st, cm, ivn, ok) static_cast((ok)&DBK_MASK),
#include "gtlist.h"
};
#endif // DEBUG
@@ -171,7 +171,7 @@ static void printIndent(IndentStack* indentStack)
#if defined(DEBUG) || NODEBASH_STATS || MEASURE_NODE_SIZE || COUNT_AST_OPERS || DUMP_FLOWGRAPHS
static const char* opNames[] = {
-#define GTNODE(en, st, cm, ok) #en,
+#define GTNODE(en, st, cm, ivn, ok) #en,
#include "gtlist.h"
};
@@ -187,7 +187,7 @@ const char* GenTree::OpName(genTreeOps op)
#if MEASURE_NODE_SIZE
static const char* opStructNames[] = {
-#define GTNODE(en, st, cm, ok) #st,
+#define GTNODE(en, st, cm, ivn, ok) #st,
#include "gtlist.h"
};
@@ -214,7 +214,7 @@ unsigned char GenTree::s_gtNodeSizes[GT_COUNT + 1];
#if NODEBASH_STATS || MEASURE_NODE_SIZE || COUNT_AST_OPERS
unsigned char GenTree::s_gtTrueSizes[GT_COUNT + 1]{
-#define GTNODE(en, st, cm, ok) sizeof(st),
+#define GTNODE(en, st, cm, ivn, ok) sizeof(st),
#include "gtlist.h"
};
@@ -12400,11 +12400,21 @@ void Compiler::gtDispTree(GenTree* tree,
printf(" (FramesRoot last use)");
}
- if (((call->gtFlags & GTF_CALL_INLINE_CANDIDATE) != 0) &&
- (call->GetSingleInlineCandidateInfo() != nullptr) &&
- (call->GetSingleInlineCandidateInfo()->exactContextHnd != nullptr))
+ if ((call->gtFlags & GTF_CALL_INLINE_CANDIDATE) != 0)
{
- printf(" (exactContextHnd=0x%p)", dspPtr(call->GetSingleInlineCandidateInfo()->exactContextHnd));
+ InlineCandidateInfo* inlineInfo;
+ if (call->IsGuardedDevirtualizationCandidate())
+ {
+ inlineInfo = call->GetGDVCandidateInfo(0);
+ }
+ else
+ {
+ inlineInfo = call->GetSingleInlineCandidateInfo();
+ }
+ if ((inlineInfo != nullptr) && (inlineInfo->exactContextHnd != nullptr))
+ {
+ printf(" (exactContextHnd=0x%p)", dspPtr(inlineInfo->exactContextHnd));
+ }
}
gtDispCommonEndLine(tree);
@@ -17878,6 +17888,7 @@ CORINFO_CLASS_HANDLE Compiler::gtGetClassHandle(GenTree* tree, bool* pIsExact, b
{
// if we managed to get a class handle it's definitely not null
*pIsNonNull = true;
+ *pIsExact = true;
}
}
break;
@@ -19065,6 +19076,18 @@ bool GenTree::isContainableHWIntrinsic() const
return true;
}
+ case NI_SSE3_MoveAndDuplicate:
+ case NI_AVX_BroadcastScalarToVector128:
+ case NI_AVX2_BroadcastScalarToVector128:
+ case NI_AVX_BroadcastScalarToVector256:
+ case NI_AVX2_BroadcastScalarToVector256:
+ case NI_AVX512F_BroadcastScalarToVector512:
+ {
+ // These intrinsic operations are contained as part of the operand of embedded broadcast compatible
+ // instruction
+ return true;
+ }
+
default:
{
return false;
@@ -20418,7 +20441,20 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
#if defined(TARGET_XARCH)
case GT_EQ:
{
- if (simdSize == 32)
+ if (simdSize == 64)
+ {
+ assert(IsBaselineVector512IsaSupportedDebugOnly());
+
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_CompareEqual;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_CompareEqual;
+ }
+ }
+ else if (simdSize == 32)
{
assert(compIsaSupportedDebugOnly(InstructionSet_AVX));
@@ -20432,17 +20468,15 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
intrinsic = NI_AVX2_CompareEqual;
}
}
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_AVX512F_CompareEqualSpecial;
- }
else if (simdBaseType == TYP_FLOAT)
{
+ assert((simdSize == 8) || (simdSize == 12) || (simdSize == 16));
intrinsic = NI_SSE_CompareEqual;
}
else if (varTypeIsLong(simdBaseType))
{
+ assert(simdSize == 16);
+
if (compOpportunisticallyDependsOn(InstructionSet_SSE41))
{
intrinsic = NI_SSE41_CompareEqual;
@@ -20471,6 +20505,7 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
}
else
{
+ assert(simdSize == 16);
intrinsic = NI_SSE2_CompareEqual;
}
break;
@@ -20478,6 +20513,37 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
case GT_GE:
{
+ if (IsBaselineVector512IsaSupported())
+ {
+ if (simdSize == 64)
+ {
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_CompareGreaterThanOrEqual;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_CompareGreaterThanOrEqual;
+ }
+ break;
+ }
+ else if (!varTypeIsFloating(simdBaseType))
+ {
+ assert((simdSize == 16) || (simdSize == 32));
+
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_VL_CompareGreaterThanOrEqual;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_VL_CompareGreaterThanOrEqual;
+ }
+
+ break;
+ }
+ }
+
if (simdSize == 32)
{
assert(compIsaSupportedDebugOnly(InstructionSet_AVX));
@@ -20487,17 +20553,14 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
intrinsic = NI_AVX_CompareGreaterThanOrEqual;
}
}
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_AVX512F_CompareGreaterThanOrEqualSpecial;
- }
else if (simdBaseType == TYP_FLOAT)
{
+ assert((simdSize == 8) || (simdSize == 12) || (simdSize == 16));
intrinsic = NI_SSE_CompareGreaterThanOrEqual;
}
else if (simdBaseType == TYP_DOUBLE)
{
+ assert(simdSize == 16);
intrinsic = NI_SSE2_CompareGreaterThanOrEqual;
}
@@ -20545,6 +20608,37 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
case GT_GT:
{
+ if (IsBaselineVector512IsaSupported())
+ {
+ if (simdSize == 64)
+ {
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_CompareGreaterThan;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_CompareGreaterThan;
+ }
+ break;
+ }
+ else if (varTypeIsUnsigned(simdBaseType))
+ {
+ assert((simdSize == 16) || (simdSize == 32));
+
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_VL_CompareGreaterThan;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_VL_CompareGreaterThan;
+ }
+
+ break;
+ }
+ }
+
if (varTypeIsUnsigned(simdBaseType))
{
// Vector of byte, ushort, uint and ulong:
@@ -20638,17 +20732,15 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
intrinsic = NI_AVX2_CompareGreaterThan;
}
}
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_AVX512F_CompareGreaterThanSpecial;
- }
else if (simdBaseType == TYP_FLOAT)
{
+ assert((simdSize == 8) || (simdSize == 12) || (simdSize == 16));
intrinsic = NI_SSE_CompareGreaterThan;
}
else if (varTypeIsLong(simdBaseType))
{
+ assert(simdSize == 16);
+
if (compOpportunisticallyDependsOn(InstructionSet_SSE42))
{
intrinsic = NI_SSE42_CompareGreaterThan;
@@ -20710,6 +20802,7 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
}
else
{
+ assert(simdSize == 16);
intrinsic = NI_SSE2_CompareGreaterThan;
}
break;
@@ -20717,6 +20810,37 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
case GT_LE:
{
+ if (IsBaselineVector512IsaSupported())
+ {
+ if (simdSize == 64)
+ {
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_CompareLessThanOrEqual;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_CompareLessThanOrEqual;
+ }
+ break;
+ }
+ else if (!varTypeIsFloating(simdBaseType))
+ {
+ assert((simdSize == 16) || (simdSize == 32));
+
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_VL_CompareLessThanOrEqual;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_VL_CompareLessThanOrEqual;
+ }
+
+ break;
+ }
+ }
+
if (simdSize == 32)
{
assert(compIsaSupportedDebugOnly(InstructionSet_AVX));
@@ -20726,17 +20850,14 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
intrinsic = NI_AVX_CompareLessThanOrEqual;
}
}
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_AVX512F_CompareLessThanOrEqualSpecial;
- }
else if (simdBaseType == TYP_FLOAT)
{
+ assert((simdSize == 8) || (simdSize == 12) || (simdSize == 16));
intrinsic = NI_SSE_CompareLessThanOrEqual;
}
else if (simdBaseType == TYP_DOUBLE)
{
+ assert(simdSize == 16);
intrinsic = NI_SSE2_CompareLessThanOrEqual;
}
@@ -20784,6 +20905,37 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
case GT_LT:
{
+ if (IsBaselineVector512IsaSupported())
+ {
+ if (simdSize == 64)
+ {
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_CompareLessThan;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_CompareLessThan;
+ }
+ break;
+ }
+ else if (varTypeIsUnsigned(simdBaseType))
+ {
+ assert((simdSize == 16) || (simdSize == 32));
+
+ if (varTypeIsSmall(simdBaseType))
+ {
+ intrinsic = NI_AVX512BW_VL_CompareLessThan;
+ }
+ else
+ {
+ intrinsic = NI_AVX512F_VL_CompareLessThan;
+ }
+
+ break;
+ }
+ }
+
if (varTypeIsUnsigned(simdBaseType))
{
// Vector of byte, ushort, uint and ulong:
@@ -20877,17 +21029,15 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
intrinsic = NI_AVX2_CompareLessThan;
}
}
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_AVX512F_CompareLessThanSpecial;
- }
else if (simdBaseType == TYP_FLOAT)
{
+ assert((simdSize == 8) || (simdSize == 12) || (simdSize == 16));
intrinsic = NI_SSE_CompareLessThan;
}
else if (varTypeIsLong(simdBaseType))
{
+ assert(simdSize == 16);
+
if (compOpportunisticallyDependsOn(InstructionSet_SSE42))
{
intrinsic = NI_SSE42_CompareLessThan;
@@ -20949,6 +21099,7 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
}
else
{
+ assert(simdSize == 16);
intrinsic = NI_SSE2_CompareLessThan;
}
break;
@@ -21034,15 +21185,7 @@ GenTree* Compiler::gtNewSimdCmpOpNode(
assert(intrinsic != NI_Illegal);
#if defined(TARGET_XARCH)
- if (simdSize != 64)
- {
- return gtNewSimdHWIntrinsicNode(type, op1, op2, intrinsic, simdBaseJitType, simdSize);
- }
- else
- {
- GenTree* cmp = gtNewSimdHWIntrinsicNode(TYP_MASK, op1, op2, intrinsic, simdBaseJitType, simdSize);
- return gtNewSimdHWIntrinsicNode(type, cmp, NI_AVX512F_MoveMaskToVectorSpecial, simdBaseJitType, simdSize);
- }
+ return gtNewSimdHWIntrinsicNode(type, op1, op2, intrinsic, simdBaseJitType, simdSize);
#else
return gtNewSimdHWIntrinsicNode(type, op1, op2, intrinsic, simdBaseJitType, simdSize);
#endif
@@ -21093,125 +21236,8 @@ GenTree* Compiler::gtNewSimdCmpOpAllNode(
}
case GT_GE:
- {
- // We want to generate a comparison along the lines of
- // GT_XX(op1, op2).As() == Vector128.AllBitsSet
-
- if (simdSize == 32)
- {
- // TODO-XArch-CQ: It's a non-trivial amount of work to support these
- // for floating-point while only utilizing AVX. It would require, among
- // other things, inverting the comparison and potentially support for a
- // new Avx.TestNotZ intrinsic to ensure the codegen remains efficient.
- assert(compIsaSupportedDebugOnly(InstructionSet_AVX2));
- intrinsic = NI_Vector256_op_Equality;
- }
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_Vector512_GreaterThanOrEqualAll;
- break;
- }
- else
- {
- intrinsic = NI_Vector128_op_Equality;
- }
-
- op1 = gtNewSimdCmpOpNode(op, simdType, op1, op2, simdBaseJitType, simdSize);
- op2 = gtNewAllBitsSetConNode(simdType);
-
- if (simdBaseType == TYP_FLOAT)
- {
- simdBaseType = TYP_INT;
- simdBaseJitType = CORINFO_TYPE_INT;
- }
- else if (simdBaseType == TYP_DOUBLE)
- {
- simdBaseType = TYP_LONG;
- simdBaseJitType = CORINFO_TYPE_LONG;
- }
- break;
- }
case GT_GT:
- {
- // We want to generate a comparison along the lines of
- // GT_XX(op1, op2).As() == Vector128.AllBitsSet
-
- if (simdSize == 32)
- {
- // TODO-XArch-CQ: It's a non-trivial amount of work to support these
- // for floating-point while only utilizing AVX. It would require, among
- // other things, inverting the comparison and potentially support for a
- // new Avx.TestNotZ intrinsic to ensure the codegen remains efficient.
- assert(compIsaSupportedDebugOnly(InstructionSet_AVX2));
- intrinsic = NI_Vector256_op_Equality;
- }
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_Vector512_GreaterThanAll;
- break;
- }
- else
- {
- intrinsic = NI_Vector128_op_Equality;
- }
-
- op1 = gtNewSimdCmpOpNode(op, simdType, op1, op2, simdBaseJitType, simdSize);
- op2 = gtNewAllBitsSetConNode(simdType);
-
- if (simdBaseType == TYP_FLOAT)
- {
- simdBaseType = TYP_INT;
- simdBaseJitType = CORINFO_TYPE_INT;
- }
- else if (simdBaseType == TYP_DOUBLE)
- {
- simdBaseType = TYP_LONG;
- simdBaseJitType = CORINFO_TYPE_LONG;
- }
- break;
- }
case GT_LE:
- {
- // We want to generate a comparison along the lines of
- // GT_XX(op1, op2).As() == Vector128.AllBitsSet
-
- if (simdSize == 32)
- {
- // TODO-XArch-CQ: It's a non-trivial amount of work to support these
- // for floating-point while only utilizing AVX. It would require, among
- // other things, inverting the comparison and potentially support for a
- // new Avx.TestNotZ intrinsic to ensure the codegen remains efficient.
- assert(compIsaSupportedDebugOnly(InstructionSet_AVX2));
- intrinsic = NI_Vector256_op_Equality;
- }
- else if (simdSize == 64)
- {
- assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_Vector512_LessThanOrEqualAll;
- break;
- }
- else
- {
- intrinsic = NI_Vector128_op_Equality;
- }
-
- op1 = gtNewSimdCmpOpNode(op, simdType, op1, op2, simdBaseJitType, simdSize);
- op2 = gtNewAllBitsSetConNode(simdType);
-
- if (simdBaseType == TYP_FLOAT)
- {
- simdBaseType = TYP_INT;
- simdBaseJitType = CORINFO_TYPE_INT;
- }
- else if (simdBaseType == TYP_DOUBLE)
- {
- simdBaseType = TYP_LONG;
- simdBaseJitType = CORINFO_TYPE_LONG;
- }
- break;
- }
case GT_LT:
{
// We want to generate a comparison along the lines of
@@ -21229,8 +21255,7 @@ GenTree* Compiler::gtNewSimdCmpOpAllNode(
else if (simdSize == 64)
{
assert(IsBaselineVector512IsaSupportedDebugOnly());
- intrinsic = NI_Vector512_LessThanAll;
- break;
+ intrinsic = NI_Vector512_op_Equality;
}
else
{
@@ -21471,14 +21496,12 @@ GenTree* Compiler::gtNewSimdCndSelNode(
return gtNewSimdTernaryLogicNode(type, op1, op2, op3, control, simdBaseJitType, simdSize);
}
+ assert(simdSize != 64);
+
if (simdSize == 32)
{
intrinsic = NI_Vector256_ConditionalSelect;
}
- else if (simdSize == 64)
- {
- intrinsic = NI_Vector512_ConditionalSelect;
- }
else
{
intrinsic = NI_Vector128_ConditionalSelect;
@@ -21908,6 +21931,7 @@ GenTree* Compiler::gtNewSimdDotProdNode(
#if defined(TARGET_XARCH)
assert(!varTypeIsByte(simdBaseType) && !varTypeIsLong(simdBaseType));
+ assert(simdSize != 64);
if (simdSize == 32)
{
@@ -23804,6 +23828,7 @@ GenTree* Compiler::gtNewSimdSumNode(var_types type, GenTree* op1, CorInfoType si
#if defined(TARGET_XARCH)
assert(!varTypeIsByte(simdBaseType) && !varTypeIsLong(simdBaseType));
+ assert(simdSize != 64);
// HorizontalAdd combines pairs so we need log2(vectorLength) passes to sum all elements together.
unsigned vectorLength = getSIMDVectorLength(simdSize, simdBaseType);
@@ -24771,6 +24796,25 @@ GenTreeHWIntrinsic* Compiler::gtNewScalarHWIntrinsicNode(
GenTreeHWIntrinsic(type, getAllocator(CMK_ASTNode), hwIntrinsicID, CORINFO_TYPE_UNDEF, 0, op1, op2, op3);
}
+//------------------------------------------------------------------------
+// OperIsHWIntrinsic: Is this a hwintrinsic with the specified id
+//
+// Arguments:
+// intrinsicId -- the id to compare with the current node
+//
+// Return Value:
+// true if the node is a hwintrinsic intrinsic with the specified id
+// otherwise; false
+//
+bool GenTree::OperIsHWIntrinsic(NamedIntrinsic intrinsicId) const
+{
+ if (OperIsHWIntrinsic())
+ {
+ return AsHWIntrinsic()->GetHWIntrinsicId() == intrinsicId;
+ }
+ return false;
+}
+
//------------------------------------------------------------------------
// OperIsMemoryLoad: Does this HWI node have memory load semantics?
//
@@ -24998,6 +25042,75 @@ bool GenTreeHWIntrinsic::OperIsMemoryStoreOrBarrier() const
}
}
+//------------------------------------------------------------------------
+// OperIsEmbBroadcastCompatible: Checks if the intrinsic is a embedded broadcast compatible inintrsic.
+//
+// Return Value:
+// true if the intrinsic node lowering instruction is embedded broadcast compatible.
+//
+bool GenTreeHWIntrinsic::OperIsEmbBroadcastCompatible() const
+{
+ return HWIntrinsicInfo::IsEmbBroadcastCompatible(GetHWIntrinsicId());
+}
+
+//------------------------------------------------------------------------
+// OperIsBroadcastScalar: Is this HWIntrinsic a broadcast node from scalar.
+//
+// Return Value:
+// Whether "this" is a broadcast node from scalar.
+//
+bool GenTreeHWIntrinsic::OperIsBroadcastScalar() const
+{
+#if defined(TARGET_XARCH)
+ NamedIntrinsic intrinsicId = GetHWIntrinsicId();
+ switch (intrinsicId)
+ {
+ case NI_AVX2_BroadcastScalarToVector128:
+ case NI_AVX2_BroadcastScalarToVector256:
+ case NI_AVX_BroadcastScalarToVector128:
+ case NI_AVX_BroadcastScalarToVector256:
+ case NI_SSE3_MoveAndDuplicate:
+ case NI_AVX512F_BroadcastScalarToVector512:
+ return true;
+ default:
+ return false;
+ }
+#else
+ return false;
+#endif
+}
+
+//------------------------------------------------------------------------
+// OperIsCreateScalarUnsafe: Is this HWIntrinsic a CreateScalarUnsafe node.
+//
+// Return Value:
+// Whether "this" is a CreateScalarUnsafe node.
+//
+bool GenTreeHWIntrinsic::OperIsCreateScalarUnsafe() const
+{
+ NamedIntrinsic intrinsicId = GetHWIntrinsicId();
+
+ switch (intrinsicId)
+ {
+#if defined(TARGET_ARM64)
+ case NI_Vector64_CreateScalarUnsafe:
+#endif // TARGET_ARM64
+ case NI_Vector128_CreateScalarUnsafe:
+#if defined(TARGET_XARCH)
+ case NI_Vector256_CreateScalarUnsafe:
+ case NI_Vector512_CreateScalarUnsafe:
+#endif // TARGET_XARCH
+ {
+ return true;
+ }
+
+ default:
+ {
+ return false;
+ }
+ }
+}
+
//------------------------------------------------------------------------------
// OperRequiresAsgFlag : Check whether the operation requires GTF_ASG flag regardless
// of the children's flags.
diff --git a/src/coreclr/jit/gentree.h b/src/coreclr/jit/gentree.h
index e5f3e11f426792..4483cd73812a05 100644
--- a/src/coreclr/jit/gentree.h
+++ b/src/coreclr/jit/gentree.h
@@ -554,6 +554,7 @@ enum GenTreeFlags : unsigned int
GTF_MDARRLEN_NONFAULTING = 0x20000000, // GT_MDARR_LENGTH -- An MD array length operation that cannot fault. Same as GT_IND_NONFAULTING.
GTF_MDARRLOWERBOUND_NONFAULTING = 0x20000000, // GT_MDARR_LOWER_BOUND -- An MD array lower bound operation that cannot fault. Same as GT_IND_NONFAULTING.
+
};
inline constexpr GenTreeFlags operator ~(GenTreeFlags a)
@@ -1086,7 +1087,7 @@ struct GenTree
return TypeIs(type) || TypeIs(rest...);
}
- static bool StaticOperIs(genTreeOps operCompare, genTreeOps oper)
+ static constexpr bool StaticOperIs(genTreeOps operCompare, genTreeOps oper)
{
return operCompare == oper;
}
@@ -1650,6 +1651,8 @@ struct GenTree
return OperIsHWIntrinsic(gtOper);
}
+ bool OperIsHWIntrinsic(NamedIntrinsic intrinsicId) const;
+
// This is here for cleaner GT_LONG #ifdefs.
static bool OperIsLong(genTreeOps gtOper)
{
@@ -6242,6 +6245,9 @@ struct GenTreeHWIntrinsic : public GenTreeJitIntrinsic
bool OperIsMemoryStore(GenTree** pAddr = nullptr) const;
bool OperIsMemoryLoadOrStore() const;
bool OperIsMemoryStoreOrBarrier() const;
+ bool OperIsEmbBroadcastCompatible() const;
+ bool OperIsBroadcastScalar() const;
+ bool OperIsCreateScalarUnsafe() const;
bool OperRequiresAsgFlag() const;
bool OperRequiresCallFlag() const;
diff --git a/src/coreclr/jit/gentreeopsdef.h b/src/coreclr/jit/gentreeopsdef.h
index 9e8e91508c0177..acffd4a12045de 100644
--- a/src/coreclr/jit/gentreeopsdef.h
+++ b/src/coreclr/jit/gentreeopsdef.h
@@ -8,7 +8,7 @@
enum genTreeOps : BYTE
{
-#define GTNODE(en, st, cm, ok) GT_##en,
+#define GTNODE(en, st, cm, ivn, ok) GT_##en,
#include "gtlist.h"
GT_COUNT,
diff --git a/src/coreclr/jit/gtlist.h b/src/coreclr/jit/gtlist.h
index 33925ae801a87f..b3a3d2dbb470c9 100644
--- a/src/coreclr/jit/gtlist.h
+++ b/src/coreclr/jit/gtlist.h
@@ -11,164 +11,165 @@
// Node enum
// , GenTree struct flavor
// ,commutative
-// ,oper kind | DEBUG oper kind
+// ,illegal as VN func
+// ,oper kind | DEBUG oper kind
-GTNODE(NONE , char ,0,GTK_SPECIAL)
+GTNODE(NONE , char ,0,0,GTK_SPECIAL)
//-----------------------------------------------------------------------------
// Nodes related to locals:
//-----------------------------------------------------------------------------
-GTNODE(PHI , GenTreePhi ,0,GTK_SPECIAL) // phi node for ssa.
-GTNODE(PHI_ARG , GenTreePhiArg ,0,GTK_LEAF) // phi(phiarg, phiarg, phiarg)
-GTNODE(LCL_VAR , GenTreeLclVar ,0,GTK_LEAF) // local variable
-GTNODE(LCL_FLD , GenTreeLclFld ,0,GTK_LEAF) // field in a non-primitive variable
-GTNODE(STORE_LCL_VAR , GenTreeLclVar ,0,GTK_UNOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // store to local variable
-GTNODE(STORE_LCL_FLD , GenTreeLclFld ,0,GTK_UNOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // store to a part of the variable
-GTNODE(LCL_ADDR , GenTreeLclFld ,0,GTK_LEAF) // local address
+GTNODE(PHI , GenTreePhi ,0,0,GTK_SPECIAL) // phi node for ssa.
+GTNODE(PHI_ARG , GenTreePhiArg ,0,0,GTK_LEAF) // phi(phiarg, phiarg, phiarg)
+GTNODE(LCL_VAR , GenTreeLclVar ,0,0,GTK_LEAF) // local variable
+GTNODE(LCL_FLD , GenTreeLclFld ,0,0,GTK_LEAF) // field in a non-primitive variable
+GTNODE(STORE_LCL_VAR , GenTreeLclVar ,0,1,GTK_UNOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // store to local variable
+GTNODE(STORE_LCL_FLD , GenTreeLclFld ,0,1,GTK_UNOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // store to a part of the variable
+GTNODE(LCL_ADDR , GenTreeLclFld ,0,0,GTK_LEAF) // local address
//-----------------------------------------------------------------------------
// Leaf nodes (i.e. these nodes have no sub-operands):
//-----------------------------------------------------------------------------
-GTNODE(CATCH_ARG , GenTree ,0,GTK_LEAF) // Exception object in a catch block
-GTNODE(LABEL , GenTree ,0,GTK_LEAF) // Jump-target
-GTNODE(JMP , GenTreeVal ,0,GTK_LEAF|GTK_NOVALUE) // Jump to another function
-GTNODE(FTN_ADDR , GenTreeFptrVal ,0,GTK_LEAF) // Address of a function
-GTNODE(RET_EXPR , GenTreeRetExpr ,0,GTK_LEAF|DBK_NOTLIR) // Place holder for the return expression from an inline candidate
+GTNODE(CATCH_ARG , GenTree ,0,0,GTK_LEAF) // Exception object in a catch block
+GTNODE(LABEL , GenTree ,0,0,GTK_LEAF) // Jump-target
+GTNODE(JMP , GenTreeVal ,0,0,GTK_LEAF|GTK_NOVALUE) // Jump to another function
+GTNODE(FTN_ADDR , GenTreeFptrVal ,0,0,GTK_LEAF) // Address of a function
+GTNODE(RET_EXPR , GenTreeRetExpr ,0,0,GTK_LEAF|DBK_NOTLIR) // Place holder for the return expression from an inline candidate
//-----------------------------------------------------------------------------
// Constant nodes:
//-----------------------------------------------------------------------------
-GTNODE(CNS_INT , GenTreeIntCon ,0,GTK_LEAF)
-GTNODE(CNS_LNG , GenTreeLngCon ,0,GTK_LEAF)
-GTNODE(CNS_DBL , GenTreeDblCon ,0,GTK_LEAF)
-GTNODE(CNS_STR , GenTreeStrCon ,0,GTK_LEAF)
-GTNODE(CNS_VEC , GenTreeVecCon ,0,GTK_LEAF)
+GTNODE(CNS_INT , GenTreeIntCon ,0,0,GTK_LEAF)
+GTNODE(CNS_LNG , GenTreeLngCon ,0,0,GTK_LEAF)
+GTNODE(CNS_DBL , GenTreeDblCon ,0,0,GTK_LEAF)
+GTNODE(CNS_STR , GenTreeStrCon ,0,0,GTK_LEAF)
+GTNODE(CNS_VEC , GenTreeVecCon ,0,0,GTK_LEAF)
//-----------------------------------------------------------------------------
// Unary operators (1 operand):
//-----------------------------------------------------------------------------
-GTNODE(NOT , GenTreeOp ,0,GTK_UNOP)
-GTNODE(NOP , GenTree ,0,GTK_UNOP|DBK_NOCONTAIN)
-GTNODE(NEG , GenTreeOp ,0,GTK_UNOP)
+GTNODE(NOT , GenTreeOp ,0,0,GTK_UNOP)
+GTNODE(NOP , GenTree ,0,0,GTK_UNOP|DBK_NOCONTAIN)
+GTNODE(NEG , GenTreeOp ,0,0,GTK_UNOP)
-GTNODE(INTRINSIC , GenTreeIntrinsic ,0,GTK_BINOP|GTK_EXOP)
+GTNODE(INTRINSIC , GenTreeIntrinsic ,0,0,GTK_BINOP|GTK_EXOP)
-GTNODE(LOCKADD , GenTreeOp ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
-GTNODE(XAND , GenTreeOp ,0,GTK_BINOP)
-GTNODE(XORR , GenTreeOp ,0,GTK_BINOP)
-GTNODE(XADD , GenTreeOp ,0,GTK_BINOP)
-GTNODE(XCHG , GenTreeOp ,0,GTK_BINOP)
-GTNODE(CMPXCHG , GenTreeCmpXchg ,0,GTK_SPECIAL)
-GTNODE(MEMORYBARRIER , GenTree ,0,GTK_LEAF|GTK_NOVALUE)
+GTNODE(LOCKADD , GenTreeOp ,0,1,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(XAND , GenTreeOp ,0,1,GTK_BINOP)
+GTNODE(XORR , GenTreeOp ,0,1,GTK_BINOP)
+GTNODE(XADD , GenTreeOp ,0,1,GTK_BINOP)
+GTNODE(XCHG , GenTreeOp ,0,1,GTK_BINOP)
+GTNODE(CMPXCHG , GenTreeCmpXchg ,0,1,GTK_SPECIAL)
+GTNODE(MEMORYBARRIER , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE)
-GTNODE(KEEPALIVE , GenTree ,0,GTK_UNOP|GTK_NOVALUE) // keep operand alive, generate no code, produce no result
+GTNODE(KEEPALIVE , GenTree ,0,0,GTK_UNOP|GTK_NOVALUE) // keep operand alive, generate no code, produce no result
-GTNODE(CAST , GenTreeCast ,0,GTK_UNOP|GTK_EXOP) // conversion to another type
+GTNODE(CAST , GenTreeCast ,0,0,GTK_UNOP|GTK_EXOP) // conversion to another type
#if defined(TARGET_ARM)
-GTNODE(BITCAST , GenTreeMultiRegOp ,0,GTK_UNOP) // reinterpretation of bits as another type
+GTNODE(BITCAST , GenTreeMultiRegOp ,0,1,GTK_UNOP) // reinterpretation of bits as another type
#else
-GTNODE(BITCAST , GenTreeOp ,0,GTK_UNOP) // reinterpretation of bits as another type
+GTNODE(BITCAST , GenTreeOp ,0,1,GTK_UNOP) // reinterpretation of bits as another type
#endif
-GTNODE(CKFINITE , GenTreeOp ,0,GTK_UNOP|DBK_NOCONTAIN) // Check for NaN
-GTNODE(LCLHEAP , GenTreeOp ,0,GTK_UNOP|DBK_NOCONTAIN) // alloca()
+GTNODE(CKFINITE , GenTreeOp ,0,1,GTK_UNOP|DBK_NOCONTAIN) // Check for NaN
+GTNODE(LCLHEAP , GenTreeOp ,0,1,GTK_UNOP|DBK_NOCONTAIN) // alloca()
-GTNODE(BOUNDS_CHECK , GenTreeBoundsChk ,0,GTK_BINOP|GTK_EXOP|GTK_NOVALUE) // a bounds check - for arrays/spans/SIMDs/HWINTRINSICs
+GTNODE(BOUNDS_CHECK , GenTreeBoundsChk ,0,1,GTK_BINOP|GTK_EXOP|GTK_NOVALUE) // a bounds check - for arrays/spans/SIMDs/HWINTRINSICs
-GTNODE(IND , GenTreeIndir ,0,GTK_UNOP) // Load indirection
-GTNODE(STOREIND , GenTreeStoreInd ,0,GTK_BINOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // Store indirection
-GTNODE(BLK , GenTreeBlk ,0,GTK_UNOP|GTK_EXOP) // Struct load
-GTNODE(STORE_BLK , GenTreeBlk ,0,GTK_BINOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // Struct store
-GTNODE(STORE_DYN_BLK , GenTreeStoreDynBlk ,0,GTK_SPECIAL|GTK_NOVALUE) // Dynamically sized block store, with native uint size
-GTNODE(NULLCHECK , GenTreeIndir ,0,GTK_UNOP|GTK_NOVALUE) // Null checks the source
+GTNODE(IND , GenTreeIndir ,0,1,GTK_UNOP) // Load indirection
+GTNODE(STOREIND , GenTreeStoreInd ,0,1,GTK_BINOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // Store indirection
+GTNODE(BLK , GenTreeBlk ,0,1,GTK_UNOP|GTK_EXOP) // Struct load
+GTNODE(STORE_BLK , GenTreeBlk ,0,1,GTK_BINOP|GTK_EXOP|GTK_NOVALUE|GTK_STORE) // Struct store
+GTNODE(STORE_DYN_BLK , GenTreeStoreDynBlk ,0,1,GTK_SPECIAL|GTK_NOVALUE) // Dynamically sized block store, with native uint size
+GTNODE(NULLCHECK , GenTreeIndir ,0,1,GTK_UNOP|GTK_NOVALUE) // Null checks the source
-GTNODE(ARR_LENGTH , GenTreeArrLen ,0,GTK_UNOP|GTK_EXOP) // single-dimension (SZ) array length
-GTNODE(MDARR_LENGTH , GenTreeMDArr ,0,GTK_UNOP|GTK_EXOP) // multi-dimension (MD) array length of a specific dimension
-GTNODE(MDARR_LOWER_BOUND, GenTreeMDArr ,0,GTK_UNOP|GTK_EXOP) // multi-dimension (MD) array lower bound of a specific dimension
-GTNODE(FIELD_ADDR , GenTreeFieldAddr ,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Field address
-GTNODE(ALLOCOBJ , GenTreeAllocObj ,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // object allocator
+GTNODE(ARR_LENGTH , GenTreeArrLen ,0,0,GTK_UNOP|GTK_EXOP) // single-dimension (SZ) array length
+GTNODE(MDARR_LENGTH , GenTreeMDArr ,0,1,GTK_UNOP|GTK_EXOP) // multi-dimension (MD) array length of a specific dimension
+GTNODE(MDARR_LOWER_BOUND, GenTreeMDArr ,0,1,GTK_UNOP|GTK_EXOP) // multi-dimension (MD) array lower bound of a specific dimension
+GTNODE(FIELD_ADDR , GenTreeFieldAddr ,0,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Field address
+GTNODE(ALLOCOBJ , GenTreeAllocObj ,0,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // object allocator
-GTNODE(INIT_VAL , GenTreeOp ,0,GTK_UNOP) // Initialization value for an initBlk
+GTNODE(INIT_VAL , GenTreeOp ,0,1,GTK_UNOP) // Initialization value for an initBlk
-GTNODE(BOX , GenTreeBox ,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Marks its first operands (a local) as being a box
-GTNODE(RUNTIMELOOKUP , GenTreeRuntimeLookup, 0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Runtime handle lookup
-GTNODE(ARR_ADDR , GenTreeArrAddr ,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Wraps an array address expression
+GTNODE(BOX , GenTreeBox ,0,1,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Marks its first operands (a local) as being a box
+GTNODE(RUNTIMELOOKUP , GenTreeRuntimeLookup, 0,0,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Runtime handle lookup
+GTNODE(ARR_ADDR , GenTreeArrAddr ,0,1,GTK_UNOP|GTK_EXOP|DBK_NOTLIR) // Wraps an array address expression
-GTNODE(BSWAP , GenTreeOp ,0,GTK_UNOP) // Byte swap (32-bit or 64-bit)
-GTNODE(BSWAP16 , GenTreeOp ,0,GTK_UNOP) // Byte swap lower 16-bits and zero upper 16 bits
+GTNODE(BSWAP , GenTreeOp ,0,0,GTK_UNOP) // Byte swap (32-bit or 64-bit)
+GTNODE(BSWAP16 , GenTreeOp ,0,0,GTK_UNOP) // Byte swap lower 16-bits and zero upper 16 bits
-GTNODE(LZCNT , GenTreeOp ,0,GTK_UNOP) // Leading Zero Count - Only used for SIMD VN evaluation today
+GTNODE(LZCNT , GenTreeOp ,0,0,GTK_UNOP) // Leading Zero Count - Only used for SIMD VN evaluation today
//-----------------------------------------------------------------------------
// Binary operators (2 operands):
//-----------------------------------------------------------------------------
-GTNODE(ADD , GenTreeOp ,1,GTK_BINOP)
-GTNODE(SUB , GenTreeOp ,0,GTK_BINOP)
-GTNODE(MUL , GenTreeOp ,1,GTK_BINOP)
-GTNODE(DIV , GenTreeOp ,0,GTK_BINOP)
-GTNODE(MOD , GenTreeOp ,0,GTK_BINOP)
+GTNODE(ADD , GenTreeOp ,1,0,GTK_BINOP)
+GTNODE(SUB , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(MUL , GenTreeOp ,1,0,GTK_BINOP)
+GTNODE(DIV , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(MOD , GenTreeOp ,0,0,GTK_BINOP)
-GTNODE(UDIV , GenTreeOp ,0,GTK_BINOP)
-GTNODE(UMOD , GenTreeOp ,0,GTK_BINOP)
+GTNODE(UDIV , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(UMOD , GenTreeOp ,0,0,GTK_BINOP)
-GTNODE(OR , GenTreeOp ,1,GTK_BINOP)
-GTNODE(XOR , GenTreeOp ,1,GTK_BINOP)
-GTNODE(AND , GenTreeOp ,1,GTK_BINOP)
+GTNODE(OR , GenTreeOp ,1,0,GTK_BINOP)
+GTNODE(XOR , GenTreeOp ,1,0,GTK_BINOP)
+GTNODE(AND , GenTreeOp ,1,0,GTK_BINOP)
-GTNODE(LSH , GenTreeOp ,0,GTK_BINOP)
-GTNODE(RSH , GenTreeOp ,0,GTK_BINOP)
-GTNODE(RSZ , GenTreeOp ,0,GTK_BINOP)
-GTNODE(ROL , GenTreeOp ,0,GTK_BINOP)
-GTNODE(ROR , GenTreeOp ,0,GTK_BINOP)
+GTNODE(LSH , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(RSH , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(RSZ , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(ROL , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(ROR , GenTreeOp ,0,0,GTK_BINOP)
-GTNODE(EQ , GenTreeOp ,0,GTK_BINOP)
-GTNODE(NE , GenTreeOp ,0,GTK_BINOP)
-GTNODE(LT , GenTreeOp ,0,GTK_BINOP)
-GTNODE(LE , GenTreeOp ,0,GTK_BINOP)
-GTNODE(GE , GenTreeOp ,0,GTK_BINOP)
-GTNODE(GT , GenTreeOp ,0,GTK_BINOP)
+GTNODE(EQ , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(NE , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(LT , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(LE , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(GE , GenTreeOp ,0,0,GTK_BINOP)
+GTNODE(GT , GenTreeOp ,0,0,GTK_BINOP)
// These implement EQ/NE(AND(x, y), 0). They always produce a value (GT_TEST is the version that does not).
-GTNODE(TEST_EQ , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
-GTNODE(TEST_NE , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(TEST_EQ , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(TEST_NE , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
#ifdef TARGET_XARCH
// BITTEST_EQ/NE(a, n) == EQ/NE(AND(a, LSH(1, n)), 0), but only used in xarch that has the BT instruction
-GTNODE(BITTEST_EQ , GenTreeOp ,0,(GTK_BINOP|DBK_NOTHIR))
-GTNODE(BITTEST_NE , GenTreeOp ,0,(GTK_BINOP|DBK_NOTHIR))
+GTNODE(BITTEST_EQ , GenTreeOp ,0,0,(GTK_BINOP|DBK_NOTHIR))
+GTNODE(BITTEST_NE , GenTreeOp ,0,0,(GTK_BINOP|DBK_NOTHIR))
#endif
// Conditional select with 3 operands: condition, true value, false value
-GTNODE(SELECT , GenTreeConditional ,0,GTK_SPECIAL)
+GTNODE(SELECT , GenTreeConditional ,0,0,GTK_SPECIAL)
-GTNODE(COMMA , GenTreeOp ,0,GTK_BINOP|DBK_NOTLIR)
-GTNODE(QMARK , GenTreeQmark ,0,GTK_BINOP|GTK_EXOP|DBK_NOTLIR)
-GTNODE(COLON , GenTreeColon ,0,GTK_BINOP|DBK_NOTLIR)
+GTNODE(COMMA , GenTreeOp ,0,1,GTK_BINOP|DBK_NOTLIR)
+GTNODE(QMARK , GenTreeQmark ,0,1,GTK_BINOP|GTK_EXOP|DBK_NOTLIR)
+GTNODE(COLON , GenTreeColon ,0,1,GTK_BINOP|DBK_NOTLIR)
-GTNODE(INDEX_ADDR , GenTreeIndexAddr ,0,GTK_BINOP|GTK_EXOP) // Address of SZ-array-element.
-GTNODE(MKREFANY , GenTreeOp ,0,GTK_BINOP|DBK_NOTLIR)
-GTNODE(LEA , GenTreeAddrMode ,0,GTK_BINOP|GTK_EXOP|DBK_NOTHIR)
+GTNODE(INDEX_ADDR , GenTreeIndexAddr ,0,0,GTK_BINOP|GTK_EXOP) // Address of SZ-array-element.
+GTNODE(MKREFANY , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTLIR)
+GTNODE(LEA , GenTreeAddrMode ,0,0,GTK_BINOP|GTK_EXOP|DBK_NOTHIR)
#if !defined(TARGET_64BIT)
// A GT_LONG node simply represents the long value produced by the concatenation
// of its two (lower and upper half) operands. Some GT_LONG nodes are transient,
// during the decomposing of longs; others are handled by codegen as operands of
// nodes such as calls, returns and stores of long lclVars.
-GTNODE(LONG , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(LONG , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
// The following are nodes representing x86/arm32 specific long operators, including
// high operators of a 64-bit operations that requires a carry/borrow, which are
// named GT_XXX_HI for consistency, low operators of 64-bit operations that need
// to not be modified in phases post-decompose, and operators that return 64-bit
// results in one instruction.
-GTNODE(ADD_LO , GenTreeOp ,1,GTK_BINOP|DBK_NOTHIR)
-GTNODE(ADD_HI , GenTreeOp ,1,GTK_BINOP|DBK_NOTHIR)
-GTNODE(SUB_LO , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
-GTNODE(SUB_HI , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(ADD_LO , GenTreeOp ,1,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(ADD_HI , GenTreeOp ,1,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SUB_LO , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SUB_HI , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
// The following are nodes that specify shifts that take a GT_LONG op1. The GT_LONG
// contains the hi and lo parts of three operand shift form where one op will be
@@ -177,13 +178,13 @@ GTNODE(SUB_HI , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
// will shift the lo bits of the high operand into the lo operand). LSH_HI
// represents the high operation of a 64-bit left shift by a constant int, and
// RSH_LO represents the lo operation of a 64-bit right shift by a constant int.
-GTNODE(LSH_HI , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
-GTNODE(RSH_LO , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(LSH_HI , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(RSH_LO , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
#endif // !defined(TARGET_64BIT)
#ifdef FEATURE_HW_INTRINSICS
-GTNODE(HWINTRINSIC , GenTreeHWIntrinsic ,0,GTK_SPECIAL) // hardware intrinsics
+GTNODE(HWINTRINSIC , GenTreeHWIntrinsic ,0,0,GTK_SPECIAL) // hardware intrinsics
#endif // FEATURE_HW_INTRINSICS
//-----------------------------------------------------------------------------
@@ -191,13 +192,13 @@ GTNODE(HWINTRINSIC , GenTreeHWIntrinsic ,0,GTK_SPECIAL) // ha
//-----------------------------------------------------------------------------
// Saturating increment, used in division by a constant (LowerUnsignedDivOrMod).
-GTNODE(INC_SATURATE , GenTreeOp ,0,GTK_UNOP|DBK_NOTHIR)
+GTNODE(INC_SATURATE , GenTreeOp ,0,0,GTK_UNOP|DBK_NOTHIR)
// Returns high bits (top N bits of the 2N bit result of an NxN multiply)
// GT_MULHI is used in division by a constant (LowerUnsignedDivOrMod). We turn
// the div into a MULHI + some adjustments. In codegen, we only use the
// results of the high register, and we drop the low results.
-GTNODE(MULHI , GenTreeOp ,1,GTK_BINOP|DBK_NOTHIR)
+GTNODE(MULHI , GenTreeOp ,1,0,GTK_BINOP|DBK_NOTHIR)
// A mul that returns the 2N bit result of an NxN multiply. This op is used for
// multiplies that take two ints and return a long result. For 32 bit targets,
@@ -206,15 +207,15 @@ GTNODE(MULHI , GenTreeOp ,1,GTK_BINOP|DBK_NOTHIR)
// part of the result, whereas GT_MUL_LONG keeps both parts of the result.
// MUL_LONG is also used on ARM64, where 64 bit multiplication is more expensive.
#if !defined(TARGET_64BIT)
-GTNODE(MUL_LONG , GenTreeMultiRegOp ,1,GTK_BINOP|DBK_NOTHIR)
+GTNODE(MUL_LONG , GenTreeMultiRegOp ,1,0,GTK_BINOP|DBK_NOTHIR)
#elif defined(TARGET_ARM64)
-GTNODE(MUL_LONG , GenTreeOp ,1,GTK_BINOP|DBK_NOTHIR)
+GTNODE(MUL_LONG , GenTreeOp ,1,0,GTK_BINOP|DBK_NOTHIR)
#endif
// AndNot - emitted on ARM/ARM64 as the BIC instruction. Also used for creating AndNot HWINTRINSIC vector nodes in a cross-ISA manner.
-GTNODE(AND_NOT , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(AND_NOT , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
#ifdef TARGET_ARM64
-GTNODE(BFIZ , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR) // Bitfield Insert in Zero.
+GTNODE(BFIZ , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR) // Bitfield Insert in Zero.
#endif
//-----------------------------------------------------------------------------
@@ -222,102 +223,102 @@ GTNODE(BFIZ , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR) // Bitfiel
//-----------------------------------------------------------------------------
// Sets the condition flags according to the compare result. N.B. Not a relop, it does not produce a value and it cannot be reversed.
-GTNODE(CMP , GenTreeOp ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(CMP , GenTreeOp ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
// Generate a test instruction; sets the CPU flags according to (a & b) and does not produce a value.
-GTNODE(TEST , GenTreeOp ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(TEST , GenTreeOp ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
#ifdef TARGET_XARCH
// The XARCH BT instruction. Like CMP, this sets the condition flags (CF to be precise) and does not produce a value.
-GTNODE(BT , GenTreeOp ,0,(GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR))
+GTNODE(BT , GenTreeOp ,0,0,(GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR))
#endif
// Makes a comparison and jumps if the condition specified by gtCondition is true. Does not set flags.
-GTNODE(JCMP , GenTreeOpCC ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(JCMP , GenTreeOpCC ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
// Do a bit test and jump if set/not set.
-GTNODE(JTEST , GenTreeOpCC ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(JTEST , GenTreeOpCC ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
// Checks the condition flags and branch if the condition specified by GenTreeCC::gtCondition is true.
-GTNODE(JCC , GenTreeCC ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(JCC , GenTreeCC ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR)
// Checks the condition flags and produces 1 if the condition specified by GenTreeCC::gtCondition is true and 0 otherwise.
-GTNODE(SETCC , GenTreeCC ,0,GTK_LEAF|DBK_NOTHIR)
+GTNODE(SETCC , GenTreeCC ,0,0,GTK_LEAF|DBK_NOTHIR)
// Variant of SELECT that reuses flags computed by a previous node with the specified condition.
-GTNODE(SELECTCC , GenTreeOpCC ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECTCC , GenTreeOpCC ,0,0,GTK_BINOP|DBK_NOTHIR)
#ifdef TARGET_ARM64
// The arm64 ccmp instruction. If the specified condition is true, compares two
// operands and sets the condition flags according to the result. Otherwise
// sets the condition flags to the specified immediate value.
-GTNODE(CCMP , GenTreeCCMP ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
+GTNODE(CCMP , GenTreeCCMP ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR)
// Maps to arm64 csinc/cinc instruction. Computes result = condition ? op1 : op2 + 1.
// If op2 is null, computes result = condition ? op1 + 1 : op1.
-GTNODE(SELECT_INC , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_INC , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
// Variant of SELECT_INC that reuses flags computed by a previous node with the specified condition.
-GTNODE(SELECT_INCCC , GenTreeOpCC ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_INCCC , GenTreeOpCC ,0,0,GTK_BINOP|DBK_NOTHIR)
// Maps to arm64 csinv/cinv instruction. Computes result = condition ? op1 : ~op2.
// If op2 is null, computes result = condition ? ~op1 : op1.
-GTNODE(SELECT_INV , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_INV , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
// Variant of SELECT_INV that reuses flags computed by a previous node with the specified condition.
-GTNODE(SELECT_INVCC , GenTreeOpCC ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_INVCC , GenTreeOpCC ,0,0,GTK_BINOP|DBK_NOTHIR)
// Maps to arm64 csneg/cneg instruction.. Computes result = condition ? op1 : -op2.
// If op2 is null, computes result = condition ? -op1 : op1.
-GTNODE(SELECT_NEG , GenTreeOp ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_NEG , GenTreeOp ,0,0,GTK_BINOP|DBK_NOTHIR)
// Variant of SELECT_NEG that reuses flags computed by a previous node with the specified condition.
-GTNODE(SELECT_NEGCC , GenTreeOpCC ,0,GTK_BINOP|DBK_NOTHIR)
+GTNODE(SELECT_NEGCC , GenTreeOpCC ,0,0,GTK_BINOP|DBK_NOTHIR)
#endif
//-----------------------------------------------------------------------------
// Other nodes that look like unary/binary operators:
//-----------------------------------------------------------------------------
-GTNODE(JTRUE , GenTreeOp ,0,GTK_UNOP|GTK_NOVALUE)
+GTNODE(JTRUE , GenTreeOp ,0,1,GTK_UNOP|GTK_NOVALUE)
//-----------------------------------------------------------------------------
// Other nodes that have special structure:
//-----------------------------------------------------------------------------
-GTNODE(ARR_ELEM , GenTreeArrElem ,0,GTK_SPECIAL) // Multi-dimensional array-element address
-GTNODE(CALL , GenTreeCall ,0,GTK_SPECIAL|DBK_NOCONTAIN)
-GTNODE(FIELD_LIST , GenTreeFieldList ,0,GTK_SPECIAL) // List of fields of a struct, when passed as an argument
+GTNODE(ARR_ELEM , GenTreeArrElem ,0,0,GTK_SPECIAL) // Multi-dimensional array-element address
+GTNODE(CALL , GenTreeCall ,0,0,GTK_SPECIAL|DBK_NOCONTAIN)
+GTNODE(FIELD_LIST , GenTreeFieldList ,0,0,GTK_SPECIAL) // List of fields of a struct, when passed as an argument
-GTNODE(RETURN , GenTreeOp ,0,GTK_UNOP|GTK_NOVALUE)
-GTNODE(SWITCH , GenTreeOp ,0,GTK_UNOP|GTK_NOVALUE)
-GTNODE(NO_OP , GenTree ,0,GTK_LEAF|GTK_NOVALUE) // A NOP that cannot be deleted.
+GTNODE(RETURN , GenTreeOp ,0,1,GTK_UNOP|GTK_NOVALUE)
+GTNODE(SWITCH , GenTreeOp ,0,1,GTK_UNOP|GTK_NOVALUE)
+GTNODE(NO_OP , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE) // A NOP that cannot be deleted.
-GTNODE(START_NONGC , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Starts a new instruction group that will be non-gc interruptible.
-GTNODE(START_PREEMPTGC , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Starts a new instruction group where preemptive GC is enabled.
-GTNODE(PROF_HOOK , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Profiler Enter/Leave/TailCall hook.
+GTNODE(START_NONGC , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Starts a new instruction group that will be non-gc interruptible.
+GTNODE(START_PREEMPTGC , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Starts a new instruction group where preemptive GC is enabled.
+GTNODE(PROF_HOOK , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // Profiler Enter/Leave/TailCall hook.
-GTNODE(RETFILT , GenTreeOp ,0,GTK_UNOP|GTK_NOVALUE) // End filter with TYP_I_IMPL return value.
+GTNODE(RETFILT , GenTreeOp ,0,1,GTK_UNOP|GTK_NOVALUE) // End filter with TYP_I_IMPL return value.
#if !defined(FEATURE_EH_FUNCLETS)
-GTNODE(END_LFIN , GenTreeVal ,0,GTK_LEAF|GTK_NOVALUE) // End locally-invoked finally.
+GTNODE(END_LFIN , GenTreeVal ,0,0,GTK_LEAF|GTK_NOVALUE) // End locally-invoked finally.
#endif // !FEATURE_EH_FUNCLETS
//-----------------------------------------------------------------------------
// Nodes used by Lower to generate a closer CPU representation of other nodes
//-----------------------------------------------------------------------------
-GTNODE(JMPTABLE , GenTree ,0,GTK_LEAF|DBK_NOCONTAIN|DBK_NOTHIR) // Generates the jump table for switches
-GTNODE(SWITCH_TABLE , GenTreeOp ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR) // Jump Table based switch construct
+GTNODE(JMPTABLE , GenTree ,0,0,GTK_LEAF|DBK_NOCONTAIN|DBK_NOTHIR) // Generates the jump table for switches
+GTNODE(SWITCH_TABLE , GenTreeOp ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR) // Jump Table based switch construct
//-----------------------------------------------------------------------------
// Nodes used only within the code generator:
//-----------------------------------------------------------------------------
-GTNODE(CLS_VAR_ADDR , GenTreeClsVar ,0,GTK_LEAF|DBK_NOTHIR) // static data member address
-GTNODE(PHYSREG , GenTreePhysReg ,0,GTK_LEAF|DBK_NOTHIR) // read from a physical register
-GTNODE(EMITNOP , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // emitter-placed nop
-GTNODE(PINVOKE_PROLOG , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // pinvoke prolog seq
-GTNODE(PINVOKE_EPILOG , GenTree ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // pinvoke epilog seq
-GTNODE(RETURNTRAP , GenTreeOp ,0,GTK_UNOP|GTK_NOVALUE|DBK_NOTHIR) // a conditional call to wait on gc
+GTNODE(CLS_VAR_ADDR , GenTreeClsVar ,0,0,GTK_LEAF|DBK_NOTHIR) // static data member address
+GTNODE(PHYSREG , GenTreePhysReg ,0,0,GTK_LEAF|DBK_NOTHIR) // read from a physical register
+GTNODE(EMITNOP , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // emitter-placed nop
+GTNODE(PINVOKE_PROLOG , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // pinvoke prolog seq
+GTNODE(PINVOKE_EPILOG , GenTree ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // pinvoke epilog seq
+GTNODE(RETURNTRAP , GenTreeOp ,0,0,GTK_UNOP|GTK_NOVALUE|DBK_NOTHIR) // a conditional call to wait on gc
#if defined(TARGET_ARM)
-GTNODE(PUTARG_REG , GenTreeMultiRegOp ,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in register
+GTNODE(PUTARG_REG , GenTreeMultiRegOp ,0,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in register
#else
-GTNODE(PUTARG_REG , GenTreeOp ,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in register
+GTNODE(PUTARG_REG , GenTreeOp ,0,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in register
#endif
-GTNODE(PUTARG_STK , GenTreePutArgStk ,0,GTK_UNOP|GTK_NOVALUE|DBK_NOTHIR) // operator that places outgoing arg in stack
+GTNODE(PUTARG_STK , GenTreePutArgStk ,0,0,GTK_UNOP|GTK_NOVALUE|DBK_NOTHIR) // operator that places outgoing arg in stack
#if FEATURE_ARG_SPLIT
-GTNODE(PUTARG_SPLIT , GenTreePutArgSplit ,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in registers with stack (split struct in ARM32)
+GTNODE(PUTARG_SPLIT , GenTreePutArgSplit ,0,0,GTK_UNOP|DBK_NOTHIR) // operator that places outgoing arg in registers with stack (split struct in ARM32)
#endif // FEATURE_ARG_SPLIT
-GTNODE(SWAP , GenTreeOp ,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR) // op1 and op2 swap (registers)
-GTNODE(COPY , GenTreeCopyOrReload,0,GTK_UNOP|DBK_NOTHIR) // Copies a variable from its current location to a register that satisfies
-GTNODE(RELOAD , GenTreeCopyOrReload,0,GTK_UNOP|DBK_NOTHIR) // code generation constraints. The operand is the actual lclVar node.
-GTNODE(IL_OFFSET , GenTreeILOffset ,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // marks an IL offset for debugging purposes
+GTNODE(SWAP , GenTreeOp ,0,0,GTK_BINOP|GTK_NOVALUE|DBK_NOTHIR) // op1 and op2 swap (registers)
+GTNODE(COPY , GenTreeCopyOrReload,0,0,GTK_UNOP|DBK_NOTHIR) // Copies a variable from its current location to a register that satisfies
+GTNODE(RELOAD , GenTreeCopyOrReload,0,0,GTK_UNOP|DBK_NOTHIR) // code generation constraints. The operand is the actual lclVar node.
+GTNODE(IL_OFFSET , GenTreeILOffset ,0,0,GTK_LEAF|GTK_NOVALUE|DBK_NOTHIR) // marks an IL offset for debugging purposes
/*****************************************************************************/
#undef GTNODE
diff --git a/src/coreclr/jit/hwintrinsic.cpp b/src/coreclr/jit/hwintrinsic.cpp
index 35d1b6aa249bd5..0b3c1ec052bd00 100644
--- a/src/coreclr/jit/hwintrinsic.cpp
+++ b/src/coreclr/jit/hwintrinsic.cpp
@@ -9,7 +9,7 @@
static const HWIntrinsicInfo hwIntrinsicInfoArray[] = {
// clang-format off
#if defined(TARGET_XARCH)
-#define HARDWARE_INTRINSIC(isa, name, size, numarg, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+#define HARDWARE_INTRINSIC(isa, name, size, numarg, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
{ \
/* name */ #name, \
/* flags */ static_cast(flag), \
@@ -22,7 +22,7 @@ static const HWIntrinsicInfo hwIntrinsicInfoArray[] = {
},
#include "hwintrinsiclistxarch.h"
#elif defined (TARGET_ARM64)
-#define HARDWARE_INTRINSIC(isa, name, size, numarg, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+#define HARDWARE_INTRINSIC(isa, name, size, numarg, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
{ \
/* name */ #name, \
/* flags */ static_cast(flag), \
@@ -495,9 +495,9 @@ NamedIntrinsic HWIntrinsicInfo::lookupId(Compiler* comp,
return NI_Illegal;
}
- bool isIsaSupported = comp->compSupportsHWIntrinsic(isa);
-
+ bool isIsaSupported = comp->compSupportsHWIntrinsic(isa);
bool isHardwareAcceleratedProp = (strcmp(methodName, "get_IsHardwareAccelerated") == 0);
+
#ifdef TARGET_XARCH
if (isHardwareAcceleratedProp)
{
@@ -541,22 +541,22 @@ NamedIntrinsic HWIntrinsicInfo::lookupId(Compiler* comp,
//
// When the target hardware does support the instruction set, we can return a
// constant true. When it doesn't then we want to report the check as dynamically
- // supported instead. This allows some targets, such as AOT, to emit a check against
- // a cached CPU query so lightup can still happen (such as for SSE4.1 when the target
- // hardware is SSE2).
+ // supported instead if the opportunistic support does exist. This allows some targets,
+ // such as AOT, to emit a check against a cached CPU query so lightup can still happen
+ // (such as for SSE4.1 when the target hardware is SSE2).
//
// When the compiler doesn't support ISA or when it does but the target hardware does
// not and we aren't in a scenario with support for a dynamic check, we want to return false.
- if (isIsaSupported)
+ if (isIsaSupported && comp->compSupportsHWIntrinsic(isa))
{
- if (comp->compExactlyDependsOn(isa))
+ if (!comp->IsTargetAbi(CORINFO_NATIVEAOT_ABI) || comp->compExactlyDependsOn(isa))
{
return NI_IsSupported_True;
}
-
- if (comp->IsTargetAbi(CORINFO_NATIVEAOT_ABI))
+ else
{
+ assert(comp->IsTargetAbi(CORINFO_NATIVEAOT_ABI));
return NI_IsSupported_Dynamic;
}
}
diff --git a/src/coreclr/jit/hwintrinsic.h b/src/coreclr/jit/hwintrinsic.h
index 26599ba5b1de25..01b886744ea4f7 100644
--- a/src/coreclr/jit/hwintrinsic.h
+++ b/src/coreclr/jit/hwintrinsic.h
@@ -201,6 +201,8 @@ enum HWIntrinsicFlag : unsigned int
// The intrinsic is a PermuteVar2x intrinsic
HW_Flag_PermuteVar2x = 0x4000000,
#endif // TARGET_XARCH
+ // The intrinsic is an embedded broadcast compatiable intrinsic
+ HW_Flag_EmbBroadcastCompatible = 0x8000000,
};
#if defined(TARGET_XARCH)
@@ -328,6 +330,25 @@ enum class FloatRoundingMode : uint8_t
NoException = 0x08,
};
+enum class IntComparisonMode : uint8_t
+{
+ Equal = 0,
+ LessThan = 1,
+ LessThanOrEqual = 2,
+ False = 3,
+
+ NotEqual = 4,
+ GreaterThanOrEqual = 5,
+ GreaterThan = 6,
+ True = 7,
+
+ NotGreaterThanOrEqual = LessThan,
+ NotGreaterThan = LessThanOrEqual,
+
+ NotLessThan = GreaterThanOrEqual,
+ NotLessThanOrEqual = GreaterThan
+};
+
enum class TernaryLogicUseFlags : uint8_t
{
// Indicates no flags are present
@@ -491,217 +512,7 @@ struct HWIntrinsicInfo
}
#ifdef TARGET_XARCH
- static int lookupIval(NamedIntrinsic id, bool opportunisticallyDependsOnAVX)
- {
- switch (id)
- {
- case NI_SSE_CompareEqual:
- case NI_SSE_CompareScalarEqual:
- case NI_SSE2_CompareEqual:
- case NI_SSE2_CompareScalarEqual:
- case NI_AVX_CompareEqual:
- {
- return static_cast(FloatComparisonMode::OrderedEqualNonSignaling);
- }
-
- case NI_SSE_CompareGreaterThan:
- case NI_SSE_CompareScalarGreaterThan:
- case NI_SSE2_CompareGreaterThan:
- case NI_SSE2_CompareScalarGreaterThan:
- case NI_AVX_CompareGreaterThan:
- {
- if (opportunisticallyDependsOnAVX)
- {
- return static_cast(FloatComparisonMode::OrderedGreaterThanSignaling);
- }
-
- // CompareGreaterThan is not directly supported in hardware without AVX support.
- // We will return the inverted case here and lowering will itself swap the ops
- // to ensure the emitted code remains correct. This simplifies the overall logic
- // here and for other use cases.
-
- assert(id != NI_AVX_CompareGreaterThan);
- return static_cast(FloatComparisonMode::OrderedLessThanSignaling);
- }
-
- case NI_SSE_CompareLessThan:
- case NI_SSE_CompareScalarLessThan:
- case NI_SSE2_CompareLessThan:
- case NI_SSE2_CompareScalarLessThan:
- case NI_AVX_CompareLessThan:
- {
- return static_cast(FloatComparisonMode::OrderedLessThanSignaling);
- }
-
- case NI_SSE_CompareGreaterThanOrEqual:
- case NI_SSE_CompareScalarGreaterThanOrEqual:
- case NI_SSE2_CompareGreaterThanOrEqual:
- case NI_SSE2_CompareScalarGreaterThanOrEqual:
- case NI_AVX_CompareGreaterThanOrEqual:
- {
- if (opportunisticallyDependsOnAVX)
- {
- return static_cast(FloatComparisonMode::OrderedGreaterThanOrEqualSignaling);
- }
-
- // CompareGreaterThanOrEqual is not directly supported in hardware without AVX support.
- // We will return the inverted case here and lowering will itself swap the ops
- // to ensure the emitted code remains correct. This simplifies the overall logic
- // here and for other use cases.
-
- assert(id != NI_AVX_CompareGreaterThanOrEqual);
- return static_cast(FloatComparisonMode::OrderedLessThanOrEqualSignaling);
- }
-
- case NI_SSE_CompareLessThanOrEqual:
- case NI_SSE_CompareScalarLessThanOrEqual:
- case NI_SSE2_CompareLessThanOrEqual:
- case NI_SSE2_CompareScalarLessThanOrEqual:
- case NI_AVX_CompareLessThanOrEqual:
- {
- return static_cast(FloatComparisonMode::OrderedLessThanOrEqualSignaling);
- }
-
- case NI_SSE_CompareNotEqual:
- case NI_SSE_CompareScalarNotEqual:
- case NI_SSE2_CompareNotEqual:
- case NI_SSE2_CompareScalarNotEqual:
- case NI_AVX_CompareNotEqual:
- {
- return static_cast(FloatComparisonMode::UnorderedNotEqualNonSignaling);
- }
-
- case NI_SSE_CompareNotGreaterThan:
- case NI_SSE_CompareScalarNotGreaterThan:
- case NI_SSE2_CompareNotGreaterThan:
- case NI_SSE2_CompareScalarNotGreaterThan:
- case NI_AVX_CompareNotGreaterThan:
- {
- if (opportunisticallyDependsOnAVX)
- {
- return static_cast(FloatComparisonMode::UnorderedNotGreaterThanSignaling);
- }
-
- // CompareNotGreaterThan is not directly supported in hardware without AVX support.
- // We will return the inverted case here and lowering will itself swap the ops
- // to ensure the emitted code remains correct. This simplifies the overall logic
- // here and for other use cases.
-
- assert(id != NI_AVX_CompareNotGreaterThan);
- return static_cast(FloatComparisonMode::UnorderedNotLessThanSignaling);
- }
-
- case NI_SSE_CompareNotLessThan:
- case NI_SSE_CompareScalarNotLessThan:
- case NI_SSE2_CompareNotLessThan:
- case NI_SSE2_CompareScalarNotLessThan:
- case NI_AVX_CompareNotLessThan:
- {
- return static_cast(FloatComparisonMode::UnorderedNotLessThanSignaling);
- }
-
- case NI_SSE_CompareNotGreaterThanOrEqual:
- case NI_SSE_CompareScalarNotGreaterThanOrEqual:
- case NI_SSE2_CompareNotGreaterThanOrEqual:
- case NI_SSE2_CompareScalarNotGreaterThanOrEqual:
- case NI_AVX_CompareNotGreaterThanOrEqual:
- {
- if (opportunisticallyDependsOnAVX)
- {
- return static_cast(FloatComparisonMode::UnorderedNotGreaterThanOrEqualSignaling);
- }
-
- // CompareNotGreaterThanOrEqual is not directly supported in hardware without AVX support.
- // We will return the inverted case here and lowering will itself swap the ops
- // to ensure the emitted code remains correct. This simplifies the overall logic
- // here and for other use cases.
-
- assert(id != NI_AVX_CompareNotGreaterThanOrEqual);
- return static_cast(FloatComparisonMode::UnorderedNotLessThanOrEqualSignaling);
- }
-
- case NI_SSE_CompareNotLessThanOrEqual:
- case NI_SSE_CompareScalarNotLessThanOrEqual:
- case NI_SSE2_CompareNotLessThanOrEqual:
- case NI_SSE2_CompareScalarNotLessThanOrEqual:
- case NI_AVX_CompareNotLessThanOrEqual:
- {
- return static_cast(FloatComparisonMode::UnorderedNotLessThanOrEqualSignaling);
- }
-
- case NI_SSE_CompareOrdered:
- case NI_SSE_CompareScalarOrdered:
- case NI_SSE2_CompareOrdered:
- case NI_SSE2_CompareScalarOrdered:
- case NI_AVX_CompareOrdered:
- {
- return static_cast(FloatComparisonMode::OrderedNonSignaling);
- }
-
- case NI_SSE_CompareUnordered:
- case NI_SSE_CompareScalarUnordered:
- case NI_SSE2_CompareUnordered:
- case NI_SSE2_CompareScalarUnordered:
- case NI_AVX_CompareUnordered:
- {
- return static_cast(FloatComparisonMode::UnorderedNonSignaling);
- }
-
- case NI_SSE41_Ceiling:
- case NI_SSE41_CeilingScalar:
- case NI_AVX_Ceiling:
- {
- FALLTHROUGH;
- }
-
- case NI_SSE41_RoundToPositiveInfinity:
- case NI_SSE41_RoundToPositiveInfinityScalar:
- case NI_AVX_RoundToPositiveInfinity:
- {
- return static_cast(FloatRoundingMode::ToPositiveInfinity);
- }
-
- case NI_SSE41_Floor:
- case NI_SSE41_FloorScalar:
- case NI_AVX_Floor:
- {
- FALLTHROUGH;
- }
-
- case NI_SSE41_RoundToNegativeInfinity:
- case NI_SSE41_RoundToNegativeInfinityScalar:
- case NI_AVX_RoundToNegativeInfinity:
- {
- return static_cast(FloatRoundingMode::ToNegativeInfinity);
- }
-
- case NI_SSE41_RoundCurrentDirection:
- case NI_SSE41_RoundCurrentDirectionScalar:
- case NI_AVX_RoundCurrentDirection:
- {
- return static_cast(FloatRoundingMode::CurrentDirection);
- }
-
- case NI_SSE41_RoundToNearestInteger:
- case NI_SSE41_RoundToNearestIntegerScalar:
- case NI_AVX_RoundToNearestInteger:
- {
- return static_cast(FloatRoundingMode::ToNearestInteger);
- }
-
- case NI_SSE41_RoundToZero:
- case NI_SSE41_RoundToZeroScalar:
- case NI_AVX_RoundToZero:
- {
- return static_cast(FloatRoundingMode::ToZero);
- }
-
- default:
- {
- return -1;
- }
- }
- }
+ static int lookupIval(Compiler* comp, NamedIntrinsic id, var_types simdBaseType);
#endif
static bool tryLookupSimdSize(NamedIntrinsic id, unsigned* pSimdSize)
@@ -769,6 +580,12 @@ struct HWIntrinsicInfo
return (flags & HW_Flag_Commutative) != 0;
}
+ static bool IsEmbBroadcastCompatible(NamedIntrinsic id)
+ {
+ HWIntrinsicFlag flags = lookupFlags(id);
+ return (flags & HW_Flag_EmbBroadcastCompatible) != 0;
+ }
+
static bool IsMaybeCommutative(NamedIntrinsic id)
{
HWIntrinsicFlag flags = lookupFlags(id);
diff --git a/src/coreclr/jit/hwintrinsiccodegenxarch.cpp b/src/coreclr/jit/hwintrinsiccodegenxarch.cpp
index 973d4176a00b63..7570765fcdadd9 100644
--- a/src/coreclr/jit/hwintrinsiccodegenxarch.cpp
+++ b/src/coreclr/jit/hwintrinsiccodegenxarch.cpp
@@ -97,9 +97,6 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
// We need to validate that other phases of the compiler haven't introduced unsupported intrinsics
assert(compiler->compIsaSupportedDebugOnly(isa));
-
- int ival = HWIntrinsicInfo::lookupIval(intrinsicId, compiler->compOpportunisticallyDependsOn(InstructionSet_AVX));
-
assert(HWIntrinsicInfo::RequiresCodegen(intrinsicId));
if (genIsTableDrivenHWIntrinsic(intrinsicId, category))
@@ -119,12 +116,15 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
emitter* emit = GetEmitter();
assert(numArgs >= 0);
+
instruction ins = HWIntrinsicInfo::lookupIns(intrinsicId, baseType);
assert(ins != INS_invalid);
- emitAttr simdSize = emitActualTypeSize(Compiler::getSIMDTypeForSize(node->GetSimdSize()));
+ emitAttr simdSize = emitActualTypeSize(Compiler::getSIMDTypeForSize(node->GetSimdSize()));
assert(simdSize != 0);
+ int ival = HWIntrinsicInfo::lookupIval(compiler, intrinsicId, baseType);
+
switch (numArgs)
{
case 1:
@@ -144,7 +144,7 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
genConsumeRegs(op1);
op1Reg = op1->GetRegNum();
- if ((ival != -1) && varTypeIsFloating(baseType))
+ if (ival != -1)
{
assert((ival >= 0) && (ival <= 127));
if (HWIntrinsicInfo::CopiesUpperBits(intrinsicId))
@@ -208,7 +208,7 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
op1Reg = targetReg;
}
- if ((ival != -1) && varTypeIsFloating(baseType))
+ if (ival != -1)
{
assert((ival >= 0) && (ival <= 127));
genHWIntrinsic_R_R_RM_I(node, ins, simdSize, static_cast(ival));
@@ -235,7 +235,6 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
}
else if (HWIntrinsicInfo::isImmOp(intrinsicId, op2))
{
- assert(ival == -1);
auto emitSwCase = [&](int8_t i) {
if (HWIntrinsicInfo::CopiesUpperBits(intrinsicId))
{
@@ -293,10 +292,10 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
genConsumeRegs(op3);
op3Reg = op3->GetRegNum();
+ assert(ival == -1);
+
if (HWIntrinsicInfo::isImmOp(intrinsicId, op3))
{
- assert(ival == -1);
-
auto emitSwCase = [&](int8_t i) { genHWIntrinsic_R_R_RM_I(node, ins, simdSize, i); };
if (op3->IsCnsIntOrI())
@@ -384,10 +383,10 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
genConsumeRegs(op4);
op4Reg = op4->GetRegNum();
+ assert(ival == -1);
+
if (HWIntrinsicInfo::isImmOp(intrinsicId, op4))
{
- assert(ival == -1);
-
auto emitSwCase = [&](int8_t i) { genHWIntrinsic_R_R_R_RM_I(node, ins, simdSize, i); };
if (op4->IsCnsIntOrI())
@@ -455,6 +454,8 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
case InstructionSet_AVX512F_X64:
case InstructionSet_AVX512BW:
case InstructionSet_AVX512BW_VL:
+ case InstructionSet_AVX512VBMI:
+ case InstructionSet_AVX512VBMI_VL:
genAvxFamilyIntrinsic(node);
break;
case InstructionSet_AES:
@@ -1913,6 +1914,7 @@ void CodeGen::genAvxFamilyIntrinsic(GenTreeHWIntrinsic* node)
instruction maskIns;
instruction kmovIns;
+ emitAttr kmovAttr = EA_4BYTE;
// TODO-XARCH-AVX512 note that this type/kmov combination assumes 512-bit vector types but would change
// if used for other vector lengths, i.e., TYPE_BYTE requires kmovq for for 512-bit vector, but kmovd
@@ -1921,130 +1923,78 @@ void CodeGen::genAvxFamilyIntrinsic(GenTreeHWIntrinsic* node)
{
case TYP_BYTE:
case TYP_UBYTE:
- maskIns = INS_vpmovb2m;
- kmovIns = INS_kmovq_gpr;
+ {
+ maskIns = INS_vpmovb2m;
+ kmovIns = INS_kmovq_gpr;
+ kmovAttr = EA_8BYTE;
break;
+ }
+
case TYP_SHORT:
case TYP_USHORT:
+ {
maskIns = INS_vpmovw2m;
kmovIns = INS_kmovd_gpr;
break;
+ }
+
case TYP_INT:
case TYP_UINT:
case TYP_FLOAT:
+ {
maskIns = INS_vpmovd2m;
kmovIns = INS_kmovw_gpr;
break;
+ }
+
case TYP_DOUBLE:
case TYP_LONG:
case TYP_ULONG:
+ {
maskIns = INS_vpmovq2m;
kmovIns = INS_kmovb_gpr;
break;
+ }
+
default:
+ {
unreached();
+ }
}
assert(emitter::isMaskReg(maskReg));
emit->emitIns_R_R(maskIns, attr, maskReg, op1Reg);
- emit->emitIns_Mov(kmovIns, EA_8BYTE, targetReg, maskReg, INS_FLAGS_DONT_CARE);
- break;
- }
-
- case NI_AVX512F_CompareEqualSpecial:
- {
- GenTree* op2 = node->Op(2);
- op1Reg = op1->GetRegNum();
- regNumber op2Reg = op2->GetRegNum();
-
- instruction compareIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_CompareEqualSpecial, baseType);
-
- assert(compareIns != INS_invalid);
- assert(emitter::isMaskReg(targetReg));
-
- emit->emitIns_R_R_R_I(compareIns, attr, targetReg, op1Reg, op2Reg, 0);
+ emit->emitIns_Mov(kmovIns, kmovAttr, targetReg, maskReg, INS_FLAGS_DONT_CARE);
break;
}
- case NI_AVX512F_CompareGreaterThanOrEqualSpecial:
- {
- GenTree* op2 = node->Op(2);
- op1Reg = op1->GetRegNum();
- regNumber op2Reg = op2->GetRegNum();
-
- instruction compareIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_CompareGreaterThanOrEqualSpecial, baseType);
-
- assert(compareIns != INS_invalid);
- assert(emitter::isMaskReg(targetReg));
-
- emit->emitIns_R_R_R_I(compareIns, attr, targetReg, op1Reg, op2Reg, 5);
- break;
- }
-
- case NI_AVX512F_CompareGreaterThanSpecial:
- {
- GenTree* op2 = node->Op(2);
- op1Reg = op1->GetRegNum();
- regNumber op2Reg = op2->GetRegNum();
-
- instruction compareIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_CompareGreaterThanSpecial, baseType);
-
- assert(compareIns != INS_invalid);
- assert(emitter::isMaskReg(targetReg));
-
- emit->emitIns_R_R_R_I(compareIns, attr, targetReg, op1Reg, op2Reg, 6);
- break;
- }
-
- case NI_AVX512F_CompareLessThanOrEqualSpecial:
- {
- GenTree* op2 = node->Op(2);
- op1Reg = op1->GetRegNum();
- regNumber op2Reg = op2->GetRegNum();
-
- instruction compareIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_CompareLessThanOrEqualSpecial, baseType);
-
- assert(compareIns != INS_invalid);
- assert(emitter::isMaskReg(targetReg));
-
- emit->emitIns_R_R_R_I(compareIns, attr, targetReg, op1Reg, op2Reg, 2);
- break;
- }
-
- case NI_AVX512F_CompareLessThanSpecial:
- {
- GenTree* op2 = node->Op(2);
- op1Reg = op1->GetRegNum();
- regNumber op2Reg = op2->GetRegNum();
-
- instruction compareIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_CompareLessThanSpecial, baseType);
-
- assert(compareIns != INS_invalid);
- assert(emitter::isMaskReg(targetReg));
-
- emit->emitIns_R_R_R_I(compareIns, attr, targetReg, op1Reg, op2Reg, 1);
- break;
- }
-
- case NI_AVX512F_MoveMaskToVectorSpecial:
+ case NI_AVX512F_KORTEST:
{
op1Reg = op1->GetRegNum();
- instruction maskMovIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_MoveMaskToVectorSpecial, baseType);
+ uint32_t simdSize = node->GetSimdSize();
+ uint32_t count = simdSize / genTypeSize(baseType);
- assert(maskMovIns != INS_invalid);
- assert(emitter::isMaskReg(op1Reg));
+ instruction testIns;
- emit->emitIns_R_R(maskMovIns, attr, targetReg, op1Reg);
- break;
- }
-
- case NI_AVX512F_KORTEST:
- {
- op1Reg = op1->GetRegNum();
-
- instruction testIns = HWIntrinsicInfo::lookupIns(NI_AVX512F_KORTEST, baseType);
+ if (count <= 8)
+ {
+ testIns = INS_kortestb;
+ }
+ else if (count == 16)
+ {
+ testIns = INS_kortestw;
+ }
+ else if (count == 32)
+ {
+ testIns = INS_kortestd;
+ }
+ else
+ {
+ assert(count == 64);
+ testIns = INS_kortestq;
+ }
assert(testIns != INS_invalid);
assert(emitter::isMaskReg(op1Reg));
diff --git a/src/coreclr/jit/hwintrinsiclistarm64.h b/src/coreclr/jit/hwintrinsiclistarm64.h
index 2f1e4e03217133..97c9b66ad149b6 100644
--- a/src/coreclr/jit/hwintrinsiclistarm64.h
+++ b/src/coreclr/jit/hwintrinsiclistarm64.h
@@ -11,747 +11,747 @@
#ifdef FEATURE_HW_INTRINSICS
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Vector64 Intrinsics
-HARDWARE_INTRINSIC(Vector64, Abs, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Add, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, AndNot, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, As, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsByte, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsDouble, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsInt16, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsInt32, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsInt64, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsNInt, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsNUInt, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsSByte, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsSingle, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsUInt16, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsUInt32, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, AsUInt64, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, BitwiseAnd, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, BitwiseOr, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Ceiling, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ConditionalSelect, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ConvertToDouble, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, ConvertToInt32, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, ConvertToInt64, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, ConvertToSingle, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, ConvertToUInt32, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, ConvertToUInt64, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, Create, 8, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, CreateScalar, 8, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, CreateScalarUnsafe, 8, 1, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_invalid, INS_invalid, INS_fmov, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(Vector64, Divide, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Dot, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Equals, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, EqualsAll, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, EqualsAny, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ExtractMostSignificantBits, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Floor, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, get_AllBitsSet, 8, 0, {INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, get_One, 8, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, get_Zero, 8, 0, {INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, GetElement, 8, 2, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, GreaterThan, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, GreaterThanAll, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, GreaterThanAny, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqual, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqualAll, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqualAny, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThan, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThanAll, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThanAny, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThanOrEqual, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThanOrEqualAll, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LessThanOrEqualAny, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Load, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LoadAligned, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LoadAlignedNonTemporal, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, LoadUnsafe, 8, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Max, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Min, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Multiply, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Narrow, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Negate, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, OnesComplement, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_Addition, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_BitwiseAnd, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector64, op_BitwiseOr, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector64, op_Division, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_Equality, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector64, op_ExclusiveOr, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_Inequality, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector64, op_LeftShift, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_Multiply, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_OnesComplement, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_RightShift, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_Subtraction, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_UnaryNegation, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_UnaryPlus, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, op_UnsignedRightShift, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ShiftLeft, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ShiftRightArithmetic, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ShiftRightLogical, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Shuffle, 8, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Sqrt, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Store, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, StoreAligned, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, StoreAlignedNonTemporal, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, StoreUnsafe, 8, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Subtract, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, Sum, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector64, ToScalar, 8, 1, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector64, ToVector128, 8, 1, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector64, ToVector128Unsafe, 8, 1, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector64, WidenLower, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, WidenUpper, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector64, WithElement, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector64, Xor, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Abs, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Add, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, AndNot, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, As, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsByte, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsDouble, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsInt16, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsInt32, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsInt64, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsNInt, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsNUInt, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsSByte, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsSingle, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsUInt16, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsUInt32, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, AsUInt64, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, BitwiseAnd, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, BitwiseOr, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Ceiling, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ConditionalSelect, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ConvertToDouble, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, ConvertToInt32, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, ConvertToInt64, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, ConvertToSingle, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, ConvertToUInt32, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, ConvertToUInt64, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, Create, 8, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, CreateScalar, 8, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, CreateScalarUnsafe, 8, 1, true, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_invalid, INS_invalid, INS_fmov, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(Vector64, Divide, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Dot, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Equals, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, EqualsAll, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, EqualsAny, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ExtractMostSignificantBits, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Floor, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, get_AllBitsSet, 8, 0, true, {INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, get_One, 8, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, get_Zero, 8, 0, true, {INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, GetElement, 8, 2, true, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, GreaterThan, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, GreaterThanAll, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, GreaterThanAny, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqual, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqualAll, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, GreaterThanOrEqualAny, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThan, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThanAll, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThanAny, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThanOrEqual, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThanOrEqualAll, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LessThanOrEqualAny, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Load, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LoadAligned, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LoadAlignedNonTemporal, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, LoadUnsafe, 8, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Max, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Min, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Multiply, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Narrow, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Negate, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, OnesComplement, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_Addition, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_BitwiseAnd, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector64, op_BitwiseOr, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector64, op_Division, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_Equality, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector64, op_ExclusiveOr, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_Inequality, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector64, op_LeftShift, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_Multiply, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_OnesComplement, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_RightShift, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_Subtraction, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_UnaryNegation, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_UnaryPlus, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, op_UnsignedRightShift, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ShiftLeft, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ShiftRightArithmetic, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ShiftRightLogical, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Shuffle, 8, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Sqrt, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Store, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, StoreAligned, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, StoreAlignedNonTemporal, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, StoreUnsafe, 8, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Subtract, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, Sum, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector64, ToScalar, 8, 1, true, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector64, ToVector128, 8, 1, true, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector64, ToVector128Unsafe, 8, 1, true, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector64, WidenLower, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, WidenUpper, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector64, WithElement, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector64, Xor, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Vector128 Intrinsics
-HARDWARE_INTRINSIC(Vector128, Abs, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Add, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AndNot, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, As, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsByte, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsDouble, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsInt16, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsNInt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsNUInt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsSByte, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsSingle, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsUInt16, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsVector, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsVector2, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsVector3, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mov, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsVector4, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, AsVector128, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, BitwiseAnd, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, BitwiseOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Ceiling, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ConditionalSelect, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ConvertToDouble, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToSingle, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, Create, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, CreateScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, CreateScalarUnsafe, 16, 1, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_fmov, INS_fmov}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(Vector128, Divide, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Dot, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Equals, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, EqualsAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, EqualsAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ExtractMostSignificantBits, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Floor, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, get_AllBitsSet, 16, 0, {INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, get_One, 16, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, get_Zero, 16, 0, {INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, GetElement, 16, 2, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, GetLower, 16, 1, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector128, GetUpper, 16, 1, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Load, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadAligned, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadAlignedNonTemporal, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadUnsafe, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Max, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Min, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Narrow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Negate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, OnesComplement, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Addition, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_BitwiseAnd, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_BitwiseOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_Division, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Equality, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_ExclusiveOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Inequality, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_LeftShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_RightShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnsignedRightShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_OnesComplement, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Subtraction, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnaryNegation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnaryPlus, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftLeft, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftRightArithmetic, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftRightLogical, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Shuffle, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Sqrt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Store, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreAligned, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreAlignedNonTemporal, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreUnsafe, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Subtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Sum, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ToScalar, 16, 1, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector128, WidenLower, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, WidenUpper, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, WithElement, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(Vector128, WithLower, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, WithUpper, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Xor, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Abs, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Add, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AndNot, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, As, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsByte, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsDouble, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsInt16, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsNInt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsNUInt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsSByte, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsSingle, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsUInt16, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsUInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsUInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsVector, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsVector2, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsVector3, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mov, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsVector4, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, AsVector128, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, BitwiseAnd, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, BitwiseOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Ceiling, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ConditionalSelect, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ConvertToDouble, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToSingle, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToUInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToUInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, Create, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, CreateScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, CreateScalarUnsafe, 16, 1, true, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_fmov, INS_fmov}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(Vector128, Divide, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Dot, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Equals, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, EqualsAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, EqualsAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ExtractMostSignificantBits, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Floor, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, get_AllBitsSet, 16, 0, true, {INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni, INS_mvni}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, get_One, 16, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, get_Zero, 16, 0, true, {INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi, INS_movi}, HW_Category_Helper, HW_Flag_NoCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, GetElement, 16, 2, true, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, GetLower, 16, 1, true, {INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov, INS_mov}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector128, GetUpper, 16, 1, true, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Load, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadAligned, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadAlignedNonTemporal, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadUnsafe, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Max, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Min, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Narrow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Negate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, OnesComplement, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Addition, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_BitwiseAnd, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_BitwiseOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_Division, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Equality, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_ExclusiveOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Inequality, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_LeftShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_RightShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnsignedRightShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_OnesComplement, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Subtraction, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnaryNegation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnaryPlus, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftLeft, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftRightArithmetic, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftRightLogical, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Shuffle, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Sqrt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Store, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreAligned, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreAlignedNonTemporal, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreUnsafe, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Subtract, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Sum, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ToScalar, 16, 1, true, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector128, WidenLower, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, WidenUpper, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, WithElement, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(Vector128, WithLower, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, WithUpper, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Xor, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AdvSimd Intrinsics
-HARDWARE_INTRINSIC(AdvSimd, Abs, -1, 1, {INS_abs, INS_invalid, INS_abs, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_invalid, INS_fabs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, AbsSaturate, -1, 1, {INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, AbsScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabs, INS_fabs}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareGreaterThan, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareGreaterThanOrEqual, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareLessThan, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareLessThanOrEqual, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifference, -1, 2, {INS_sabd, INS_uabd, INS_sabd, INS_uabd, INS_sabd, INS_uabd, INS_invalid, INS_invalid, INS_fabd, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceAdd, -1, 3, {INS_saba, INS_uaba, INS_saba, INS_uaba, INS_saba, INS_uaba, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningLower, 8, 2, {INS_sabdl, INS_uabdl, INS_sabdl, INS_uabdl, INS_sabdl, INS_uabdl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningLowerAndAdd, 8, 3, {INS_sabal, INS_uabal, INS_sabal, INS_uabal, INS_sabal, INS_uabal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningUpper, 16, 2, {INS_sabdl2, INS_uabdl2, INS_sabdl2, INS_uabdl2, INS_sabdl2, INS_uabdl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningUpperAndAdd, 16, 3, {INS_sabal2, INS_uabal2, INS_sabal2, INS_uabal2, INS_sabal2, INS_uabal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, Add, -1, 2, {INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_fadd, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AddHighNarrowingLower, 8, 2, {INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AddHighNarrowingUpper, 16, 3, {INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AddPairwise, 8, 2, {INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_invalid, INS_invalid, INS_faddp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWidening, -1, 1, {INS_saddlp, INS_uaddlp, INS_saddlp, INS_uaddlp, INS_saddlp, INS_uaddlp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningAndAdd, -1, 2, {INS_sadalp, INS_uadalp, INS_sadalp, INS_uadalp, INS_sadalp, INS_uadalp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningAndAddScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sadalp, INS_uadalp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_saddlp, INS_uaddlp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, AddRoundedHighNarrowingLower, 8, 2, {INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AddRoundedHighNarrowingUpper, 16, 3, {INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, AddSaturate, -1, 2, {INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, AddSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqadd, INS_uqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, AddScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_add, INS_add, INS_fadd, INS_fadd}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, AddWideningLower, 8, 2, {INS_saddl, INS_uaddl, INS_saddl, INS_uaddl, INS_saddl, INS_uaddl, INS_saddw, INS_uaddw, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, AddWideningUpper, 16, 2, {INS_saddl2, INS_uaddl2, INS_saddl2, INS_uaddl2, INS_saddl2, INS_uaddl2, INS_saddw2, INS_uaddw2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, And, -1, 2, {INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, BitwiseClear, -1, 2, {INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, BitwiseSelect, -1, 3, {INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, Ceiling, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, CeilingScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_frintp}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, CompareEqual, -1, 2, {INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_invalid, INS_invalid, INS_fcmeq, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, CompareGreaterThan, -1, 2, {INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_invalid, INS_invalid, INS_fcmgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, CompareGreaterThanOrEqual, -1, 2, {INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_invalid, INS_invalid, INS_fcmge, INS_invalid}, HW_Category_SIMD, HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, CompareLessThan, -1, 2, {INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_invalid, INS_invalid, INS_fcmgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, CompareLessThanOrEqual, -1, 2, {INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_invalid, INS_invalid, INS_fcmge, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, CompareTest, -1, 2, {INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_invalid, INS_invalid, INS_cmtst, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundAwayFromZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundAwayFromZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToEven, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToEvenScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToNegativeInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToNegativeInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToPositiveInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToPositiveInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToSingle, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToSingleScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundAwayFromZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundAwayFromZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToEven, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToEvenScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToNegativeInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToNegativeInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToPositiveInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToPositiveInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, DivideScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fdiv, INS_fdiv}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, DuplicateSelectedScalarToVector64, -1, 2, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, DuplicateSelectedScalarToVector128, -1, 2, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, DuplicateToVector64, 8, 1, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, DuplicateToVector128, 16, 1, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, Extract, -1, 2, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingLower, 8, 1, {INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateLower, 8, 1, {INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUnsignedLower, 8, 1, {INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUnsignedUpper, 16, 2, {INS_invalid, INS_sqxtun2, INS_invalid, INS_sqxtun2, INS_invalid, INS_sqxtun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUpper, 16, 2, {INS_sqxtn2, INS_uqxtn2, INS_sqxtn2, INS_uqxtn2, INS_sqxtn2, INS_uqxtn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingUpper, 16, 2, {INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ExtractVector64, 8, 3, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_invalid, INS_invalid, INS_ext, INS_invalid}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, ExtractVector128, 16, 3, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, Floor, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, FloorScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_frintm}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, FusedAddHalving, -1, 2, {INS_shadd, INS_uhadd, INS_shadd, INS_uhadd, INS_shadd, INS_uhadd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, FusedAddRoundedHalving, -1, 2, {INS_srhadd, INS_urhadd, INS_srhadd, INS_urhadd, INS_srhadd, INS_urhadd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAdd, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAddScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmadd, INS_fmadd}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAddNegatedScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fnmadd, INS_fnmadd}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtract, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtractScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmsub, INS_fmsub}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtractNegatedScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fnmsub, INS_fnmsub}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, FusedSubtractHalving, -1, 2, {INS_shsub, INS_uhsub, INS_shsub, INS_uhsub, INS_shsub, INS_uhsub, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, Insert, -1, 3, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd, InsertScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ins, INS_ins, INS_invalid, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, LeadingSignCount, -1, 1, {INS_cls, INS_invalid, INS_cls, INS_invalid, INS_cls, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, LeadingZeroCount, -1, 1, {INS_clz, INS_clz, INS_clz, INS_clz, INS_clz, INS_clz, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, LoadAndInsertScalar, -1, 3, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_MemoryLoad, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, LoadAndReplicateToVector64, 8, 1, {INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_invalid, INS_invalid, INS_ld1r, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, LoadAndReplicateToVector128, 16, 1, {INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_invalid, INS_invalid, INS_ld1r, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, LoadVector64, 8, 1, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, LoadVector128, 16, 1, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, Max, -1, 2, {INS_smax, INS_umax, INS_smax, INS_umax, INS_smax, INS_umax, INS_invalid, INS_invalid, INS_fmax, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MaxNumber, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MaxNumberScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm, INS_fmaxnm}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, MaxPairwise, 8, 2, {INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_invalid, INS_invalid, INS_fmaxp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, Min, -1, 2, {INS_smin, INS_umin, INS_smin, INS_umin, INS_smin, INS_umin, INS_invalid, INS_invalid, INS_fmin, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MinNumber, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MinNumberScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm, INS_fminnm}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, MinPairwise, 8, 2, {INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_invalid, INS_invalid, INS_fminp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, Multiply, -1, 2, {INS_mul, INS_mul, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyAdd, -1, 3, {INS_mla, INS_mla, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyAddByScalar, -1, 3, {INS_invalid, INS_invalid, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyAddBySelectedScalar, -1, 4, {INS_invalid, INS_invalid, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyByScalar, -1, 2, {INS_invalid, INS_invalid, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalar, -1, 3, {INS_invalid, INS_invalid, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLower, 8, 3, {INS_invalid, INS_invalid, INS_smull, INS_umull, INS_smull, INS_umull, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLowerAndAdd, 8, 4, {INS_invalid, INS_invalid, INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLowerAndSubtract, 8, 4, {INS_invalid, INS_invalid, INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpper, 16, 3, {INS_invalid, INS_invalid, INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpperAndAdd, 16, 4, {INS_invalid, INS_invalid, INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpperAndSubtract, 16, 4, {INS_invalid, INS_invalid, INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingByScalarSaturateHigh, -1, 2, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingBySelectedScalarSaturateHigh, -1, 3, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingSaturateHigh, -1, 2, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerAndAddSaturate, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerAndSubtractSaturate, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerByScalarAndAddSaturate, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerByScalarAndSubtractSaturate, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerBySelectedScalarAndAddSaturate, 8, 4, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerBySelectedScalarAndSubtractSaturate, 8, 4, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLower, 8, 2, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLowerByScalar, 8, 2, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLowerBySelectedScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpper, 16, 2, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpperByScalar, 16, 2, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpperBySelectedScalar, 16, 3, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperAndAddSaturate, 16, 3, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperAndSubtractSaturate, 16, 3, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperByScalarAndAddSaturate, 16, 3, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperByScalarAndSubtractSaturate, 16, 3, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperBySelectedScalarAndAddSaturate, 16, 4, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperBySelectedScalarAndSubtractSaturate, 16, 4, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingByScalarSaturateHigh, -1, 2, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingBySelectedScalarSaturateHigh, -1, 3, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingSaturateHigh, -1, 2, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul, INS_fmul}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyScalarBySelectedScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, MultiplySubtract, -1, 3, {INS_mls, INS_mls, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplySubtractByScalar, -1, 3, {INS_invalid, INS_invalid, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplySubtractBySelectedScalar, -1, 4, {INS_invalid, INS_invalid, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLower, 8, 2, {INS_smull, INS_umull, INS_smull, INS_umull, INS_smull, INS_umull, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLowerAndAdd, 8, 3, {INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLowerAndSubtract, 8, 3, {INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpper, 16, 2, {INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpperAndAdd, 16, 3, {INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpperAndSubtract, 16, 3, {INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, Negate, -1, 1, {INS_neg, INS_invalid, INS_neg, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_invalid, INS_fneg, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, NegateSaturate, -1, 1, {INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, NegateScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fneg, INS_fneg}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, Not, -1, 1, {INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, Or, -1, 2, {INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, OrNot, -1, 2, {INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiply, -1, 2, {INS_pmul, INS_pmul, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiplyWideningLower, 8, 2, {INS_pmull, INS_pmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiplyWideningUpper, 16, 2, {INS_pmull2, INS_pmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, PopCount, -1, 1, {INS_cnt, INS_cnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ReciprocalEstimate, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urecpe, INS_invalid, INS_invalid, INS_frecpe, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ReciprocalSquareRootEstimate, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ursqrte, INS_invalid, INS_invalid, INS_frsqrte, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ReciprocalSquareRootStep, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, ReciprocalStep, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, ReverseElement16, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rev32, INS_rev32, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, ReverseElement32, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, ReverseElement8, -1, 1, {INS_invalid, INS_invalid, INS_rev16, INS_rev16, INS_rev32, INS_rev32, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, RoundAwayFromZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, RoundAwayFromZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta, INS_frinta}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, RoundToNearest, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, RoundToNearestScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn, INS_frintn}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, RoundToNegativeInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, RoundToNegativeInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_frintm}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, RoundToPositiveInfinity, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, RoundToPositiveInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_frintp}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, RoundToZero, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, RoundToZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz, INS_frintz}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmetic, -1, 2, {INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRounded, -1, 2, {INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedSaturate, -1, 2, {INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticSaturate, -1, 2, {INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftAndInsert, -1, 3, {INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftAndInsertScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sli, INS_sli, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogical, -1, 2, {INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturate, -1, 2, {INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateUnsigned, -1, 2, {INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateUnsignedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shl, INS_shl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalWideningLower, 8, 2, {INS_sshll, INS_ushll, INS_sshll, INS_ushll, INS_sshll, INS_ushll, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalWideningUpper, 16, 2, {INS_sshll2, INS_ushll2, INS_sshll2, INS_ushll2, INS_sshll2, INS_ushll2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogical, -1, 2, {INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRounded, -1, 2, {INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedSaturate, -1, 2, {INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urshl, INS_urshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalSaturate, -1, 2, {INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ushl, INS_ushl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightAndInsert, -1, 3, {INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightAndInsertScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sri, INS_sri, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmetic, -1, 2, {INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticAdd, -1, 3, {INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticAddScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ssra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateLower, 8, 2, {INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUnsignedLower, 8, 2, {INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUnsignedUpper, 16, 3, {INS_invalid, INS_sqshrun2, INS_invalid, INS_sqshrun2, INS_invalid, INS_sqshrun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUpper, 16, 3, {INS_sqshrn2, INS_invalid, INS_sqshrn2, INS_invalid, INS_sqshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRounded, -1, 2, {INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedAdd, -1, 3, {INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedAddScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srsra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateLower, 8, 2, {INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedLower, 8, 2, {INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedUpper, 16, 3, {INS_invalid, INS_sqrshrun2, INS_invalid, INS_sqrshrun2, INS_invalid, INS_sqrshrun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUpper, 16, 3, {INS_sqrshrn2, INS_invalid, INS_sqrshrn2, INS_invalid, INS_sqrshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogical, -1, 2, {INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalAdd, -1, 3, {INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalAddScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_usra, INS_usra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingLower, 8, 2, {INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingSaturateLower, 8, 2, {INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingSaturateUpper, 16, 3, {INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingUpper, 16, 3, {INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRounded, -1, 2, {INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedAdd, -1, 3, {INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedAddScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ursra, INS_ursra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingLower, 8, 2, {INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingSaturateLower, 8, 2, {INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingSaturateUpper, 16, 3, {INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingUpper, 16, 3, {INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urshr, INS_urshr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ushr, INS_ushr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, SignExtendWideningLower, 8, 1, {INS_sxtl, INS_invalid, INS_sxtl, INS_invalid, INS_sxtl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, SignExtendWideningUpper, 16, 1, {INS_sxtl2, INS_invalid, INS_sxtl2, INS_invalid, INS_sxtl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, SqrtScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsqrt, INS_fsqrt}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, Store, -1, 2, {INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, StoreSelectedScalar, -1, 3, {INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, Subtract, -1, 2, {INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_fsub, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, SubtractHighNarrowingLower, 8, 2, {INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, SubtractHighNarrowingUpper, 16, 3, {INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, SubtractRoundedHighNarrowingLower, 8, 2, {INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, SubtractRoundedHighNarrowingUpper, 16, 3, {INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd, SubtractSaturate, -1, 2, {INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd, SubtractSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, SubtractScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sub, INS_sub, INS_fsub, INS_fsub}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd, SubtractWideningLower, 8, 2, {INS_ssubl, INS_usubl, INS_ssubl, INS_usubl, INS_ssubl, INS_usubl, INS_ssubw, INS_usubw, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, SubtractWideningUpper, 16, 2, {INS_ssubl2, INS_usubl2, INS_ssubl2, INS_usubl2, INS_ssubl2, INS_usubl2, INS_ssubw2, INS_usubw2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd, VectorTableLookup, 8, 2, {INS_tbl, INS_tbl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NeedsConsecutiveRegisters)
-HARDWARE_INTRINSIC(AdvSimd, VectorTableLookupExtension, 8, 3, {INS_tbx, INS_tbx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_HasRMWSemantics|HW_Flag_NeedsConsecutiveRegisters)
-HARDWARE_INTRINSIC(AdvSimd, Xor, -1, 2, {INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd, ZeroExtendWideningLower, 8, 1, {INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd, ZeroExtendWideningUpper, 16, 1, {INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, Abs, -1, 1, true, {INS_abs, INS_invalid, INS_abs, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_invalid, INS_fabs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, AbsSaturate, -1, 1, true, {INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, AbsScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabs, INS_fabs}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareGreaterThan, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareGreaterThanOrEqual, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareLessThan, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteCompareLessThanOrEqual, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifference, -1, 2, true, {INS_sabd, INS_uabd, INS_sabd, INS_uabd, INS_sabd, INS_uabd, INS_invalid, INS_invalid, INS_fabd, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceAdd, -1, 3, true, {INS_saba, INS_uaba, INS_saba, INS_uaba, INS_saba, INS_uaba, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningLower, 8, 2, true, {INS_sabdl, INS_uabdl, INS_sabdl, INS_uabdl, INS_sabdl, INS_uabdl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningLowerAndAdd, 8, 3, true, {INS_sabal, INS_uabal, INS_sabal, INS_uabal, INS_sabal, INS_uabal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningUpper, 16, 2, true, {INS_sabdl2, INS_uabdl2, INS_sabdl2, INS_uabdl2, INS_sabdl2, INS_uabdl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AbsoluteDifferenceWideningUpperAndAdd, 16, 3, true, {INS_sabal2, INS_uabal2, INS_sabal2, INS_uabal2, INS_sabal2, INS_uabal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, Add, -1, 2, true, {INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_add, INS_fadd, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AddHighNarrowingLower, 8, 2, true, {INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_addhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AddHighNarrowingUpper, 16, 3, true, {INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_addhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AddPairwise, 8, 2, true, {INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_invalid, INS_invalid, INS_faddp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWidening, -1, 1, true, {INS_saddlp, INS_uaddlp, INS_saddlp, INS_uaddlp, INS_saddlp, INS_uaddlp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningAndAdd, -1, 2, true, {INS_sadalp, INS_uadalp, INS_sadalp, INS_uadalp, INS_sadalp, INS_uadalp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningAndAddScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sadalp, INS_uadalp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AddPairwiseWideningScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_saddlp, INS_uaddlp, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, AddRoundedHighNarrowingLower, 8, 2, true, {INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_raddhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AddRoundedHighNarrowingUpper, 16, 3, true, {INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_raddhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, AddSaturate, -1, 2, true, {INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, AddSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqadd, INS_uqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, AddScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_add, INS_add, INS_fadd, INS_fadd}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, AddWideningLower, 8, 2, true, {INS_saddl, INS_uaddl, INS_saddl, INS_uaddl, INS_saddl, INS_uaddl, INS_saddw, INS_uaddw, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, AddWideningUpper, 16, 2, true, {INS_saddl2, INS_uaddl2, INS_saddl2, INS_uaddl2, INS_saddl2, INS_uaddl2, INS_saddw2, INS_uaddw2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, And, -1, 2, true, {INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and, INS_and}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, BitwiseClear, -1, 2, true, {INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic, INS_bic}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, BitwiseSelect, -1, 3, true, {INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl, INS_bsl}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, Ceiling, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, CeilingScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_frintp}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, CompareEqual, -1, 2, true, {INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_cmeq, INS_invalid, INS_invalid, INS_fcmeq, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, CompareGreaterThan, -1, 2, true, {INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_invalid, INS_invalid, INS_fcmgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, CompareGreaterThanOrEqual, -1, 2, true, {INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_invalid, INS_invalid, INS_fcmge, INS_invalid}, HW_Category_SIMD, HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, CompareLessThan, -1, 2, true, {INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_cmgt, INS_cmhi, INS_invalid, INS_invalid, INS_fcmgt, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, CompareLessThanOrEqual, -1, 2, true, {INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_cmge, INS_cmhs, INS_invalid, INS_invalid, INS_fcmge, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, CompareTest, -1, 2, true, {INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_cmtst, INS_invalid, INS_invalid, INS_cmtst, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundAwayFromZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundAwayFromZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToEven, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToEvenScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToNegativeInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToNegativeInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToPositiveInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToPositiveInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToInt32RoundToZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToSingle, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToSingleScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundAwayFromZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundAwayFromZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToEven, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToEvenScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToNegativeInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToNegativeInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToPositiveInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToPositiveInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ConvertToUInt32RoundToZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, DivideScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fdiv, INS_fdiv}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, DuplicateSelectedScalarToVector64, -1, 2, true, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, DuplicateSelectedScalarToVector128, -1, 2, true, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, DuplicateToVector64, 8, 1, true, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, DuplicateToVector128, 16, 1, true, {INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_dup, INS_invalid, INS_invalid, INS_dup, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, Extract, -1, 2, true, {INS_smov, INS_umov, INS_smov, INS_umov, INS_smov, INS_umov, INS_umov, INS_umov, INS_dup, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingLower, 8, 1, true, {INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_xtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateLower, 8, 1, true, {INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUnsignedLower, 8, 1, true, {INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUnsignedUpper, 16, 2, true, {INS_invalid, INS_sqxtun2, INS_invalid, INS_sqxtun2, INS_invalid, INS_sqxtun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingSaturateUpper, 16, 2, true, {INS_sqxtn2, INS_uqxtn2, INS_sqxtn2, INS_uqxtn2, INS_sqxtn2, INS_uqxtn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ExtractNarrowingUpper, 16, 2, true, {INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_xtn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ExtractVector64, 8, 3, true, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_invalid, INS_invalid, INS_ext, INS_invalid}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, ExtractVector128, 16, 3, true, {INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext, INS_ext}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, Floor, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, FloorScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_frintm}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, FusedAddHalving, -1, 2, true, {INS_shadd, INS_uhadd, INS_shadd, INS_uhadd, INS_shadd, INS_uhadd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, FusedAddRoundedHalving, -1, 2, true, {INS_srhadd, INS_urhadd, INS_srhadd, INS_urhadd, INS_srhadd, INS_urhadd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAdd, -1, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAddScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmadd, INS_fmadd}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplyAddNegatedScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fnmadd, INS_fnmadd}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtract, -1, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtractScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmsub, INS_fmsub}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, FusedMultiplySubtractNegatedScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fnmsub, INS_fnmsub}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, FusedSubtractHalving, -1, 2, true, {INS_shsub, INS_uhsub, INS_shsub, INS_uhsub, INS_shsub, INS_uhsub, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, Insert, -1, 3, true, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd, InsertScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ins, INS_ins, INS_invalid, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, LeadingSignCount, -1, 1, true, {INS_cls, INS_invalid, INS_cls, INS_invalid, INS_cls, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, LeadingZeroCount, -1, 1, true, {INS_clz, INS_clz, INS_clz, INS_clz, INS_clz, INS_clz, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, LoadAndInsertScalar, -1, 3, true, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_MemoryLoad, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, LoadAndReplicateToVector64, 8, 1, true, {INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_invalid, INS_invalid, INS_ld1r, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, LoadAndReplicateToVector128, 16, 1, true, {INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_ld1r, INS_invalid, INS_invalid, INS_ld1r, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, LoadVector64, 8, 1, true, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, LoadVector128, 16, 1, true, {INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1, INS_ld1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, Max, -1, 2, true, {INS_smax, INS_umax, INS_smax, INS_umax, INS_smax, INS_umax, INS_invalid, INS_invalid, INS_fmax, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MaxNumber, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MaxNumberScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm, INS_fmaxnm}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, MaxPairwise, 8, 2, true, {INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_invalid, INS_invalid, INS_fmaxp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, Min, -1, 2, true, {INS_smin, INS_umin, INS_smin, INS_umin, INS_smin, INS_umin, INS_invalid, INS_invalid, INS_fmin, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MinNumber, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MinNumberScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm, INS_fminnm}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, MinPairwise, 8, 2, true, {INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_invalid, INS_invalid, INS_fminp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, Multiply, -1, 2, true, {INS_mul, INS_mul, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyAdd, -1, 3, true, {INS_mla, INS_mla, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyAddByScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyAddBySelectedScalar, -1, 4, true, {INS_invalid, INS_invalid, INS_mla, INS_mla, INS_mla, INS_mla, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyByScalar, -1, 2, true, {INS_invalid, INS_invalid, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_mul, INS_mul, INS_mul, INS_mul, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLower, 8, 3, true, {INS_invalid, INS_invalid, INS_smull, INS_umull, INS_smull, INS_umull, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLowerAndAdd, 8, 4, true, {INS_invalid, INS_invalid, INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningLowerAndSubtract, 8, 4, true, {INS_invalid, INS_invalid, INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpper, 16, 3, true, {INS_invalid, INS_invalid, INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpperAndAdd, 16, 4, true, {INS_invalid, INS_invalid, INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyBySelectedScalarWideningUpperAndSubtract, 16, 4, true, {INS_invalid, INS_invalid, INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingByScalarSaturateHigh, -1, 2, true, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingBySelectedScalarSaturateHigh, -1, 3, true, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingSaturateHigh, -1, 2, true, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerAndAddSaturate, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerAndSubtractSaturate, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerByScalarAndAddSaturate, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerByScalarAndSubtractSaturate, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerBySelectedScalarAndAddSaturate, 8, 4, true, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningLowerBySelectedScalarAndSubtractSaturate, 8, 4, true, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLower, 8, 2, true, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLowerByScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateLowerBySelectedScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpper, 16, 2, true, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpperByScalar, 16, 2, true, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningSaturateUpperBySelectedScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_sqdmull2, INS_invalid, INS_sqdmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperAndAddSaturate, 16, 3, true, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperAndSubtractSaturate, 16, 3, true, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperByScalarAndAddSaturate, 16, 3, true, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperByScalarAndSubtractSaturate, 16, 3, true, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperBySelectedScalarAndAddSaturate, 16, 4, true, {INS_invalid, INS_invalid, INS_sqdmlal2, INS_invalid, INS_sqdmlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyDoublingWideningUpperBySelectedScalarAndSubtractSaturate, 16, 4, true, {INS_invalid, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_sqdmlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingByScalarSaturateHigh, -1, 2, true, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingBySelectedScalarSaturateHigh, -1, 3, true, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyRoundedDoublingSaturateHigh, -1, 2, true, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul, INS_fmul}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyScalarBySelectedScalar, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, MultiplySubtract, -1, 3, true, {INS_mls, INS_mls, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplySubtractByScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplySubtractBySelectedScalar, -1, 4, true, {INS_invalid, INS_invalid, INS_mls, INS_mls, INS_mls, INS_mls, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLower, 8, 2, true, {INS_smull, INS_umull, INS_smull, INS_umull, INS_smull, INS_umull, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLowerAndAdd, 8, 3, true, {INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_smlal, INS_umlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningLowerAndSubtract, 8, 3, true, {INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_smlsl, INS_umlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpper, 16, 2, true, {INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_smull2, INS_umull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpperAndAdd, 16, 3, true, {INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_smlal2, INS_umlal2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, MultiplyWideningUpperAndSubtract, 16, 3, true, {INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_smlsl2, INS_umlsl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, Negate, -1, 1, true, {INS_neg, INS_invalid, INS_neg, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_invalid, INS_fneg, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, NegateSaturate, -1, 1, true, {INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, NegateScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fneg, INS_fneg}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, Not, -1, 1, true, {INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn, INS_mvn}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, Or, -1, 2, true, {INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr, INS_orr}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, OrNot, -1, 2, true, {INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn, INS_orn}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiply, -1, 2, true, {INS_pmul, INS_pmul, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiplyWideningLower, 8, 2, true, {INS_pmull, INS_pmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, PolynomialMultiplyWideningUpper, 16, 2, true, {INS_pmull2, INS_pmull2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, PopCount, -1, 1, true, {INS_cnt, INS_cnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ReciprocalEstimate, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urecpe, INS_invalid, INS_invalid, INS_frecpe, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ReciprocalSquareRootEstimate, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ursqrte, INS_invalid, INS_invalid, INS_frsqrte, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ReciprocalSquareRootStep, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, ReciprocalStep, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, ReverseElement16, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rev32, INS_rev32, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, ReverseElement32, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, ReverseElement8, -1, 1, true, {INS_invalid, INS_invalid, INS_rev16, INS_rev16, INS_rev32, INS_rev32, INS_rev64, INS_rev64, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, RoundAwayFromZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, RoundAwayFromZeroScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta, INS_frinta}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, RoundToNearest, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, RoundToNearestScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn, INS_frintn}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, RoundToNegativeInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, RoundToNegativeInfinityScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm, INS_frintm}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, RoundToPositiveInfinity, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, RoundToPositiveInfinityScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp, INS_frintp}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, RoundToZero, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, RoundToZeroScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz, INS_frintz}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmetic, -1, 2, true, {INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_sshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRounded, -1, 2, true, {INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_srshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedSaturate, -1, 2, true, {INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedSaturateScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticRoundedScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticSaturate, -1, 2, true, {INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticSaturateScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftArithmeticScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sshl, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftAndInsert, -1, 3, true, {INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli, INS_sli}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftAndInsertScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sli, INS_sli, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogical, -1, 2, true, {INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_shl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturate, -1, 2, true, {INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateUnsigned, -1, 2, true, {INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalSaturateUnsignedScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shl, INS_shl, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalWideningLower, 8, 2, true, {INS_sshll, INS_ushll, INS_sshll, INS_ushll, INS_sshll, INS_ushll, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLeftLogicalWideningUpper, 16, 2, true, {INS_sshll2, INS_ushll2, INS_sshll2, INS_ushll2, INS_sshll2, INS_ushll2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogical, -1, 2, true, {INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_ushl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRounded, -1, 2, true, {INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_urshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedSaturate, -1, 2, true, {INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalRoundedScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urshl, INS_urshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalSaturate, -1, 2, true, {INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftLogicalScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ushl, INS_ushl, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightAndInsert, -1, 3, true, {INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri, INS_sri}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightAndInsertScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sri, INS_sri, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmetic, -1, 2, true, {INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_sshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticAdd, -1, 3, true, {INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_ssra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticAddScalar, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ssra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateLower, 8, 2, true, {INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUnsignedLower, 8, 2, true, {INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUnsignedUpper, 16, 3, true, {INS_invalid, INS_sqshrun2, INS_invalid, INS_sqshrun2, INS_invalid, INS_sqshrun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticNarrowingSaturateUpper, 16, 3, true, {INS_sqshrn2, INS_invalid, INS_sqshrn2, INS_invalid, INS_sqshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRounded, -1, 2, true, {INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_srshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedAdd, -1, 3, true, {INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_srsra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedAddScalar, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srsra, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateLower, 8, 2, true, {INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedLower, 8, 2, true, {INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedUpper, 16, 3, true, {INS_invalid, INS_sqrshrun2, INS_invalid, INS_sqrshrun2, INS_invalid, INS_sqrshrun2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedNarrowingSaturateUpper, 16, 3, true, {INS_sqrshrn2, INS_invalid, INS_sqrshrn2, INS_invalid, INS_sqrshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticRoundedScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_srshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightArithmeticScalar, 8, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sshr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogical, -1, 2, true, {INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_ushr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalAdd, -1, 3, true, {INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_usra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalAddScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_usra, INS_usra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingLower, 8, 2, true, {INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_shrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingSaturateLower, 8, 2, true, {INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingSaturateUpper, 16, 3, true, {INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_uqshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalNarrowingUpper, 16, 3, true, {INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_shrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRounded, -1, 2, true, {INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_urshr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedAdd, -1, 3, true, {INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_ursra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedAddScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ursra, INS_ursra, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingLower, 8, 2, true, {INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_rshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingSaturateLower, 8, 2, true, {INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingSaturateUpper, 16, 3, true, {INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_uqrshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedNarrowingUpper, 16, 3, true, {INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_rshrn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalRoundedScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_urshr, INS_urshr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, ShiftRightLogicalScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ushr, INS_ushr, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, SignExtendWideningLower, 8, 1, true, {INS_sxtl, INS_invalid, INS_sxtl, INS_invalid, INS_sxtl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, SignExtendWideningUpper, 16, 1, true, {INS_sxtl2, INS_invalid, INS_sxtl2, INS_invalid, INS_sxtl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, SqrtScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsqrt, INS_fsqrt}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, Store, -1, 2, true, {INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, StoreSelectedScalar, -1, 3, true, {INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1, INS_st1}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, Subtract, -1, 2, true, {INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_sub, INS_fsub, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, SubtractHighNarrowingLower, 8, 2, true, {INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_subhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, SubtractHighNarrowingUpper, 16, 3, true, {INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_subhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, SubtractRoundedHighNarrowingLower, 8, 2, true, {INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_rsubhn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, SubtractRoundedHighNarrowingUpper, 16, 3, true, {INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_rsubhn2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd, SubtractSaturate, -1, 2, true, {INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd, SubtractSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, SubtractScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sub, INS_sub, INS_fsub, INS_fsub}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd, SubtractWideningLower, 8, 2, true, {INS_ssubl, INS_usubl, INS_ssubl, INS_usubl, INS_ssubl, INS_usubl, INS_ssubw, INS_usubw, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, SubtractWideningUpper, 16, 2, true, {INS_ssubl2, INS_usubl2, INS_ssubl2, INS_usubl2, INS_ssubl2, INS_usubl2, INS_ssubw2, INS_usubw2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd, VectorTableLookup, 8, 2, true, {INS_tbl, INS_tbl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NeedsConsecutiveRegisters)
+HARDWARE_INTRINSIC(AdvSimd, VectorTableLookupExtension, 8, 3, true, {INS_tbx, INS_tbx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_HasRMWSemantics|HW_Flag_NeedsConsecutiveRegisters)
+HARDWARE_INTRINSIC(AdvSimd, Xor, -1, 2, true, {INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor, INS_eor}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd, ZeroExtendWideningLower, 8, 1, true, {INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_uxtl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd, ZeroExtendWideningUpper, 16, 1, true, {INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_uxtl2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AdvSimd 64-bit only Intrinsics
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Abs, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_fabs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsSaturate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsSaturateScalar, 8, 1, {INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanOrEqualScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_facge}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_facgt}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanOrEqualScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_facge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_facgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteDifference, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabd}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteDifferenceScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabd, INS_fabd}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Add, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fadd}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddAcross, -1, 1, {INS_addv, INS_addv, INS_addv, INS_addv, INS_addv, INS_addv, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddAcrossWidening, -1, 1, {INS_saddlv, INS_uaddlv, INS_saddlv, INS_uaddlv, INS_saddlv, INS_uaddlv, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddPairwise, 16, 2, {INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_faddp, INS_faddp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddPairwiseScalar, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addp, INS_addp, INS_faddp, INS_faddp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddSaturate, -1, 2, {INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, AddSaturateScalar, 8, 2, {INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_suqadd, INS_usqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Ceiling, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmeq, INS_cmeq, INS_invalid, INS_fcmeq}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareEqualScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmeq, INS_cmeq, INS_fcmeq, INS_fcmeq}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_invalid, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_invalid, INS_fcmge}, HW_Category_SIMD, HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanOrEqualScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_fcmge, INS_fcmge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_fcmgt, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_invalid, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_invalid, INS_fcmge}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanOrEqualScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_fcmge, INS_fcmge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_fcmgt, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareTest, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmtst, INS_cmtst, INS_invalid, INS_cmtst}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareTestScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmtst, INS_cmtst, INS_invalid, INS_cmtst}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDouble, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_fcvtl, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDoubleScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDoubleUpper, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtl2, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundAwayFromZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundAwayFromZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToEven, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToEvenScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToNegativeInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToNegativeInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToPositiveInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToPositiveInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleLower, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleRoundToOddLower, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtxn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleRoundToOddUpper, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtxn2, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleUpper, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtn2, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundAwayFromZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundAwayFromZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToEven, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToEvenScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToNegativeInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToNegativeInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToPositiveInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToPositiveInfinityScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToZeroScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Divide, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fdiv, INS_fdiv}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateSelectedScalarToVector128, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dup, INS_dup, INS_invalid, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateToVector64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mov, INS_mov, INS_invalid, INS_fmov}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateToVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dup, INS_dup, INS_invalid, INS_dup}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ExtractNarrowingSaturateScalar, 8, 1, {INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ExtractNarrowingSaturateUnsignedScalar, 8, 1, {INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Floor, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAdd, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddByScalar, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddBySelectedScalar, -1, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddScalarBySelectedScalar, 8, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtract, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractByScalar, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractBySelectedScalar, -1, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractScalarBySelectedScalar, 8, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, InsertSelectedScalar, -1, 4, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_NoJmpTableIMM|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadAndReplicateToVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ld1r, INS_ld1r, INS_invalid, INS_ld1r}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairScalarVector64, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ldp, INS_ldp, INS_invalid, INS_invalid, INS_ldp, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairScalarVector64NonTemporal, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ldnp, INS_ldnp, INS_invalid, INS_invalid, INS_ldnp, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector64, 8, 1, {INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector64NonTemporal, 8, 1, {INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector128, 16, 1, {INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector128NonTemporal, 16, 1, {INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Max, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmax}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxAcross, -1, 1, {INS_smaxv, INS_umaxv, INS_smaxv, INS_umaxv, INS_smaxv, INS_umaxv, INS_invalid, INS_invalid, INS_fmaxv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumber, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberAcross, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberPairwise, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmp, INS_fmaxnmp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberPairwiseScalar, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmp, INS_fmaxnmp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxPairwise, 16, 2, {INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_invalid, INS_invalid, INS_fmaxp, INS_fmaxp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxPairwiseScalar, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxp, INS_fmaxp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmax, INS_fmax}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Min, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmin}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinAcross, -1, 1, {INS_sminv, INS_uminv, INS_sminv, INS_uminv, INS_sminv, INS_uminv, INS_invalid, INS_invalid, INS_fminv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumber, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberAcross, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberPairwise, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmp, INS_fminnmp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberPairwiseScalar, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmp, INS_fminnmp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinPairwise, 16, 2, {INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_invalid, INS_invalid, INS_fminp, INS_fminp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinPairwiseScalar, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminp, INS_fminp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MinScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmin, INS_fmin}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyByScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyBySelectedScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingSaturateHighScalar, 8, 2, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingScalarBySelectedScalarSaturateHigh, 8, 3, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningAndAddSaturateScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningAndSubtractSaturateScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningSaturateScalar, 8, 2, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningSaturateScalarBySelectedScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningScalarBySelectedScalarAndAddSaturate, 8, 4, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningScalarBySelectedScalarAndSubtractSaturate, 8, 4, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtended, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedByScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedBySelectedScalar, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedScalarBySelectedScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyRoundedDoublingSaturateHighScalar, 8, 2, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarSaturateHigh, 8, 3, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyScalarBySelectedScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Negate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_fneg}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateSaturate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateSaturateScalar, 8, 1, {INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalEstimate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpe}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalEstimateScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpe, INS_frecpe}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalExponentScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpx, INS_frecpx}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootEstimate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrte}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootEstimateScalar, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrte, INS_frsqrte}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootStep, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootStepScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts, INS_frsqrts}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalStep, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps}, HW_Category_SIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalStepScalar, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps, INS_frecps}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ReverseElementBits, -1, 1, {INS_rbit, INS_rbit, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundAwayFromZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToNearest, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToNegativeInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToPositiveInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftArithmeticRoundedSaturateScalar, 8, 2, {INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftArithmeticSaturateScalar, 8, 2, {INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLeftLogicalSaturateScalar, 8, 2, {INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLeftLogicalSaturateUnsignedScalar, 8, 2, {INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLogicalRoundedSaturateScalar, 8, 2, {INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLogicalSaturateScalar, 8, 2, {INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticNarrowingSaturateScalar, 8, 2, {INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticNarrowingSaturateUnsignedScalar, 8, 2, {INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticRoundedNarrowingSaturateScalar, 8, 2, {INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedScalar, 8, 2, {INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightLogicalNarrowingSaturateScalar, 8, 2, {INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightLogicalRoundedNarrowingSaturateScalar, 8, 2, {INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Sqrt, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsqrt, INS_fsqrt}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePair, -1, 3, {INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairScalar, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_stp, INS_stp, INS_invalid, INS_invalid, INS_stp, INS_invalid}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairScalarNonTemporal, 8, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_stnp, INS_stnp, INS_invalid, INS_invalid, INS_stnp, INS_invalid}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairNonTemporal, -1, 3, {INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stp}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, Subtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsub}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, SubtractSaturateScalar, 8, 2, {INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, TransposeEven, -1, 2, {INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, TransposeOdd, -1, 2, {INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, UnzipEven, -1, 2, {INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, UnzipOdd, -1, 2, {INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, VectorTableLookup, 16, 2, {INS_tbl, INS_tbl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NeedsConsecutiveRegisters)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, VectorTableLookupExtension, 16, 3, {INS_tbx, INS_tbx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_HasRMWSemantics|HW_Flag_NeedsConsecutiveRegisters)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ZipHigh, -1, 2, {INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AdvSimd_Arm64, ZipLow, -1, 2, {INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Abs, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_fabs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsSaturate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsSaturateScalar, 8, 1, true, {INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_sqabs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_abs, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanOrEqualScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_facge}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareGreaterThanScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_facgt}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanOrEqualScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facge, INS_facge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteCompareLessThanScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_facgt, INS_facgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteDifference, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabd}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AbsoluteDifferenceScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fabd, INS_fabd}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Add, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fadd}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddAcross, -1, 1, true, {INS_addv, INS_addv, INS_addv, INS_addv, INS_addv, INS_addv, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddAcrossWidening, -1, 1, true, {INS_saddlv, INS_uaddlv, INS_saddlv, INS_uaddlv, INS_saddlv, INS_uaddlv, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddPairwise, 16, 2, true, {INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_addp, INS_faddp, INS_faddp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddPairwiseScalar, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addp, INS_addp, INS_faddp, INS_faddp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddSaturate, -1, 2, true, {INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_suqadd, INS_usqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, AddSaturateScalar, 8, 2, true, {INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_sqadd, INS_uqadd, INS_suqadd, INS_usqadd, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Ceiling, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareEqual, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmeq, INS_cmeq, INS_invalid, INS_fcmeq}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareEqualScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmeq, INS_cmeq, INS_fcmeq, INS_fcmeq}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThan, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_invalid, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanOrEqual, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_invalid, INS_fcmge}, HW_Category_SIMD, HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanOrEqualScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_fcmge, INS_fcmge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareGreaterThanScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_fcmgt, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThan, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_invalid, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanOrEqual, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_invalid, INS_fcmge}, HW_Category_SIMD, HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanOrEqualScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmge, INS_cmhs, INS_fcmge, INS_fcmge}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareLessThanScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmgt, INS_cmhi, INS_fcmgt, INS_fcmgt}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareTest, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmtst, INS_cmtst, INS_invalid, INS_cmtst}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, CompareTestScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmtst, INS_cmtst, INS_invalid, INS_cmtst}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDouble, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_fcvtl, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDoubleScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_scvtf, INS_ucvtf, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToDoubleUpper, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtl2, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundAwayFromZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundAwayFromZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtas}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToEven, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToEvenScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtns}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToNegativeInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToNegativeInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtms}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToPositiveInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToPositiveInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtps}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToInt64RoundToZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzs}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleLower, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleRoundToOddLower, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtxn, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleRoundToOddUpper, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtxn2, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToSingleUpper, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtn2, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundAwayFromZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundAwayFromZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtau}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToEven, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToEvenScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtnu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToNegativeInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToNegativeInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtmu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToPositiveInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToPositiveInfinityScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtpu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ConvertToUInt64RoundToZeroScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fcvtzu}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Divide, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fdiv, INS_fdiv}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateSelectedScalarToVector128, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dup, INS_dup, INS_invalid, INS_dup}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateToVector64, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mov, INS_mov, INS_invalid, INS_fmov}, HW_Category_SIMD, HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, DuplicateToVector128, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dup, INS_dup, INS_invalid, INS_dup}, HW_Category_SIMD, HW_Flag_SpecialCodeGen|HW_Flag_SupportsContainment)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ExtractNarrowingSaturateScalar, 8, 1, true, {INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_sqxtn, INS_uqxtn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ExtractNarrowingSaturateUnsignedScalar, 8, 1, true, {INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_sqxtun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Floor, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAdd, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddByScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddBySelectedScalar, -1, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplyAddScalarBySelectedScalar, 8, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmla, INS_fmla}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtract, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractByScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractBySelectedScalar, -1, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, FusedMultiplySubtractScalarBySelectedScalar, 8, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmls, INS_fmls}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, InsertSelectedScalar, -1, 4, true, {INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins, INS_ins}, HW_Category_SIMD, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_NoJmpTableIMM|HW_Flag_SIMDScalar|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadAndReplicateToVector128, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ld1r, INS_ld1r, INS_invalid, INS_ld1r}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairScalarVector64, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ldp, INS_ldp, INS_invalid, INS_invalid, INS_ldp, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairScalarVector64NonTemporal, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ldnp, INS_ldnp, INS_invalid, INS_invalid, INS_ldnp, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector64, 8, 1, true, {INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector64NonTemporal, 8, 1, true, {INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector128, 16, 1, true, {INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp, INS_ldp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, LoadPairVector128NonTemporal, 16, 1, true, {INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp, INS_ldnp}, HW_Category_MemoryLoad, HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiReg|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Max, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmax}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxAcross, -1, 1, true, {INS_smaxv, INS_umaxv, INS_smaxv, INS_umaxv, INS_smaxv, INS_umaxv, INS_invalid, INS_invalid, INS_fmaxv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumber, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnm}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberAcross, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberPairwise, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmp, INS_fmaxnmp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxNumberPairwiseScalar, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxnmp, INS_fmaxnmp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxPairwise, 16, 2, true, {INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_smaxp, INS_umaxp, INS_invalid, INS_invalid, INS_fmaxp, INS_fmaxp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxPairwiseScalar, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmaxp, INS_fmaxp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MaxScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmax, INS_fmax}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Min, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmin}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinAcross, -1, 1, true, {INS_sminv, INS_uminv, INS_sminv, INS_uminv, INS_sminv, INS_uminv, INS_invalid, INS_invalid, INS_fminv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumber, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnm}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberAcross, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmv, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberPairwise, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmp, INS_fminnmp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinNumberPairwiseScalar, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminnmp, INS_fminnmp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinPairwise, 16, 2, true, {INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_sminp, INS_uminp, INS_invalid, INS_invalid, INS_fminp, INS_fminp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinPairwiseScalar, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fminp, INS_fminp}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MinScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmin, INS_fmin}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyByScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyBySelectedScalar, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingSaturateHighScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingScalarBySelectedScalarSaturateHigh, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmulh, INS_invalid, INS_sqdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningAndAddSaturateScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningAndSubtractSaturateScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningSaturateScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningSaturateScalarBySelectedScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqdmull, INS_invalid, INS_sqdmull, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningScalarBySelectedScalarAndAddSaturate, 8, 4, true, {INS_invalid, INS_invalid, INS_sqdmlal, INS_invalid, INS_sqdmlal, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyDoublingWideningScalarBySelectedScalarAndSubtractSaturate, 8, 4, true, {INS_invalid, INS_invalid, INS_sqdmlsl, INS_invalid, INS_sqdmlsl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_BaseTypeFromSecondArg|HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtended, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedByScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedBySelectedScalar, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyExtendedScalarBySelectedScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmulx, INS_fmulx}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyRoundedDoublingSaturateHighScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarSaturateHigh, 8, 3, true, {INS_invalid, INS_invalid, INS_sqrdmulh, INS_invalid, INS_sqrdmulh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, MultiplyScalarBySelectedScalar, 8, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fmul}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Negate, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_fneg}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateSaturate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateSaturateScalar, 8, 1, true, {INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_sqneg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, NegateScalar, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_neg, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalEstimate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpe}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalEstimateScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpe, INS_frecpe}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalExponentScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecpx, INS_frecpx}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootEstimate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrte}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootEstimateScalar, 8, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrte, INS_frsqrte}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootStep, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalSquareRootStepScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frsqrts, INS_frsqrts}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalStep, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps}, HW_Category_SIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReciprocalStepScalar, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frecps, INS_frecps}, HW_Category_SIMD, HW_Flag_Commutative|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ReverseElementBits, -1, 1, true, {INS_rbit, INS_rbit, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundAwayFromZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frinta}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToNearest, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintn}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToNegativeInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintm}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToPositiveInfinity, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintp}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, RoundToZero, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_frintz}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftArithmeticRoundedSaturateScalar, 8, 2, true, {INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_sqrshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftArithmeticSaturateScalar, 8, 2, true, {INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_sqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLeftLogicalSaturateScalar, 8, 2, true, {INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_sqshl, INS_uqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLeftLogicalSaturateUnsignedScalar, 8, 2, true, {INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_sqshlu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftLeftByImmediate, HW_Flag_BaseTypeFromFirstArg|HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLogicalRoundedSaturateScalar, 8, 2, true, {INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_uqrshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftLogicalSaturateScalar, 8, 2, true, {INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_uqshl, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticNarrowingSaturateScalar, 8, 2, true, {INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_sqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticNarrowingSaturateUnsignedScalar, 8, 2, true, {INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_sqshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticRoundedNarrowingSaturateScalar, 8, 2, true, {INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_sqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightArithmeticRoundedNarrowingSaturateUnsignedScalar, 8, 2, true, {INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_sqrshrun, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightLogicalNarrowingSaturateScalar, 8, 2, true, {INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_uqshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ShiftRightLogicalRoundedNarrowingSaturateScalar, 8, 2, true, {INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_uqrshrn, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_ShiftRightByImmediate, HW_Flag_HasImmediateOperand|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Sqrt, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsqrt, INS_fsqrt}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePair, -1, 3, true, {INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp, INS_stp}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_stp, INS_stp, INS_invalid, INS_invalid, INS_stp, INS_invalid}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairScalarNonTemporal, 8, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_stnp, INS_stnp, INS_invalid, INS_invalid, INS_stnp, INS_invalid}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, StorePairNonTemporal, -1, 3, true, {INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stnp, INS_stp}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, Subtract, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_fsub}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, SubtractSaturateScalar, 8, 2, true, {INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_sqsub, INS_uqsub, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, TransposeEven, -1, 2, true, {INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1, INS_trn1}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, TransposeOdd, -1, 2, true, {INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2, INS_trn2}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, UnzipEven, -1, 2, true, {INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1, INS_uzp1}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, UnzipOdd, -1, 2, true, {INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2, INS_uzp2}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, VectorTableLookup, 16, 2, true, {INS_tbl, INS_tbl, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NeedsConsecutiveRegisters)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, VectorTableLookupExtension, 16, 3, true, {INS_tbx, INS_tbx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_HasRMWSemantics|HW_Flag_NeedsConsecutiveRegisters)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ZipHigh, -1, 2, true, {INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2, INS_zip2}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AdvSimd_Arm64, ZipLow, -1, 2, true, {INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1, INS_zip1}, HW_Category_SIMD, HW_Flag_NoFlag)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AES Intrinsics
-HARDWARE_INTRINSIC(Aes, Decrypt, 16, 2, {INS_invalid, INS_aesd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Aes, Encrypt, 16, 2, {INS_invalid, INS_aese, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Aes, InverseMixColumns, 16, 1, {INS_invalid, INS_aesimc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(Aes, MixColumns, 16, 1, {INS_invalid, INS_aesmc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(Aes, PolynomialMultiplyWideningLower, 8, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmull, INS_pmull, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Aes, PolynomialMultiplyWideningUpper, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmull2, INS_pmull2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Aes, Decrypt, 16, 2, false, {INS_invalid, INS_aesd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Aes, Encrypt, 16, 2, false, {INS_invalid, INS_aese, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Aes, InverseMixColumns, 16, 1, false, {INS_invalid, INS_aesimc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(Aes, MixColumns, 16, 1, false, {INS_invalid, INS_aesmc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(Aes, PolynomialMultiplyWideningLower, 8, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmull, INS_pmull, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Aes, PolynomialMultiplyWideningUpper, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmull2, INS_pmull2, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Base Intrinsics
-HARDWARE_INTRINSIC(ArmBase, LeadingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_clz, INS_clz, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(ArmBase, ReverseElementBits, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rbit, INS_rbit, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(ArmBase, Yield, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(ArmBase, LeadingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_clz, INS_clz, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase, ReverseElementBits, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rbit, INS_rbit, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase, Yield, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_SpecialSideEffect_Other)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Base 64-bit only Intrinsics
-HARDWARE_INTRINSIC(ArmBase_Arm64, LeadingSignCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cls, INS_invalid, INS_cls, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(ArmBase_Arm64, LeadingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_clz, INS_clz, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(ArmBase_Arm64, MultiplyHigh, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_smulh, INS_umulh, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(ArmBase_Arm64, ReverseElementBits, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rbit, INS_rbit, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase_Arm64, LeadingSignCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cls, INS_invalid, INS_cls, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase_Arm64, LeadingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_clz, INS_clz, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase_Arm64, MultiplyHigh, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_smulh, INS_umulh, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(ArmBase_Arm64, ReverseElementBits, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rbit, INS_rbit, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// CRC32 Intrinsics
-HARDWARE_INTRINSIC(Crc32, ComputeCrc32, 0, 2, {INS_invalid, INS_crc32b, INS_invalid, INS_crc32h, INS_invalid, INS_crc32w, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Crc32, ComputeCrc32C, 0, 2, {INS_invalid, INS_crc32cb, INS_invalid, INS_crc32ch, INS_invalid, INS_crc32cw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Crc32, ComputeCrc32, 0, 2, false, {INS_invalid, INS_crc32b, INS_invalid, INS_crc32h, INS_invalid, INS_crc32w, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Crc32, ComputeCrc32C, 0, 2, false, {INS_invalid, INS_crc32cb, INS_invalid, INS_crc32ch, INS_invalid, INS_crc32cw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// CRC32 64-bit only Intrinsics
-HARDWARE_INTRINSIC(Crc32_Arm64, ComputeCrc32, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32x, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Crc32_Arm64, ComputeCrc32C, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32cx, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Crc32_Arm64, ComputeCrc32, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32x, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Crc32_Arm64, ComputeCrc32C, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32cx, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// DP Intrinsics
-HARDWARE_INTRINSIC(Dp, DotProduct, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sdot, INS_udot, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Dp, DotProductBySelectedQuadruplet, -1, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sdot, INS_udot, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Dp, DotProduct, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sdot, INS_udot, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Dp, DotProductBySelectedQuadruplet, -1, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sdot, INS_udot, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// RDM Intrinsics
-HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingAndAddSaturateHigh, -1, 3, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingAndSubtractSaturateHigh, -1, 3, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingBySelectedScalarAndAddSaturateHigh, -1, 4, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingBySelectedScalarAndSubtractSaturateHigh, -1, 4, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingAndAddSaturateHigh, -1, 3, true, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingAndSubtractSaturateHigh, -1, 3, true, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingBySelectedScalarAndAddSaturateHigh, -1, 4, true, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Rdm, MultiplyRoundedDoublingBySelectedScalarAndSubtractSaturateHigh, -1, 4, true, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// RDM 64-bit only Intrinsics
-HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingAndAddSaturateHighScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingAndSubtractSaturateHighScalar, 8, 3, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarAndAddSaturateHigh, 8, 4, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarAndSubtractSaturateHigh, 8, 4, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingAndAddSaturateHighScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingAndSubtractSaturateHighScalar, 8, 3, true, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarAndAddSaturateHigh, 8, 4, true, {INS_invalid, INS_invalid, INS_sqrdmlah, INS_invalid, INS_sqrdmlah, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(Rdm_Arm64, MultiplyRoundedDoublingScalarBySelectedScalarAndSubtractSaturateHigh, 8, 4, true, {INS_invalid, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_sqrdmlsh, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDByIndexedElement, HW_Flag_HasImmediateOperand|HW_Flag_HasRMWSemantics|HW_Flag_SIMDScalar)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SHA1 Intrinsics
-HARDWARE_INTRINSIC(Sha1, FixedRotate, 8, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1h, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
-HARDWARE_INTRINSIC(Sha1, HashUpdateChoose, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1c, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha1, HashUpdateMajority, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1m, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha1, HashUpdateParity, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1p, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha1, ScheduleUpdate0, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1su0, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha1, ScheduleUpdate1, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1su1, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha1, FixedRotate, 8, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1h, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_SIMDScalar)
+HARDWARE_INTRINSIC(Sha1, HashUpdateChoose, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1c, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha1, HashUpdateMajority, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1m, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha1, HashUpdateParity, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1p, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha1, ScheduleUpdate0, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1su0, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha1, ScheduleUpdate1, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha1su1, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size Number of arguments Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SHA256 Intrinsics
-HARDWARE_INTRINSIC(Sha256, HashUpdate1, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256h, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha256, HashUpdate2, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256h2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha256, ScheduleUpdate0, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256su0, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
-HARDWARE_INTRINSIC(Sha256, ScheduleUpdate1, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256su1, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha256, HashUpdate1, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256h, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha256, HashUpdate2, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256h2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha256, ScheduleUpdate0, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256su0, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
+HARDWARE_INTRINSIC(Sha256, ScheduleUpdate1, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sha256su1, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMD, HW_Flag_HasRMWSemantics)
#endif // FEATURE_HW_INTRINSIC
diff --git a/src/coreclr/jit/hwintrinsiclistxarch.h b/src/coreclr/jit/hwintrinsiclistxarch.h
index b804d4a0edd5d3..e1649b2159c55f 100644
--- a/src/coreclr/jit/hwintrinsiclistxarch.h
+++ b/src/coreclr/jit/hwintrinsiclistxarch.h
@@ -17,1272 +17,1297 @@
4) SIMD intrinsics have a non-zero `SIMD size` field based-on that operate over `Vector128`(16) or `Vector256`(32)
5) Scalar intrinsics that operate over general purpose registers (e.g., Sse41.Crc32) have `SIMD size` with 0
6) Each intrinsic has a `NumArg` for number of parameters, and some intrinsics that are overloaded on multiple parameter numbers have this field with -1
- 7) Each intrinsic has 10 `instructions` fields that list the instructions should be generated based-on the base type
- 8) Each intrinsic has one category with type of `enum HWIntrinsicCategory`, please see the definition of HWIntrinsicCategory for details
- 9) Each intrinsic has one or more flags with type of `enum HWIntrinsicFlag`
+ 7) Each intrinsic has a `EncodesExtraTypeArg` when a type argument is needed to distinguish between multiple intrinsics
+ (This matches the value of Compiler::vnEncodesResultTypeForHWIntrinsic.)
+ 8) Each intrinsic has 10 `instructions` fields that list the instructions should be generated based-on the base type
+ 9) Each intrinsic has one category with type of `enum HWIntrinsicCategory`, please see the definition of HWIntrinsicCategory for details
+ 10) Each intrinsic has one or more flags with type of `enum HWIntrinsicFlag`
*/
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Vector128 Intrinsics
-HARDWARE_INTRINSIC(Vector128, Abs, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Add, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AndNot, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, As, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsByte, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsDouble, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsInt16, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsNInt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsNUInt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsSByte, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsSingle, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsUInt16, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsVector, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsVector2, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsd_simd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsVector3, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsVector4, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, AsVector128, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, BitwiseAnd, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, BitwiseOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Ceiling, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ConditionalSelect, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ConvertToDouble, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToSingle, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, ConvertToUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, Create, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, CreateScalar, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, CreateScalarUnsafe, 16, 1, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(Vector128, Divide, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Dot, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Equals, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, EqualsAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, EqualsAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ExtractMostSignificantBits, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Floor, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, get_AllBitsSet, 16, 0, {INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_cmpps, INS_cmpps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(Vector128, get_One, 16, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, get_Zero, 16, 0, {INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, GetElement, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_extractps, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, GreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, GreaterThanAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, LessThanAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAll, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAny, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Load, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadAligned, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadAlignedNonTemporal, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, LoadUnsafe, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Max, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Min, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Narrow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Negate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, OnesComplement, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Addition, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_BitwiseAnd, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_BitwiseOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_Division, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Equality, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_ExclusiveOr, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Inequality, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector128, op_LeftShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_OnesComplement, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_RightShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_Subtraction, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnaryNegation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnaryPlus, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, op_UnsignedRightShift, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftLeft, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftRightArithmetic, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ShiftRightLogical, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Shuffle, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Sqrt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Store, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreAligned, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreAlignedNonTemporal, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, StoreUnsafe, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Subtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, Sum, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector128, ToScalar, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(Vector128, ToVector256, 16, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(Vector128, ToVector256Unsafe, 16, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(Vector128, ToVector512, 16, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(Vector128, WidenLower, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, WidenUpper, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, WithElement, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector128, Xor, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Abs, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Add, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AndNot, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, As, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsByte, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsDouble, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsInt16, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsNInt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsNUInt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsSByte, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsSingle, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsUInt16, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsUInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsUInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsVector, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsVector2, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsd_simd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsVector3, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsVector4, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, AsVector128, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, BitwiseAnd, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, BitwiseOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Ceiling, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ConditionalSelect, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ConvertToDouble, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToSingle, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToUInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, ConvertToUInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, Create, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, CreateScalar, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, CreateScalarUnsafe, 16, 1, true, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(Vector128, Divide, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Dot, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Equals, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, EqualsAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, EqualsAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ExtractMostSignificantBits, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Floor, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, get_AllBitsSet, 16, 0, true, {INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_cmpps, INS_cmpps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(Vector128, get_One, 16, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, get_Zero, 16, 0, false, {INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, GetElement, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_extractps, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, GreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, GreaterThanAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, GreaterThanOrEqualAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, LessThanAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAll, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LessThanOrEqualAny, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Load, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadAligned, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadAlignedNonTemporal, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, LoadUnsafe, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Max, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Min, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Narrow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Negate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, OnesComplement, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Addition, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_BitwiseAnd, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_BitwiseOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_Division, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Equality, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_ExclusiveOr, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Inequality, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector128, op_LeftShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_OnesComplement, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_RightShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_Subtraction, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnaryNegation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnaryPlus, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, op_UnsignedRightShift, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftLeft, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftRightArithmetic, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ShiftRightLogical, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Shuffle, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Sqrt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Store, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreAligned, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreAlignedNonTemporal, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, StoreUnsafe, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Subtract, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, Sum, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector128, ToScalar, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(Vector128, ToVector256, 16, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(Vector128, ToVector256Unsafe, 16, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(Vector128, ToVector512, 16, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(Vector128, WidenLower, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, WidenUpper, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, WithElement, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector128, Xor, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Vector256 Intrinsics
-HARDWARE_INTRINSIC(Vector256, Abs, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Add, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, AndNot, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, As, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsByte, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsDouble, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsInt16, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsInt64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsNInt, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsNUInt, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsSByte, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsSingle, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsUInt16, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsUInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsUInt64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsVector, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, AsVector256, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, BitwiseAnd, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, BitwiseOr, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Ceiling, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, ConditionalSelect, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, ConvertToDouble, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, ConvertToInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, ConvertToInt64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, ConvertToSingle, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, ConvertToUInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, ConvertToUInt64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, Create, 32, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, CreateScalar, 32, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, CreateScalarUnsafe, 32, 1, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Divide, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Dot, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Equals, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector256, EqualsAll, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, EqualsAny, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, ExtractMostSignificantBits, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Floor, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, get_AllBitsSet, 32, 0, {INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_cmpps, INS_cmpps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_AvxOnlyCompatible|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(Vector256, get_One, 32, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, get_Zero, 32, 0, {INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, GetElement, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, GetLower, 32, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, GetUpper, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, GreaterThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector256, GreaterThanAll, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, GreaterThanAny, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqualAll, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqualAny, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, LessThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector256, LessThanAll, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, LessThanAny, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, LessThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector256, LessThanOrEqualAll, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, LessThanOrEqualAny, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Load, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, LoadAligned, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, LoadAlignedNonTemporal, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, LoadUnsafe, 32, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Max, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Min, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Multiply, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Narrow, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Negate, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, OnesComplement, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_Addition, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_BitwiseAnd, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_BitwiseOr, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_Division, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_Equality, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector256, op_ExclusiveOr, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_Inequality, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector256, op_LeftShift, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_Multiply, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_OnesComplement, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_RightShift, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_Subtraction, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_UnaryNegation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, op_UnaryPlus, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, op_UnsignedRightShift, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, ShiftLeft, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, ShiftRightArithmetic, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, ShiftRightLogical, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Shuffle, 32, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Sqrt, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Store, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, StoreAligned, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, StoreAlignedNonTemporal, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, StoreUnsafe, 32, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Subtract, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, Sum, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector256, ToScalar, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, ToVector512, 32, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, ToVector512Unsafe, 32, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, WidenLower, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, WidenUpper, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector256, WithElement, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, WithLower, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, WithUpper, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
-HARDWARE_INTRINSIC(Vector256, Xor, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Abs, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Add, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, AndNot, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, As, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsByte, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsDouble, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsInt16, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsInt64, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsNInt, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsNUInt, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsSByte, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsSingle, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsUInt16, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsUInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsUInt64, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsVector, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, AsVector256, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, BitwiseAnd, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, BitwiseOr, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Ceiling, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, ConditionalSelect, 32, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, ConvertToDouble, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, ConvertToInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, ConvertToInt64, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, ConvertToSingle, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, ConvertToUInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, ConvertToUInt64, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, Create, 32, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, CreateScalar, 32, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, CreateScalarUnsafe, 32, 1, true, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Divide, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Dot, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Equals, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector256, EqualsAll, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, EqualsAny, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, ExtractMostSignificantBits, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Floor, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, get_AllBitsSet, 32, 0, true, {INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqd, INS_cmpps, INS_cmpps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_AvxOnlyCompatible|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(Vector256, get_One, 32, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, get_Zero, 32, 0, false, {INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps, INS_xorps}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, GetElement, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, GetLower, 32, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, GetUpper, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, GreaterThan, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector256, GreaterThanAll, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, GreaterThanAny, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqual, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqualAll, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, GreaterThanOrEqualAny, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, LessThan, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector256, LessThanAll, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, LessThanAny, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, LessThanOrEqual, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector256, LessThanOrEqualAll, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, LessThanOrEqualAny, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Load, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, LoadAligned, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, LoadAlignedNonTemporal, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, LoadUnsafe, 32, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Max, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Min, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Multiply, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Narrow, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Negate, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, OnesComplement, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_Addition, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_BitwiseAnd, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_BitwiseOr, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_Division, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_Equality, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector256, op_ExclusiveOr, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_Inequality, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector256, op_LeftShift, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_Multiply, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_OnesComplement, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_RightShift, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_Subtraction, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_UnaryNegation, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, op_UnaryPlus, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, op_UnsignedRightShift, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, ShiftLeft, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, ShiftRightArithmetic, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, ShiftRightLogical, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Shuffle, 32, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Sqrt, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Store, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, StoreAligned, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, StoreAlignedNonTemporal, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, StoreUnsafe, 32, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Subtract, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, Sum, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector256, ToScalar, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, ToVector512, 32, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, ToVector512Unsafe, 32, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, WidenLower, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, WidenUpper, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector256, WithElement, 32, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, WithLower, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, WithUpper, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
+HARDWARE_INTRINSIC(Vector256, Xor, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_AvxOnlyCompatible)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Vector512 Intrinsics
-HARDWARE_INTRINSIC(Vector512, Abs, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Add, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AndNot, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, As, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsByte, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsDouble, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsInt16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsInt32, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsInt64, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsNInt, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsNUInt, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsSByte, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsSingle, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsUInt16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsUInt32, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsUInt64, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsVector, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, AsVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, BitwiseAnd, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ConditionalSelect, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, BitwiseOr, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Ceiling, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Create, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, CreateScalar, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, CreateScalarUnsafe, 64, 1, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector512, ConvertToSingle, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, ConvertToInt32, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, Divide, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Equals, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, EqualsAll, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, EqualsAny, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ExtractMostSignificantBits, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Floor, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, get_AllBitsSet, 64, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, get_One, 64, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, get_Zero, 64, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, GetElement, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, GetLower, 64, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(Vector512, GetLower128, 64, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, GetUpper, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, GreaterThan, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, GreaterThanAll, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, GreaterThanAny, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqual, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqualAll, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqualAny, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LessThan, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, LessThanAll, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LessThanAny, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LessThanOrEqual, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(Vector512, LessThanOrEqualAll, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LessThanOrEqualAny, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Load, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LoadAligned, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LoadAlignedNonTemporal, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, LoadUnsafe, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Max, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Min, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Multiply, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Narrow, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Negate, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, OnesComplement, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_Addition, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_BitwiseAnd, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector512, op_BitwiseOr, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector512, op_Division, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_Equality, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector512, op_ExclusiveOr, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_Inequality, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
-HARDWARE_INTRINSIC(Vector512, op_LeftShift, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_Multiply, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_OnesComplement, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_RightShift, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_Subtraction, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_UnaryNegation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_UnaryPlus, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, op_UnsignedRightShift, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ShiftLeft, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ShiftRightArithmetic, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ShiftRightLogical, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Shuffle, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Sqrt, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Store, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, StoreAligned, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, StoreAlignedNonTemporal, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, StoreUnsafe, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Subtract, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, ToScalar, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, WidenLower, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, WidenUpper, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, WithElement, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(Vector512, WithLower, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, WithUpper, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(Vector512, Xor, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Abs, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Add, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AndNot, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, As, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsByte, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsDouble, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsInt16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsInt32, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsInt64, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsNInt, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsNUInt, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsSByte, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsSingle, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsUInt16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsUInt32, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsUInt64, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsVector, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, AsVector512, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, BitwiseAnd, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ConditionalSelect, 64, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, BitwiseOr, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Ceiling, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Create, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, CreateScalar, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, CreateScalarUnsafe, 64, 1, true, {INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movd, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector512, ConvertToSingle, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, ConvertToInt32, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, Divide, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Equals, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, EqualsAll, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, EqualsAny, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ExtractMostSignificantBits, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Floor, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, get_AllBitsSet, 64, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, get_One, 64, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, get_Zero, 64, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, GetElement, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, GetLower, 64, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(Vector512, GetLower128, 64, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, GetUpper, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, GreaterThan, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, GreaterThanAll, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, GreaterThanAny, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqual, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqualAll, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, GreaterThanOrEqualAny, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LessThan, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, LessThanAll, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LessThanAny, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LessThanOrEqual, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(Vector512, LessThanOrEqualAll, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LessThanOrEqualAny, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Load, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LoadAligned, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LoadAlignedNonTemporal, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, LoadUnsafe, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Max, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Min, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Multiply, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Narrow, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Negate, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, OnesComplement, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_Addition, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_BitwiseAnd, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector512, op_BitwiseOr, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector512, op_Division, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_Equality, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector512, op_ExclusiveOr, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_Inequality, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_Commutative)
+HARDWARE_INTRINSIC(Vector512, op_LeftShift, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_Multiply, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_OnesComplement, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_RightShift, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_Subtraction, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_UnaryNegation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_UnaryPlus, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, op_UnsignedRightShift, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ShiftLeft, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ShiftRightArithmetic, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ShiftRightLogical, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Shuffle, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Sqrt, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Store, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, StoreAligned, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, StoreAlignedNonTemporal, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, StoreUnsafe, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Subtract, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, ToScalar, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, WidenLower, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, WidenUpper, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, WithElement, 64, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(Vector512, WithLower, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, WithUpper, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(Vector512, Xor, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// X86Base Intrinsics
-HARDWARE_INTRINSIC(X86Base, BitScanForward, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsf, INS_bsf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(X86Base, BitScanReverse, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsr, INS_bsr, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(X86Base, DivRem, 0, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_idiv, INS_div, INS_idiv, INS_div, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg|HW_Flag_MultiReg|HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(X86Base, Pause, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(X86Base, BitScanForward, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsf, INS_bsf, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(X86Base, BitScanReverse, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsr, INS_bsr, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(X86Base, DivRem, 0, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_idiv, INS_div, INS_idiv, INS_div, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg|HW_Flag_MultiReg|HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(X86Base, Pause, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// X86Base 64-bit-only Intrinsics
-HARDWARE_INTRINSIC(X86Base_X64, BitScanForward, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsf, INS_bsf, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(X86Base_X64, BitScanReverse, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsr, INS_bsr, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(X86Base_X64, DivRem, 0, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_idiv, INS_div, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg|HW_Flag_MultiReg|HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(X86Base_X64, BitScanForward, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsf, INS_bsf, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(X86Base_X64, BitScanReverse, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bsr, INS_bsr, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(X86Base_X64, DivRem, 0, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_idiv, INS_div, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg|HW_Flag_MultiReg|HW_Flag_SpecialImport|HW_Flag_SpecialCodeGen|HW_Flag_RmwIntrinsic)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE Intrinsics
-HARDWARE_INTRINSIC(SSE, Add, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE, AddScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, And, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE, AndNot, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE, CompareEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrderedNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, CompareNotGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarNotGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareNotGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarNotGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareNotLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarNotLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareNotLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarNotLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareOrdered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarOrdered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareUnordered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, CompareScalarUnordered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, ConvertToInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, ConvertScalarToVector128Single, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2ss32, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, ConvertToInt32WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, Divide, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE, DivideScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, LoadAlignedVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movaps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, LoadHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, LoadLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, LoadScalarVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, LoadVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(SSE, Max, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(SSE, MaxScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, Min, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(SSE, MinScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, MoveHighToLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhlps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment)
-HARDWARE_INTRINSIC(SSE, MoveLowToHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlhps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment)
-HARDWARE_INTRINSIC(SSE, MoveMask, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, MoveScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoContainment)
-HARDWARE_INTRINSIC(SSE, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE, MultiplyScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, Or, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE, Prefetch0, 0, 1, {INS_invalid, INS_prefetcht0, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
-HARDWARE_INTRINSIC(SSE, Prefetch1, 0, 1, {INS_invalid, INS_prefetcht1, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
-HARDWARE_INTRINSIC(SSE, Prefetch2, 0, 1, {INS_invalid, INS_prefetcht2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
-HARDWARE_INTRINSIC(SSE, PrefetchNonTemporal, 0, 1, {INS_invalid, INS_prefetchnta, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
-HARDWARE_INTRINSIC(SSE, Reciprocal, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, ReciprocalScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, ReciprocalSqrt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, ReciprocalSqrtScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE, Shuffle, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shufps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE, Sqrt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, SqrtScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, Store, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(SSE, StoreAligned, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movaps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE, StoreAlignedNonTemporal, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movntps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE, StoreFence, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
-HARDWARE_INTRINSIC(SSE, StoreHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE, StoreLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE, StoreScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE, Subtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE, SubtractScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE, UnpackHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpckhps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE, UnpackLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpcklps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE, Xor, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE, Add, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_EmbBroadcastCompatible)
+HARDWARE_INTRINSIC(SSE, AddScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, And, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE, AndNot, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE, CompareEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrderedNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnorderedNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, CompareNotGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarNotGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareNotGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarNotGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareNotLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarNotLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareNotLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarNotLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareOrdered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarOrdered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareUnordered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, CompareScalarUnordered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, ConvertToInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, ConvertScalarToVector128Single, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2ss32, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, ConvertToInt32WithTruncation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, Divide, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE, DivideScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, LoadAlignedVector128, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movaps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, LoadHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, LoadLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlps, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, LoadScalarVector128, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, LoadVector128, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(SSE, Max, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(SSE, MaxScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, Min, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(SSE, MinScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, MoveHighToLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhlps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment)
+HARDWARE_INTRINSIC(SSE, MoveLowToHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlhps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment)
+HARDWARE_INTRINSIC(SSE, MoveMask, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, MoveScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoContainment)
+HARDWARE_INTRINSIC(SSE, Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE, MultiplyScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, Or, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE, Prefetch0, 0, 1, false, {INS_invalid, INS_prefetcht0, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(SSE, Prefetch1, 0, 1, false, {INS_invalid, INS_prefetcht1, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(SSE, Prefetch2, 0, 1, false, {INS_invalid, INS_prefetcht2, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(SSE, PrefetchNonTemporal, 0, 1, false, {INS_invalid, INS_prefetchnta, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Other)
+HARDWARE_INTRINSIC(SSE, Reciprocal, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, ReciprocalScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, ReciprocalSqrt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, ReciprocalSqrtScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, Shuffle, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shufps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE, Sqrt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, SqrtScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, Store, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movups, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(SSE, StoreAligned, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movaps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE, StoreAlignedNonTemporal, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movntps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE, StoreFence, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
+HARDWARE_INTRINSIC(SSE, StoreHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE, StoreLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlps, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE, StoreScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movss, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE, Subtract, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE, SubtractScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE, UnpackHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpckhps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE, UnpackLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpcklps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE, Xor, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE 64-bit-only Intrinsics
-HARDWARE_INTRINSIC(SSE_X64, ConvertToInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(SSE_X64, ConvertToInt64WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(SSE_X64, ConvertScalarToVector128Single, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2ss64, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(SSE_X64, ConvertToInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(SSE_X64, ConvertToInt64WithTruncation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttss2si, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(SSE_X64, ConvertScalarToVector128Single, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2ss64, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE2 Intrinsics
-HARDWARE_INTRINSIC(SSE2, Add, 16, 2, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_invalid, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, AddSaturate, 16, 2, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, AddScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, And, 16, 2, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_invalid, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, AndNot, 16, 2, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_invalid, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, Average, 16, 2, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, CompareEqual, 16, 2, {INS_pcmpeqb, INS_pcmpeqb, INS_pcmpeqw, INS_pcmpeqw, INS_pcmpeqd, INS_pcmpeqd, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareGreaterThan, 16, 2, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareLessThan, 16, 2, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarLessThan, 16, 2, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_MultiIns|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedNotEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_MultiIns|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareNotGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarNotGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareNotGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarNotGreaterThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareNotLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareScalarNotLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareNotLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarNotLessThanOrEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareOrdered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarOrdered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, CompareUnordered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE2, CompareScalarUnordered, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToInt32WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToVector128Double, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Double, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2sd32, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2sd, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, ConvertToVector128Int32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_cvtpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Int32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToVector128Int32WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_cvttpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertToVector128Single, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Single, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsd2ss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128UInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, Divide, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, DivideScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, Extract, 16, 2, {INS_invalid, INS_invalid, INS_pextrw, INS_pextrw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, Insert, 16, 3, {INS_invalid, INS_invalid, INS_pinsrw, INS_pinsrw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, LoadAlignedVector128, 16, 1, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_invalid, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, LoadFence, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
-HARDWARE_INTRINSIC(SSE2, LoadHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhpd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, LoadLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlpd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, LoadScalarVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_movd, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, LoadVector128, 16, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_invalid, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(SSE2, MaskMove, 16, 3, {INS_maskmovdqu, INS_maskmovdqu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, Max, 16, 2, {INS_invalid, INS_pmaxub, INS_pmaxsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(SSE2, MemoryFence, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
-HARDWARE_INTRINSIC(SSE2, MaxScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, Min, 16, 2, {INS_invalid, INS_pminub, INS_pminsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(SSE2, MinScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, MoveMask, 16, 1, {INS_pmovmskb, INS_pmovmskb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskpd}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE2, MoveScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_NoContainment)
-HARDWARE_INTRINSIC(SSE2, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuludq, INS_invalid, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, MultiplyHigh, 16, 2, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, MultiplyAddAdjacent, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, MultiplyLow, 16, 2, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, MultiplyScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, Or, 16, 2, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_invalid, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE2, PackSignedSaturate, 16, 2, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, PackUnsignedSaturate, 16, 2, {INS_invalid, INS_packuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, SumAbsoluteDifferences, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, ShiftLeftLogical, 16, 2, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShiftLeftLogical128BitLane, 16, 2, {INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShiftRightArithmetic, 16, 2, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_psrad, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShiftRightLogical, 16, 2, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShiftRightLogical128BitLane, 16, 2, {INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, Shuffle, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_invalid, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShuffleHigh, 16, 2, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, ShuffleLow, 16, 2, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE2, Sqrt, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, SqrtScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, Store, 16, 2, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_invalid, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(SSE2, StoreAligned, 16, 2, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_invalid, INS_movapd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, StoreAlignedNonTemporal, 16, 2, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_invalid, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, StoreHigh, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, StoreLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, StoreNonTemporal, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movnti, INS_movnti, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, StoreScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_movd, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2, Subtract, 16, 2, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_invalid, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, SubtractSaturate, 16, 2, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, SubtractScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE2, UnpackHigh, 16, 2, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_invalid, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, UnpackLow, 16, 2, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_invalid, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE2, Xor, 16, 2, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_invalid, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, Add, 16, 2, true, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_invalid, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_EmbBroadcastCompatible)
+HARDWARE_INTRINSIC(SSE2, AddSaturate, 16, 2, true, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, AddScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, And, 16, 2, true, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_invalid, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, AndNot, 16, 2, true, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_invalid, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, Average, 16, 2, true, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, CompareEqual, 16, 2, true, {INS_pcmpeqb, INS_pcmpeqb, INS_pcmpeqw, INS_pcmpeqw, INS_pcmpeqd, INS_pcmpeqd, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareGreaterThan, 16, 2, true, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareLessThan, 16, 2, true, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarLessThan, 16, 2, true, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrderedNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_MultiIns|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnorderedNotEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_Commutative|HW_Flag_MultiIns|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareNotGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarNotGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareNotGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarNotGreaterThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_SpecialImport|HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareNotLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareScalarNotLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareNotLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarNotLessThanOrEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareOrdered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarOrdered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, CompareUnordered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE2, CompareScalarUnordered, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToInt32, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToInt32WithTruncation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToUInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToVector128Double, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Double, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2sd32, INS_invalid, INS_invalid, INS_invalid, INS_cvtss2sd, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, ConvertToVector128Int32, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_cvtpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Int32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToVector128Int32WithTruncation, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_cvttpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertToVector128Single, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128Single, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsd2ss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, ConvertScalarToVector128UInt32, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, Divide, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, DivideScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, Extract, 16, 2, false, {INS_invalid, INS_invalid, INS_pextrw, INS_pextrw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, Insert, 16, 3, false, {INS_invalid, INS_invalid, INS_pinsrw, INS_pinsrw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, LoadAlignedVector128, 16, 1, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_invalid, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, LoadFence, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
+HARDWARE_INTRINSIC(SSE2, LoadHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhpd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, LoadLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlpd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, LoadScalarVector128, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_movd, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, LoadVector128, 16, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_invalid, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(SSE2, MaskMove, 16, 3, false, {INS_maskmovdqu, INS_maskmovdqu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, Max, 16, 2, true, {INS_invalid, INS_pmaxub, INS_pmaxsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(SSE2, MemoryFence, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
+HARDWARE_INTRINSIC(SSE2, MaxScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, Min, 16, 2, true, {INS_invalid, INS_pminub, INS_pminsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(SSE2, MinScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, MoveMask, 16, 1, true, {INS_pmovmskb, INS_pmovmskb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskpd}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE2, MoveScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_SIMDScalar, HW_Flag_NoContainment)
+HARDWARE_INTRINSIC(SSE2, Multiply, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuludq, INS_invalid, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, MultiplyHigh, 16, 2, true, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, MultiplyAddAdjacent, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, MultiplyLow, 16, 2, false, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, MultiplyScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, Or, 16, 2, true, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_invalid, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE2, PackSignedSaturate, 16, 2, true, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, PackUnsignedSaturate, 16, 2, false, {INS_invalid, INS_packuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, SumAbsoluteDifferences, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, ShiftLeftLogical, 16, 2, true, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShiftLeftLogical128BitLane, 16, 2, false, {INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShiftRightArithmetic, 16, 2, true, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_psrad, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShiftRightLogical, 16, 2, true, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShiftRightLogical128BitLane, 16, 2, false, {INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, Shuffle, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_invalid, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShuffleHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, ShuffleLow, 16, 2, false, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE2, Sqrt, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, SqrtScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, Store, 16, 2, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_invalid, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(SSE2, StoreAligned, 16, 2, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_invalid, INS_movapd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, StoreAlignedNonTemporal, 16, 2, true, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_invalid, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, StoreHigh, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movhpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, StoreLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movlpd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, StoreNonTemporal, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movnti, INS_movnti, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, StoreScalar, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_movd, INS_movq, INS_movq, INS_invalid, INS_movsd_simd}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2, Subtract, 16, 2, true, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_invalid, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, SubtractSaturate, 16, 2, true, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, SubtractScalar, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE2, UnpackHigh, 16, 2, true, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_invalid, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, UnpackLow, 16, 2, true, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_invalid, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE2, Xor, 16, 2, true, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_invalid, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE2 64-bit-only Intrinsics
-HARDWARE_INTRINSIC(SSE2_X64, ConvertToInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_cvtsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2_X64, ConvertToInt64WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2_X64, ConvertToUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128Double, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2sd64, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128Int64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128UInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(SSE2_X64, StoreNonTemporal, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movnti, INS_movnti, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertToInt64, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_cvtsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertToInt64WithTruncation, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttsd2si}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertToUInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128Double, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtsi2sd64, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128Int64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(SSE2_X64, ConvertScalarToVector128UInt64, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(SSE2_X64, StoreNonTemporal, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movnti, INS_movnti, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoRMWSemantics|HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromSecondArg)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE3 Intrinsics
-HARDWARE_INTRINSIC(SSE3, AddSubtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsubps, INS_addsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE3, HorizontalAdd, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_haddps, INS_haddpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE3, HorizontalSubtract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_hsubps, INS_hsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE3, LoadAndDuplicateToVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_lddqu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movddup}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE3, LoadDquVector128, 16, 1, {INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE3, MoveAndDuplicate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE3, MoveHighAndDuplicate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE3, MoveLowAndDuplicate, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE3, AddSubtract, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsubps, INS_addsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE3, HorizontalAdd, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_haddps, INS_haddpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE3, HorizontalSubtract, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_hsubps, INS_hsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE3, LoadAndDuplicateToVector128, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_lddqu, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movddup}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE3, LoadDquVector128, 16, 1, false, {INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE3, MoveAndDuplicate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE3, MoveHighAndDuplicate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE3, MoveLowAndDuplicate, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSSE3 Intrinsics
-HARDWARE_INTRINSIC(SSSE3, Abs, 16, 1, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_pabsd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(SSSE3, AlignRight, 16, 3, {INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSSE3, HorizontalAdd, 16, 2, {INS_invalid, INS_invalid, INS_phaddw, INS_phaddw, INS_phaddd, INS_phaddd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSSE3, HorizontalAddSaturate, 16, 2, {INS_invalid, INS_invalid, INS_phaddsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSSE3, HorizontalSubtract, 16, 2, {INS_invalid, INS_invalid, INS_phsubw, INS_invalid, INS_phsubd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSSE3, HorizontalSubtractSaturate, 16, 2, {INS_invalid, INS_invalid, INS_phsubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSSE3, MultiplyAddAdjacent, 16, 2, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSSE3, MultiplyHighRoundScale, 16, 2, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSSE3, Shuffle, 16, 2, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSSE3, Sign, 16, 2, {INS_psignb, INS_invalid, INS_psignw, INS_invalid, INS_psignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSSE3, Abs, 16, 1, true, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_pabsd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(SSSE3, AlignRight, 16, 3, false, {INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSSE3, HorizontalAdd, 16, 2, true, {INS_invalid, INS_invalid, INS_phaddw, INS_phaddw, INS_phaddd, INS_phaddd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSSE3, HorizontalAddSaturate, 16, 2, false, {INS_invalid, INS_invalid, INS_phaddsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSSE3, HorizontalSubtract, 16, 2, true, {INS_invalid, INS_invalid, INS_phsubw, INS_invalid, INS_phsubd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSSE3, HorizontalSubtractSaturate, 16, 2, false, {INS_invalid, INS_invalid, INS_phsubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSSE3, MultiplyAddAdjacent, 16, 2, false, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSSE3, MultiplyHighRoundScale, 16, 2, false, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSSE3, Shuffle, 16, 2, false, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSSE3, Sign, 16, 2, true, {INS_psignb, INS_invalid, INS_psignw, INS_invalid, INS_psignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE41 Intrinsics
-HARDWARE_INTRINSIC(SSE41, Blend, 16, 3, {INS_invalid, INS_invalid, INS_pblendw, INS_pblendw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blendps, INS_blendpd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, BlendVariable, 16, 3, {INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_blendvps, INS_blendvpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, Ceiling, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, CeilingScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, CompareEqual, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpeqq, INS_pcmpeqq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int16, 16, 1, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int32, 16, 1, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int64, 16, 1, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(SSE41, DotProduct, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dpps, INS_dppd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, Extract, 16, 2, {INS_pextrb, INS_pextrb, INS_invalid, INS_invalid, INS_pextrd, INS_pextrd, INS_invalid, INS_invalid, INS_extractps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiIns|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, Floor, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, FloorScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, Insert, 16, 3, {INS_pinsrb, INS_pinsrb, INS_invalid, INS_invalid, INS_pinsrd, INS_pinsrd, INS_invalid, INS_invalid, INS_insertps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(SSE41, LoadAlignedVector128NonTemporal, 16, 1, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, Max, 16, 2, {INS_pmaxsb, INS_invalid, INS_invalid, INS_pmaxuw, INS_pmaxsd, INS_pmaxud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE41, Min, 16, 2, {INS_pminsb, INS_invalid, INS_invalid, INS_pminuw, INS_pminsd, INS_pminud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE41, MinHorizontal, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_phminposuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, MultipleSumAbsoluteDifferences, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_mpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, Multiply, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(SSE41, MultiplyLow, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE41, PackUnsignedSaturate, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(SSE41, RoundCurrentDirection, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, RoundCurrentDirectionScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, RoundToNearestInteger, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, RoundToNearestIntegerScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, RoundToNegativeInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, RoundToNegativeInfinityScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, RoundToPositiveInfinity, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, RoundToPositiveInfinityScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, RoundToZero, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, RoundToZeroScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(SSE41, TestC, 16, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, TestNotZAndNotC, 16, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(SSE41, TestZ, 16, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, Blend, 16, 3, true, {INS_invalid, INS_invalid, INS_pblendw, INS_pblendw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blendps, INS_blendpd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, BlendVariable, 16, 3, true, {INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_pblendvb, INS_blendvps, INS_blendvpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, Ceiling, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, CeilingScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, CompareEqual, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpeqq, INS_pcmpeqq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int16, 16, 1, true, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int32, 16, 1, true, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(SSE41, ConvertToVector128Int64, 16, 1, true, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoRMWSemantics|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(SSE41, DotProduct, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dpps, INS_dppd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, Extract, 16, 2, true, {INS_pextrb, INS_pextrb, INS_invalid, INS_invalid, INS_pextrd, INS_pextrd, INS_invalid, INS_invalid, INS_extractps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiIns|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, Floor, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, FloorScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, Insert, 16, 3, true, {INS_pinsrb, INS_pinsrb, INS_invalid, INS_invalid, INS_pinsrd, INS_pinsrd, INS_invalid, INS_invalid, INS_insertps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE41, LoadAlignedVector128NonTemporal, 16, 1, false, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, Max, 16, 2, true, {INS_pmaxsb, INS_invalid, INS_invalid, INS_pmaxuw, INS_pmaxsd, INS_pmaxud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE41, Min, 16, 2, true, {INS_pminsb, INS_invalid, INS_invalid, INS_pminuw, INS_pminsd, INS_pminud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE41, MinHorizontal, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_phminposuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, MultipleSumAbsoluteDifferences, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_mpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, Multiply, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(SSE41, MultiplyLow, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE41, PackUnsignedSaturate, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(SSE41, RoundCurrentDirection, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, RoundCurrentDirectionScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, RoundToNearestInteger, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, RoundToNearestIntegerScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, RoundToNegativeInfinity, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, RoundToNegativeInfinityScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, RoundToPositiveInfinity, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, RoundToPositiveInfinityScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, RoundToZero, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, RoundToZeroScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundss, INS_roundsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(SSE41, TestC, 16, 2, false, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, TestNotZAndNotC, 16, 2, false, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE41, TestZ, 16, 2, false, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE41 64-bit-only Intrinsics
-HARDWARE_INTRINSIC(SSE41_X64, Extract, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pextrq, INS_pextrq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiIns|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41_X64, Insert, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pinsrq, INS_pinsrq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(SSE41_X64, Extract, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pextrq, INS_pextrq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MultiIns|HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41_X64, Insert, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pinsrq, INS_pinsrq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE42 Intrinsics
-HARDWARE_INTRINSIC(SSE42, Crc32, 0, 2, {INS_invalid, INS_crc32, INS_invalid, INS_crc32, INS_invalid, INS_crc32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(SSE42, CompareGreaterThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(SSE42, CompareLessThan, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE42, Crc32, 0, 2, false, {INS_invalid, INS_crc32, INS_invalid, INS_crc32, INS_invalid, INS_crc32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(SSE42, CompareGreaterThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(SSE42, CompareLessThan, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// SSE42 Intrinsics
-HARDWARE_INTRINSIC(SSE42_X64, Crc32, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(SSE42_X64, Crc32, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_crc32, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_RmwIntrinsic)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX Intrinsics
-HARDWARE_INTRINSIC(AVX, Add, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addps, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX, AddSubtract, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsubps, INS_addsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, And, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX, AndNot, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Blend, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blendps, INS_blendpd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, BlendVariable, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vblendvps, INS_vblendvpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, Ceiling, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, BroadcastScalarToVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastss, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, BroadcastScalarToVector256, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, BroadcastVector128ToVector256, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastf128, INS_vbroadcastf128}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Compare, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_IMM, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareGreaterThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareGreaterThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareLessThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareLessThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareNotEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareNotGreaterThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareNotGreaterThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareNotLessThan, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareNotLessThanOrEqual, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareOrdered, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareUnordered, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
-HARDWARE_INTRINSIC(AVX, CompareScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_cmpsd}, HW_Category_IMM, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, ConvertToVector128Int32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, ConvertToVector128Single, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX, ConvertToVector256Int32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, ConvertToVector256Single, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX, ConvertToVector256Double, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX, ConvertToVector128Int32WithTruncation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttpd2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, ConvertToVector256Int32WithTruncation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX, Divide, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, DotProduct, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dpps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, DuplicateEvenIndexed, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, DuplicateOddIndexed, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, ExtractVector128, 32, 2, {INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX, Floor, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, HorizontalAdd, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_haddps, INS_haddpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, HorizontalSubtract, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_hsubps, INS_hsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, InsertVector128, 32, 3, {INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX, LoadAlignedVector256, 32, 1, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movaps, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, LoadDquVector256, 32, 1, {INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, LoadVector256, 32, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX, Max, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxps, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(AVX, Min, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minps, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(AVX, MaskLoad, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vmaskmovps, INS_vmaskmovpd}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, MaskStore, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vmaskmovps, INS_vmaskmovpd}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, MoveMask, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskps, INS_movmskpd}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, Multiply, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulps, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX, Or, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX, Permute, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilps, INS_vpermilpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX, Permute2x128, 32, 3, {INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, PermuteVar, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpsvar, INS_vpermilpdvar}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Reciprocal, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, ReciprocalSqrt, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, RoundCurrentDirection, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, RoundToNearestInteger, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, RoundToNegativeInfinity, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, RoundToPositiveInfinity, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, RoundToZero, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Shuffle, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shufps, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX, Sqrt, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Store, 32, 2, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX, StoreAligned, 32, 2, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movaps, INS_movapd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(AVX, StoreAlignedNonTemporal, 32, 2, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntps, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(AVX, Subtract, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subps, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, TestC, -1, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, TestNotZAndNotC, -1, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, TestZ, -1, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, UnpackHigh, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpckhps, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, UnpackLow, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpcklps, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX, Xor, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX, Add, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addps, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_EmbBroadcastCompatible)
+HARDWARE_INTRINSIC(AVX, AddSubtract, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_addsubps, INS_addsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, And, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX, AndNot, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Blend, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blendps, INS_blendpd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, BlendVariable, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vblendvps, INS_vblendvpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, Ceiling, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, BroadcastScalarToVector128, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastss, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, BroadcastScalarToVector256, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, BroadcastVector128ToVector256, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcastf128, INS_vbroadcastf128}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Compare, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_IMM, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareGreaterThan, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareGreaterThanOrEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareLessThan, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareLessThanOrEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareNotEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareNotGreaterThan, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareNotGreaterThanOrEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareNotLessThan, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareNotLessThanOrEqual, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareOrdered, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareUnordered, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpps, INS_cmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX, CompareScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cmpss, INS_cmpsd}, HW_Category_IMM, HW_Flag_CopyUpperBits|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, ConvertToVector128Int32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, ConvertToVector128Single, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX, ConvertToVector256Int32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, ConvertToVector256Single, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX, ConvertToVector256Double, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX, ConvertToVector128Int32WithTruncation, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttpd2dq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, ConvertToVector256Int32WithTruncation, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX, Divide, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, DotProduct, 32, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_dpps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, DuplicateEvenIndexed, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, DuplicateOddIndexed, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, ExtractVector128, 32, 2, false, {INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128, INS_vextractf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX, Floor, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, HorizontalAdd, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_haddps, INS_haddpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, HorizontalSubtract, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_hsubps, INS_hsubpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, InsertVector128, 32, 3, false, {INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128, INS_vinsertf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX, LoadAlignedVector256, 32, 1, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movaps, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, LoadDquVector256, 32, 1, false, {INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_lddqu, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, LoadVector256, 32, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX, Max, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_maxps, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(AVX, Min, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_minps, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(AVX, MaskLoad, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vmaskmovps, INS_vmaskmovpd}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, MaskStore, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vmaskmovps, INS_vmaskmovpd}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, MoveMask, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movmskps, INS_movmskpd}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, Multiply, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulps, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX, Or, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX, Permute, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilps, INS_vpermilpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX, Permute2x128, 32, 3, false, {INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128, INS_vperm2f128}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, PermuteVar, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpsvar, INS_vpermilpdvar}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Reciprocal, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rcpps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, ReciprocalSqrt, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_rsqrtps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, RoundCurrentDirection, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, RoundToNearestInteger, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, RoundToNegativeInfinity, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, RoundToPositiveInfinity, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, RoundToZero, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_roundps, INS_roundpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Shuffle, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_shufps, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX, Sqrt, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Store, 32, 2, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX, StoreAligned, 32, 2, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movaps, INS_movapd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVX, StoreAlignedNonTemporal, 32, 2, true, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntps, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVX, Subtract, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_subps, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, TestC, -1, 2, true, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, TestNotZAndNotC, -1, 2, true, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, TestZ, -1, 2, true, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, UnpackHigh, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpckhps, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, UnpackLow, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_unpcklps, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX, Xor, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX2 Intrinsics
-HARDWARE_INTRINSIC(AVX2, Abs, 32, 1, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_pabsd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX2, Add, 32, 2, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, AddSaturate, 32, 2, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, AlignRight, 32, 3, {INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, And, 32, 2, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, AndNot, 32, 2, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, Average, 32, 2, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, Blend, -1, 3, {INS_invalid, INS_invalid, INS_pblendw, INS_pblendw, INS_vpblendd, INS_vpblendd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, BlendVariable, 32, 3, {INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, BroadcastScalarToVector128, 16, 1, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_movddup}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX2, BroadcastScalarToVector256, 32, 1, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX2, BroadcastVector128ToVector256, 32, 1, {INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, CompareEqual, 32, 2, {INS_pcmpeqb, INS_pcmpeqb, INS_pcmpeqw, INS_pcmpeqw, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqq, INS_pcmpeqq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, CompareGreaterThan, 32, 2, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, CompareLessThan, 32, 2, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, ExtractVector128, 32, 2, {INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ConvertToInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX2, ConvertToUInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int16, 32, 1, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int32, 32, 1, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int64, 32, 1, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX2, GatherVector128, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, GatherVector256, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, GatherMaskVector128, 16, 5, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, GatherMaskVector256, 32, 5, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, HorizontalAdd, 32, 2, {INS_invalid, INS_invalid, INS_phaddw, INS_phaddw, INS_phaddd, INS_phaddd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, HorizontalAddSaturate, 32, 2, {INS_invalid, INS_invalid, INS_phaddsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, HorizontalSubtract, 32, 2, {INS_invalid, INS_invalid, INS_phsubw, INS_invalid, INS_phsubd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, HorizontalSubtractSaturate, 32, 2, {INS_invalid, INS_invalid, INS_phsubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, InsertVector128, 32, 3, {INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, LoadAlignedVector256NonTemporal, 32, 1, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, MaskLoad, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaskmovd, INS_vpmaskmovd, INS_vpmaskmovq, INS_vpmaskmovq, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, MaskStore, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaskmovd, INS_vpmaskmovd, INS_vpmaskmovq, INS_vpmaskmovq, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, Max, 32, 2, {INS_pmaxsb, INS_pmaxub, INS_pmaxsw, INS_pmaxuw, INS_pmaxsd, INS_pmaxud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, Min, 32, 2, {INS_pminsb, INS_pminub, INS_pminsw, INS_pminuw, INS_pminsd, INS_pminud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, MoveMask, 32, 1, {INS_pmovmskb, INS_pmovmskb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, Multiply, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_pmuludq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, MultipleSumAbsoluteDifferences, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_mpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, MultiplyAddAdjacent, 32, 2, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, MultiplyHigh, 32, 2, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, MultiplyHighRoundScale, 32, 2, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, MultiplyLow, 32, 2, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, Or, 32, 2, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX2, Permute2x128, 32, 3, {INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, Permute4x64, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq, INS_vpermq, INS_invalid, INS_vpermpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, PermuteVar8x32, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermd, INS_vpermd, INS_invalid, INS_invalid, INS_vpermps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX2, PackSignedSaturate, 32, 2, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, PackUnsignedSaturate, 32, 2, {INS_invalid, INS_packuswb, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, ShiftLeftLogical, 32, 2, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShiftLeftLogical128BitLane, 32, 2, {INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShiftLeftLogicalVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsllvd, INS_vpsllvd, INS_vpsllvq, INS_vpsllvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, ShiftRightArithmetic, 32, 2, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_psrad, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShiftRightArithmeticVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravd, INS_vpsravd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, ShiftRightLogical, 32, 2, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShiftRightLogical128BitLane, 32, 2, {INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShiftRightLogicalVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsrlvd, INS_vpsrlvd, INS_vpsrlvq, INS_vpsrlvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, Shuffle, 32, 2, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_MaybeIMM)
-HARDWARE_INTRINSIC(AVX2, ShuffleHigh, 32, 2, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, ShuffleLow, 32, 2, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX2, Sign, 32, 2, {INS_psignb, INS_invalid, INS_psignw, INS_invalid, INS_psignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX2, SumAbsoluteDifferences, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, Subtract, 32, 2, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, SubtractSaturate, 32, 2, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, UnpackHigh, 32, 2, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, UnpackLow, 32, 2, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX2, Xor, 32, 2, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, Abs, 32, 1, true, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_pabsd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX2, Add, 32, 2, true, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_EmbBroadcastCompatible)
+HARDWARE_INTRINSIC(AVX2, AddSaturate, 32, 2, true, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, AlignRight, 32, 3, false, {INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_palignr, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, And, 32, 2, false, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, AndNot, 32, 2, false, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, Average, 32, 2, true, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, Blend, -1, 3, true, {INS_invalid, INS_invalid, INS_pblendw, INS_pblendw, INS_vpblendd, INS_vpblendd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, BlendVariable, 32, 3, false, {INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_vpblendvb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, BroadcastScalarToVector128, 16, 1, true, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_movddup}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX2, BroadcastScalarToVector256, 32, 1, true, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX2, BroadcastVector128ToVector256, 32, 1, false, {INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_vbroadcasti128, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, CompareEqual, 32, 2, true, {INS_pcmpeqb, INS_pcmpeqb, INS_pcmpeqw, INS_pcmpeqw, INS_pcmpeqd, INS_pcmpeqd, INS_pcmpeqq, INS_pcmpeqq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, CompareGreaterThan, 32, 2, true, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, CompareLessThan, 32, 2, true, {INS_pcmpgtb, INS_invalid, INS_pcmpgtw, INS_invalid, INS_pcmpgtd, INS_invalid, INS_pcmpgtq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, ExtractVector128, 32, 2, false, {INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ConvertToInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX2, ConvertToUInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int16, 32, 1, true, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int32, 32, 1, true, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX2, ConvertToVector256Int64, 32, 1, true, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_BaseTypeFromFirstArg|HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX2, GatherVector128, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, GatherVector256, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, GatherMaskVector128, 16, 5, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, GatherMaskVector256, 32, 5, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpgatherdd, INS_vpgatherdd, INS_vpgatherdq, INS_vpgatherdq, INS_vgatherdps, INS_vgatherdpd}, HW_Category_IMM, HW_Flag_MaybeMemoryLoad|HW_Flag_SpecialCodeGen|HW_Flag_SpecialImport|HW_Flag_NoContainment|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, HorizontalAdd, 32, 2, true, {INS_invalid, INS_invalid, INS_phaddw, INS_phaddw, INS_phaddd, INS_phaddd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, HorizontalAddSaturate, 32, 2, false, {INS_invalid, INS_invalid, INS_phaddsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, HorizontalSubtract, 32, 2, true, {INS_invalid, INS_invalid, INS_phsubw, INS_invalid, INS_phsubd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, HorizontalSubtractSaturate, 32, 2, false, {INS_invalid, INS_invalid, INS_phsubsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, InsertVector128, 32, 3, false, {INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, LoadAlignedVector256NonTemporal, 32, 1, false, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, MaskLoad, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaskmovd, INS_vpmaskmovd, INS_vpmaskmovq, INS_vpmaskmovq, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, MaskStore, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaskmovd, INS_vpmaskmovd, INS_vpmaskmovq, INS_vpmaskmovq, INS_invalid, INS_invalid}, HW_Category_MemoryStore, HW_Flag_NoContainment|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, Max, 32, 2, true, {INS_pmaxsb, INS_pmaxub, INS_pmaxsw, INS_pmaxuw, INS_pmaxsd, INS_pmaxud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, Min, 32, 2, true, {INS_pminsb, INS_pminub, INS_pminsw, INS_pminuw, INS_pminsd, INS_pminud, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, MoveMask, 32, 1, false, {INS_pmovmskb, INS_pmovmskb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoContainment|HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, Multiply, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_pmuludq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, MultipleSumAbsoluteDifferences, 32, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_mpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, MultiplyAddAdjacent, 32, 2, true, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, MultiplyHigh, 32, 2, true, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, MultiplyHighRoundScale, 32, 2, false, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, MultiplyLow, 32, 2, true, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, Or, 32, 2, false, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX2, Permute2x128, 32, 3, false, {INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_vperm2i128, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, Permute4x64, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq, INS_vpermq, INS_invalid, INS_vpermpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, PermuteVar8x32, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermd, INS_vpermd, INS_invalid, INS_invalid, INS_vpermps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX2, PackSignedSaturate, 32, 2, true, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, PackUnsignedSaturate, 32, 2, true, {INS_invalid, INS_packuswb, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, ShiftLeftLogical, 32, 2, true, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShiftLeftLogical128BitLane, 32, 2, false, {INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_pslldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShiftLeftLogicalVariable, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsllvd, INS_vpsllvd, INS_vpsllvq, INS_vpsllvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, ShiftRightArithmetic, 32, 2, true, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_psrad, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShiftRightArithmeticVariable, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravd, INS_vpsravd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, ShiftRightLogical, 32, 2, true, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShiftRightLogical128BitLane, 32, 2, false, {INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_psrldq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShiftRightLogicalVariable, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsrlvd, INS_vpsrlvd, INS_vpsrlvq, INS_vpsrlvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, Shuffle, 32, 2, true, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_MaybeIMM)
+HARDWARE_INTRINSIC(AVX2, ShuffleHigh, 32, 2, false, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, ShuffleLow, 32, 2, false, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX2, Sign, 32, 2, true, {INS_psignb, INS_invalid, INS_psignw, INS_invalid, INS_psignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX2, SumAbsoluteDifferences, 32, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, Subtract, 32, 2, true, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, SubtractSaturate, 32, 2, true, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, UnpackHigh, 32, 2, true, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, UnpackLow, 32, 2, true, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX2, Xor, 32, 2, false, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512F Intrinsics
-HARDWARE_INTRINSIC(AVX512F, Abs, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pabsd, INS_invalid, INS_vpabsq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, Add, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_addps, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F, AlignRight32, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignd, INS_valignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, AlignRight64, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignq, INS_valignq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, And, 64, 2, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_vpandq, INS_vpandq, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F, AndNot, 64, 2, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_vpandnq, INS_vpandnq, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, BroadcastScalarToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_SIMDScalar, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, BroadcastVector128ToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti128, INS_vbroadcasti128, INS_invalid, INS_invalid, INS_vbroadcastf128, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, BroadcastVector256ToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti64x4, INS_vbroadcasti64x4, INS_invalid, INS_vbroadcastf64x4}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ConvertScalarToVector128Double, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2sd32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, ConvertScalarToVector128Single, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2ss32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, ConvertToUInt32, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtss2usi, INS_vcvtsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToUInt32WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttss2usi, INS_vcvttsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Byte, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128ByteWithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdb, INS_invalid, INS_vpmovusqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Int16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Int16WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqw, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128SByte, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128SByteWithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdb, INS_invalid, INS_vpmovsqb, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128UInt16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector128UInt16WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int16WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_cvtpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Single, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt16, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt16WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_vcvtpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32WithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Double, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int32, 64, 1, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int32WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int64, 64, 1, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Single, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt32, 64, 1, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt32WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt64, 64, 1, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F, Divide, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, DuplicateEvenIndexed, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, DuplicateOddIndexed, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ExtractVector128, 64, 2, {INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextractf128, INS_vextractf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, ExtractVector256, 64, 2, {INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextractf64x4, INS_vextractf64x4}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, Fixup, 64, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmps, INS_vfixupimmpd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, FixupScalar, 16, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmss, INS_vfixupimmsd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAdd, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ps, INS_vfmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAddNegated, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ps, INS_vfnmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAddSubtract, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmaddsub213ps, INS_vfmaddsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtract, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ps, INS_vfmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtractAdd, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsubadd213ps, INS_vfmsubadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtractNegated, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ps, INS_vfnmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, GetExponent, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpps, INS_vgetexppd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, GetExponentScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpss, INS_vgetexpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, GetMantissa, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantps, INS_vgetmantpd}, HW_Category_IMM, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, GetMantissaScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantss, INS_vgetmantsd}, HW_Category_IMM, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, InsertVector128, 64, 3, {INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinsertf128, INS_vinsertf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, InsertVector256, 64, 3, {INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinsertf64x4, INS_vinsertf64x4}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, LoadAlignedVector512, 64, 1, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_vmovdqa64, INS_vmovdqa64, INS_movaps, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, LoadAlignedVector512NonTemporal, 64, 1, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, LoadVector512, 64, 1, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX512F, Max, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmaxsd, INS_pmaxud, INS_vpmaxsq, INS_vpmaxuq, INS_maxps, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(AVX512F, Min, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pminsd, INS_pminud, INS_vpminsq, INS_vpminuq, INS_minps, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
-HARDWARE_INTRINSIC(AVX512F, Multiply, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_pmuludq, INS_mulps, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F, MultiplyLow, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F, Or, 64, 2, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_vporq, INS_vporq, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F, Permute2x64, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, Permute4x32, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, Permute4x64, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq, INS_vpermq, INS_invalid, INS_vpermpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar2x64, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpdvar}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar4x32, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpsvar, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar8x64, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq_reg, INS_vpermq_reg, INS_invalid, INS_vpermpd_reg}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar8x64x2, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar16x32, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermd, INS_vpermd, INS_invalid, INS_invalid, INS_vpermps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512F, PermuteVar16x32x2, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F, Reciprocal14, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ps, INS_vrcp14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, Reciprocal14Scalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ss, INS_vrcp14sd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, ReciprocalSqrt14, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ps, INS_vrsqrt14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ReciprocalSqrt14Scalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ss, INS_vrsqrt14sd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, RotateLeft, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprold, INS_vprold, INS_vprolq, INS_vprolq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, RotateLeftVariable, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprolvd, INS_vprolvd, INS_vprolvq, INS_vprolvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, RotateRight, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprord, INS_vprord, INS_vprorq, INS_vprorq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, RotateRightVariable, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprorvd, INS_vprorvd, INS_vprorvq, INS_vprorvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, RoundScale, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaleps, INS_vrndscalepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, RoundScaleScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaless, INS_vrndscalesd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, Scale, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefps, INS_vscalefpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ScaleScalar, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefss, INS_vscalefsd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512F, ShiftLeftLogical, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, ShiftLeftLogicalVariable, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsllvd, INS_vpsllvd, INS_vpsllvq, INS_vpsllvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ShiftRightArithmetic, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psrad, INS_invalid, INS_vpsraq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, ShiftRightArithmeticVariable, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravd, INS_invalid, INS_vpsravq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, ShiftRightLogical, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, ShiftRightLogicalVariable, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsrlvd, INS_vpsrlvd, INS_vpsrlvq, INS_vpsrlvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, Shuffle, 64, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_shufps, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, Shuffle4x128, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vshufi32x4, INS_vshufi32x4, INS_vshufi64x2, INS_vshufi64x2, INS_vshuff32x4, INS_vshuff64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, Sqrt, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, Store, 64, 2, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX512F, StoreAligned, 64, 2, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_vmovdqa64, INS_vmovdqa64, INS_movaps, INS_movapd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(AVX512F, StoreAlignedNonTemporal, 64, 2, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntps, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(AVX512F, Subtract, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_subps, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, UnpackHigh, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_unpckhps, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, TernaryLogic, 64, 4, {INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogq, INS_vpternlogq, INS_vpternlogd, INS_vpternlogq}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F, UnpackLow, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_unpcklps, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F, Xor, 64, 2, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_vpxorq, INS_vpxorq, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F, Abs, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pabsd, INS_invalid, INS_vpabsq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, Add, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_paddd, INS_paddd, INS_paddq, INS_paddq, INS_addps, INS_addpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_EmbBroadcastCompatible)
+HARDWARE_INTRINSIC(AVX512F, AlignRight32, 64, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignd, INS_valignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, AlignRight64, 64, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignq, INS_valignq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, And, 64, 2, true, {INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_pand, INS_vpandq, INS_vpandq, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F, AndNot, 64, 2, true, {INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_pandn, INS_vpandnq, INS_vpandnq, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, BroadcastScalarToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpbroadcastd, INS_vpbroadcastd, INS_vpbroadcastq, INS_vpbroadcastq, INS_vbroadcastss, INS_vbroadcastsd}, HW_Category_SIMDScalar, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, BroadcastVector128ToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti128, INS_vbroadcasti128, INS_invalid, INS_invalid, INS_vbroadcastf128, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, BroadcastVector256ToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti64x4, INS_vbroadcasti64x4, INS_invalid, INS_vbroadcastf64x4}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, CompareEqual, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpeqd, INS_vpcmpeqd, INS_vpcmpeqq, INS_vpcmpeqq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, CompareGreaterThan, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpgtd, INS_vpcmpud, INS_vpcmpgtq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, CompareGreaterThanOrEqual, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, CompareLessThan, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, CompareLessThanOrEqual, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, CompareNotEqual, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertScalarToVector128Double, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2sd32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, ConvertScalarToVector128Single, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2ss32, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, ConvertToUInt32, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtss2usi, INS_vcvtsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToUInt32WithTruncation, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttss2usi, INS_vcvttsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Byte, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128ByteWithSaturation, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdb, INS_invalid, INS_vpmovusqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Int16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128Int16WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqw, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128SByte, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128SByteWithSaturation, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdb, INS_invalid, INS_vpmovsqb, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128UInt16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector128UInt16WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int16WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_cvtpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Int32WithTruncation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttpd2dq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256Single, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtpd2ps}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt16, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt16WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_vcvtpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32WithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector256UInt32WithTruncation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Double, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2pd, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_cvtps2pd, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int32, 64, 1, true, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int32WithTruncation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvttps2dq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Int64, 64, 1, true, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512Single, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_cvtdq2ps, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt32, 64, 1, true, {INS_pmovsxbd, INS_pmovzxbd, INS_pmovsxwd, INS_pmovzxwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt32WithTruncation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, ConvertToVector512UInt64, 64, 1, true, {INS_pmovsxbq, INS_pmovzxbq, INS_pmovsxwq, INS_pmovzxwq, INS_pmovsxdq, INS_pmovzxdq, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F, Divide, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_divps, INS_divpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, DuplicateEvenIndexed, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movsldup, INS_movddup}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, DuplicateOddIndexed, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movshdup, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, ExtractVector128, 64, 2, true, {INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextracti128, INS_vextractf128, INS_vextractf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, ExtractVector256, 64, 2, true, {INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextracti64x4, INS_vextractf64x4, INS_vextractf64x4}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, Fixup, 64, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmps, INS_vfixupimmpd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, FixupScalar, 16, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmss, INS_vfixupimmsd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAdd, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ps, INS_vfmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAddNegated, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ps, INS_vfnmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplyAddSubtract, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmaddsub213ps, INS_vfmaddsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtract, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ps, INS_vfmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtractAdd, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsubadd213ps, INS_vfmsubadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, FusedMultiplySubtractNegated, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ps, INS_vfnmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, GetExponent, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpps, INS_vgetexppd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, GetExponentScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpss, INS_vgetexpsd}, HW_Category_SIMDScalar, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, GetMantissa, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantps, INS_vgetmantpd}, HW_Category_IMM, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, GetMantissaScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantss, INS_vgetmantsd}, HW_Category_IMM, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, InsertVector128, 64, 3, true, {INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinserti128, INS_vinsertf128, INS_vinsertf128}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, InsertVector256, 64, 3, true, {INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinserti64x4, INS_vinsertf64x4, INS_vinsertf64x4}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, LoadAlignedVector512, 64, 1, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_vmovdqa64, INS_vmovdqa64, INS_movaps, INS_movapd}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, LoadAlignedVector512NonTemporal, 64, 1, false, {INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_movntdqa, INS_invalid, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, LoadVector512, 64, 1, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, Max, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmaxsd, INS_pmaxud, INS_vpmaxsq, INS_vpmaxuq, INS_maxps, INS_maxpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(AVX512F, Min, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pminsd, INS_pminud, INS_vpminsq, INS_vpminuq, INS_minps, INS_minpd}, HW_Category_SimpleSIMD, HW_Flag_MaybeCommutative)
+HARDWARE_INTRINSIC(AVX512F, Multiply, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmuldq, INS_pmuludq, INS_mulps, INS_mulpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F, MultiplyLow, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pmulld, INS_pmulld, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F, Or, 64, 2, true, {INS_por, INS_por, INS_por, INS_por, INS_por, INS_por, INS_vporq, INS_vporq, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F, Permute2x64, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, Permute4x32, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilps, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, Permute4x64, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq, INS_vpermq, INS_invalid, INS_vpermpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar2x64, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpdvar}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar4x32, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermilpsvar, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar8x64, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq_reg, INS_vpermq_reg, INS_invalid, INS_vpermpd_reg}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar8x64x2, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar16x32, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermd, INS_vpermd, INS_invalid, INS_invalid, INS_vpermps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512F, PermuteVar16x32x2, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F, Reciprocal14, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ps, INS_vrcp14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, Reciprocal14Scalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ss, INS_vrcp14sd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, ReciprocalSqrt14, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ps, INS_vrsqrt14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, ReciprocalSqrt14Scalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ss, INS_vrsqrt14sd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, RotateLeft, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprold, INS_vprold, INS_vprolq, INS_vprolq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, RotateLeftVariable, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprolvd, INS_vprolvd, INS_vprolvq, INS_vprolvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, RotateRight, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprord, INS_vprord, INS_vprorq, INS_vprorq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, RotateRightVariable, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprorvd, INS_vprorvd, INS_vprorvq, INS_vprorvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, RoundScale, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaleps, INS_vrndscalepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, RoundScaleScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaless, INS_vrndscalesd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, Scale, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefps, INS_vscalefpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, ScaleScalar, 16, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefss, INS_vscalefsd}, HW_Category_SimpleSIMD, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512F, ShiftLeftLogical, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pslld, INS_pslld, INS_psllq, INS_psllq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, ShiftLeftLogicalVariable, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsllvd, INS_vpsllvd, INS_vpsllvq, INS_vpsllvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, ShiftRightArithmetic, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psrad, INS_invalid, INS_vpsraq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, ShiftRightArithmeticVariable, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravd, INS_invalid, INS_vpsravq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, ShiftRightLogical, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psrld, INS_psrld, INS_psrlq, INS_psrlq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, ShiftRightLogicalVariable, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsrlvd, INS_vpsrlvd, INS_vpsrlvq, INS_vpsrlvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, Shuffle, 64, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pshufd, INS_pshufd, INS_invalid, INS_invalid, INS_shufps, INS_shufpd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, Shuffle4x128, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vshufi32x4, INS_vshufi32x4, INS_vshufi64x2, INS_vshufi64x2, INS_vshuff32x4, INS_vshuff64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, Sqrt, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_sqrtps, INS_sqrtpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, Store, 64, 2, true, {INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_movdqu, INS_vmovdqu64, INS_vmovdqu64, INS_movups, INS_movupd}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F, StoreAligned, 64, 2, true, {INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_movdqa, INS_vmovdqa64, INS_vmovdqa64, INS_movaps, INS_movapd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVX512F, StoreAlignedNonTemporal, 64, 2, true, {INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntdq, INS_movntps, INS_movntpd}, HW_Category_MemoryStore, HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVX512F, Subtract, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_psubd, INS_psubd, INS_psubq, INS_psubq, INS_subps, INS_subpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, UnpackHigh, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_punpckhdq, INS_punpckhdq, INS_punpckhqdq, INS_punpckhqdq, INS_unpckhps, INS_unpckhpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, TernaryLogic, 64, 4, true, {INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogq, INS_vpternlogq, INS_vpternlogd, INS_vpternlogq}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F, UnpackLow, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_punpckldq, INS_punpckldq, INS_punpcklqdq, INS_punpcklqdq, INS_unpcklps, INS_unpcklpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F, Xor, 64, 2, true, {INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_pxor, INS_vpxorq, INS_vpxorq, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512F.VL Intrinsics
-HARDWARE_INTRINSIC(AVX512F_VL, Abs, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpabsq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, AlignRight32, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignd, INS_valignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, AlignRight64, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignq, INS_valignq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, Max, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaxsq, INS_vpmaxuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F_VL, Min, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpminsq, INS_vpminuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Byte, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128ByteWithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdb, INS_invalid, INS_vpmovusqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Double, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int16, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int16WithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdw, INS_invalid, INS_vpmovsqw, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int32, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int32WithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128SByte, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128SByteWithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdb, INS_invalid, INS_vpmovsqb, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Single, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt16, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt16WithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdw, INS_invalid, INS_vpmovusqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_vcvtps2udq, INS_vcvtpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32WithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32WithTruncation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_vcvttpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256Double, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256Single, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256UInt32, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256UInt32WithTruncation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512F_VL, Fixup, -1, 4, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmps, INS_vfixupimmpd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, GetExponent, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpps, INS_vgetexppd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, GetMantissa, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantps, INS_vgetmantpd}, HW_Category_IMM, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar2x64x2, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x32x2, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x64, 32, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq_reg, INS_vpermq_reg, INS_invalid, INS_vpermpd_reg}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x64x2, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar8x32x2, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512F_VL, Reciprocal14, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ps, INS_vrcp14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, ReciprocalSqrt14, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ps, INS_vrsqrt14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, RotateLeft, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprold, INS_vprold, INS_vprolq, INS_vprolq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, RotateLeftVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprolvd, INS_vprolvd, INS_vprolvq, INS_vprolvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, RotateRight, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprord, INS_vprord, INS_vprorq, INS_vprorq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, RotateRightVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprorvd, INS_vprorvd, INS_vprorvq, INS_vprorvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, RoundScale, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaleps, INS_vrndscalepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, Scale, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefps, INS_vscalefpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, ShiftRightArithmetic, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsraq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, ShiftRightArithmeticVariable, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512F_VL, Shuffle2x128, 32, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vshufi32x4, INS_vshufi32x4, INS_vshufi64x2, INS_vshufi64x2, INS_vshuff32x4, INS_vshuff64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512F_VL, TernaryLogic, -1, 4, {INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogq, INS_vpternlogq, INS_vpternlogd, INS_vpternlogq}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, Abs, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpabsq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, AlignRight32, -1, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignd, INS_valignd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, AlignRight64, -1, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_valignq, INS_valignq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, Max, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmaxsq, INS_vpmaxuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F_VL, Min, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpminsq, INS_vpminuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512F_VL, CompareGreaterThan, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpud, INS_invalid, INS_vpcmpuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, CompareGreaterThanOrEqual, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, CompareLessThan, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, CompareLessThanOrEqual, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, CompareNotEqual, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Byte, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128ByteWithSaturation, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdb, INS_invalid, INS_vpmovusqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Double, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int16, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int16WithSaturation, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdw, INS_invalid, INS_vpmovsqw, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int32, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Int32WithSaturation, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsqd, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128SByte, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdb, INS_vpmovdb, INS_vpmovqb, INS_vpmovqb, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128SByteWithSaturation, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovsdb, INS_invalid, INS_vpmovsqb, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128Single, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt16, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovdw, INS_vpmovdw, INS_vpmovqw, INS_vpmovqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt16WithSaturation, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusdw, INS_invalid, INS_vpmovusqw, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovqd, INS_vpmovqd, INS_vcvtps2udq, INS_vcvtpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32WithSaturation, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmovusqd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector128UInt32WithTruncation, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_vcvttpd2udq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256Double, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2pd, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256Single, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtudq2ps, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256UInt32, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, ConvertToVector256UInt32WithTruncation, 32, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2udq, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512F_VL, Fixup, -1, 4, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfixupimmps, INS_vfixupimmpd}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, GetExponent, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetexpps, INS_vgetexppd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, GetMantissa, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vgetmantps, INS_vgetmantpd}, HW_Category_IMM, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar2x64x2, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x32x2, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x64, 32, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermq_reg, INS_vpermq_reg, INS_invalid, INS_vpermpd_reg}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar4x64x2, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2q, INS_vpermt2q, INS_invalid, INS_vpermt2pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F_VL, PermuteVar8x32x2, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpermt2d, INS_vpermt2d, INS_invalid, INS_invalid, INS_vpermt2ps, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512F_VL, Reciprocal14, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrcp14ps, INS_vrcp14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, ReciprocalSqrt14, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrsqrt14ps, INS_vrsqrt14pd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, RotateLeft, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprold, INS_vprold, INS_vprolq, INS_vprolq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, RotateLeftVariable, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprolvd, INS_vprolvd, INS_vprolvq, INS_vprolvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, RotateRight, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprord, INS_vprord, INS_vprorq, INS_vprorq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_MaybeNoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, RotateRightVariable, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vprorvd, INS_vprorvd, INS_vprorvq, INS_vprorvq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, RoundScale, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrndscaleps, INS_vrndscalepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, Scale, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vscalefps, INS_vscalefpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, ShiftRightArithmetic, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsraq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, ShiftRightArithmeticVariable, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpsravq, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512F_VL, Shuffle2x128, 32, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vshufi32x4, INS_vshufi32x4, INS_vshufi64x2, INS_vshufi64x2, INS_vshuff32x4, INS_vshuff64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512F_VL, TernaryLogic, -1, 4, true, {INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogd, INS_vpternlogq, INS_vpternlogq, INS_vpternlogd, INS_vpternlogq}, HW_Category_IMM, HW_Flag_SpecialImport|HW_Flag_FullRangeIMM)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512F.X64 Intrinsics
-HARDWARE_INTRINSIC(AVX512F_X64, ConvertScalarToVector128Double, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2sd64, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_X64, ConvertScalarToVector128Single, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2ss64, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_X64, ConvertToUInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtss2usi, INS_vcvtsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512F_X64, ConvertToUInt64WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttss2usi, INS_vcvttsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_X64, ConvertScalarToVector128Double, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2sd64, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_X64, ConvertScalarToVector128Single, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtusi2ss64, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromSecondArg|HW_Flag_CopyUpperBits|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_X64, ConvertToUInt64, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtss2usi, INS_vcvtsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512F_X64, ConvertToUInt64WithTruncation, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttss2usi, INS_vcvttsd2usi}, HW_Category_SIMDScalar, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512BW Intrinsics
-HARDWARE_INTRINSIC(AVX512BW, Abs, 64, 1, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512BW, Add, 64, 2, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, AddSaturate, 64, 2, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, AlignRight, 64, 3, {INS_palignr, INS_palignr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, Average, 64, 2, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, BroadcastScalarToVector512, 64, 1, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256Byte, 64, 1, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256ByteWithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_vpmovuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256SByte, 64, 1, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256SByteWithSaturation, 64, 1, {INS_invalid, INS_invalid, INS_vpmovswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector512Int16, 64, 1, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512BW, ConvertToVector512UInt16, 64, 1, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512BW, LoadVector512, 64, 1, {INS_vmovdqu8, INS_vmovdqu8, INS_vmovdqu16, INS_vmovdqu16, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, Max, 64, 2, {INS_pmaxsb, INS_pmaxub, INS_pmaxsw, INS_pmaxuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, Min, 64, 2, {INS_pminsb, INS_pminub, INS_pminsw, INS_pminuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, MultiplyAddAdjacent, 64, 2, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, MultiplyHigh, 64, 2, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, MultiplyHighRoundScale, 64, 2, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, MultiplyLow, 64, 2, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512BW, PackSignedSaturate, 64, 2, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, PackUnsignedSaturate, 64, 2, {INS_invalid, INS_packuswb, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, PermuteVar32x16, 64, 2, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512BW, PermuteVar32x16x2, 64, 3, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogical, 64, 2, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogical128BitLane, 64, 2, {INS_pslldq, INS_pslldq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogicalVariable, 64, 2, {INS_invalid, INS_invalid, INS_vpsllvw, INS_vpsllvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, ShiftRightArithmetic, 64, 2, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShiftRightArithmeticVariable, 64, 2, {INS_invalid, INS_invalid, INS_vpsravw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogical, 64, 2, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogical128BitLane, 64, 2, {INS_psrldq, INS_psrldq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogicalVariable, 64, 2, {INS_invalid, INS_invalid, INS_vpsrlvw, INS_vpsrlvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, Shuffle, 64, 2, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, ShuffleHigh, 64, 2, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, ShuffleLow, 64, 2, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, Store, 64, 2, {INS_vmovdqu8, INS_vmovdqu8, INS_vmovdqu16, INS_vmovdqu16, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
-HARDWARE_INTRINSIC(AVX512BW, Subtract, 64, 2, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, SubtractSaturate, 64, 2, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, SumAbsoluteDifferences, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, SumAbsoluteDifferencesInBlock32, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_vdbpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512BW, UnpackHigh, 64, 2, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW, UnpackLow, 64, 2, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, Abs, 64, 1, true, {INS_pabsb, INS_invalid, INS_pabsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512BW, Add, 64, 2, true, {INS_paddb, INS_paddb, INS_paddw, INS_paddw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, AddSaturate, 64, 2, true, {INS_paddsb, INS_paddusb, INS_paddsw, INS_paddusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, AlignRight, 64, 3, false, {INS_palignr, INS_palignr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, Average, 64, 2, true, {INS_invalid, INS_pavgb, INS_invalid, INS_pavgw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, BroadcastScalarToVector512, 64, 1, true, {INS_vpbroadcastb, INS_vpbroadcastb, INS_vpbroadcastw, INS_vpbroadcastw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_MaybeMemoryLoad)
+HARDWARE_INTRINSIC(AVX512BW, CompareEqual, 64, 2, true, {INS_vpcmpeqb, INS_vpcmpeqb, INS_vpcmpeqw, INS_vpcmpeqw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, CompareGreaterThan, 64, 2, true, {INS_vpcmpgtb, INS_vpcmpub, INS_vpcmpgtw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, CompareGreaterThanOrEqual, 64, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, CompareLessThan, 64, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, CompareLessThanOrEqual, 64, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, CompareNotEqual, 64, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256Byte, 64, 1, false, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256ByteWithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_vpmovuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256SByte, 64, 1, false, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector256SByteWithSaturation, 64, 1, false, {INS_invalid, INS_invalid, INS_vpmovswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector512Int16, 64, 1, true, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512BW, ConvertToVector512UInt16, 64, 1, true, {INS_pmovsxbw, INS_pmovzxbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512BW, LoadVector512, 64, 1, true, {INS_vmovdqu8, INS_vmovdqu8, INS_vmovdqu16, INS_vmovdqu16, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, Max, 64, 2, true, {INS_pmaxsb, INS_pmaxub, INS_pmaxsw, INS_pmaxuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, Min, 64, 2, true, {INS_pminsb, INS_pminub, INS_pminsw, INS_pminuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, MultiplyAddAdjacent, 64, 2, true, {INS_invalid, INS_invalid, INS_pmaddubsw, INS_invalid, INS_pmaddwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, MultiplyHigh, 64, 2, true, {INS_invalid, INS_invalid, INS_pmulhw, INS_pmulhuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, MultiplyHighRoundScale, 64, 2, false, {INS_invalid, INS_invalid, INS_pmulhrsw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, MultiplyLow, 64, 2, false, {INS_invalid, INS_invalid, INS_pmullw, INS_pmullw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512BW, PackSignedSaturate, 64, 2, true, {INS_packsswb, INS_invalid, INS_packssdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, PackUnsignedSaturate, 64, 2, true, {INS_invalid, INS_packuswb, INS_invalid, INS_packusdw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, PermuteVar32x16, 64, 2, false, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512BW, PermuteVar32x16x2, 64, 3, false, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogical, 64, 2, false, {INS_invalid, INS_invalid, INS_psllw, INS_psllw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogical128BitLane, 64, 2, false, {INS_pslldq, INS_pslldq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShiftLeftLogicalVariable, 64, 2, false, {INS_invalid, INS_invalid, INS_vpsllvw, INS_vpsllvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, ShiftRightArithmetic, 64, 2, false, {INS_invalid, INS_invalid, INS_psraw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShiftRightArithmeticVariable, 64, 2, false, {INS_invalid, INS_invalid, INS_vpsravw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogical, 64, 2, false, {INS_invalid, INS_invalid, INS_psrlw, INS_psrlw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_MaybeIMM|HW_Flag_NoJmpTableIMM|HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogical128BitLane, 64, 2, false, {INS_psrldq, INS_psrldq, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShiftRightLogicalVariable, 64, 2, false, {INS_invalid, INS_invalid, INS_vpsrlvw, INS_vpsrlvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, Shuffle, 64, 2, false, {INS_pshufb, INS_pshufb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, ShuffleHigh, 64, 2, false, {INS_invalid, INS_invalid, INS_pshufhw, INS_pshufhw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, ShuffleLow, 64, 2, false, {INS_invalid, INS_invalid, INS_pshuflw, INS_pshuflw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, Store, 64, 2, true, {INS_vmovdqu8, INS_vmovdqu8, INS_vmovdqu16, INS_vmovdqu16, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Helper, HW_Flag_SpecialImport|HW_Flag_BaseTypeFromSecondArg|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW, Subtract, 64, 2, true, {INS_psubb, INS_psubb, INS_psubw, INS_psubw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, SubtractSaturate, 64, 2, true, {INS_psubsb, INS_psubusb, INS_psubsw, INS_psubusw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, SumAbsoluteDifferences, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_psadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, SumAbsoluteDifferencesInBlock32, 64, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_vdbpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW, UnpackHigh, 64, 2, true, {INS_punpckhbw, INS_punpckhbw, INS_punpckhwd, INS_punpckhwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW, UnpackLow, 64, 2, true, {INS_punpcklbw, INS_punpcklbw, INS_punpcklwd, INS_punpcklwd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512BW.VL Intrinsics
-HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128Byte, -1, 1, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128ByteWithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_vpmovuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128SByte, -1, 1, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128SByteWithSaturation, -1, 1, {INS_invalid, INS_invalid, INS_vpmovswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
-HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar8x16 , 16, 2, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar8x16x2, 16, 3, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar16x16, 32, 2, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar16x16x2, 32, 3, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512BW_VL, ShiftLeftLogicalVariable, -1, 2, {INS_invalid, INS_invalid, INS_vpsllvw, INS_vpsllvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW_VL, ShiftRightArithmeticVariable, -1, 2, {INS_invalid, INS_invalid, INS_vpsravw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW_VL, ShiftRightLogicalVariable, -1, 2, {INS_invalid, INS_invalid, INS_vpsrlvw, INS_vpsrlvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512BW_VL, SumAbsoluteDifferencesInBlock32, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_vdbpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512BW_VL, CompareGreaterThan, -1, 2, true, {INS_invalid, INS_vpcmpub, INS_invalid, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, CompareGreaterThanOrEqual, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, CompareLessThan, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, CompareLessThanOrEqual, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, CompareNotEqual, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask|HW_Flag_NoCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128Byte, -1, 1, false, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128ByteWithSaturation, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_vpmovuswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128SByte, -1, 1, false, {INS_invalid, INS_invalid, INS_vpmovwb, INS_vpmovwb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, ConvertToVector128SByteWithSaturation, -1, 1, false, {INS_invalid, INS_invalid, INS_vpmovswb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg|HW_Flag_SpecialCodeGen)
+HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar8x16 , 16, 2, false, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar8x16x2, 16, 3, false, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar16x16, 32, 2, false, {INS_invalid, INS_invalid, INS_vpermw, INS_vpermw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512BW_VL, PermuteVar16x16x2, 32, 3, false, {INS_invalid, INS_invalid, INS_vpermt2w, INS_vpermt2w, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512BW_VL, ShiftLeftLogicalVariable, -1, 2, false, {INS_invalid, INS_invalid, INS_vpsllvw, INS_vpsllvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW_VL, ShiftRightArithmeticVariable, -1, 2, false, {INS_invalid, INS_invalid, INS_vpsravw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW_VL, ShiftRightLogicalVariable, -1, 2, false, {INS_invalid, INS_invalid, INS_vpsrlvw, INS_vpsrlvw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512BW_VL, SumAbsoluteDifferencesInBlock32, -1, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_vdbpsadbw, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512CD Intrinsics
-HARDWARE_INTRINSIC(AVX512CD, DetectConflicts, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpconflictd, INS_vpconflictd, INS_vpconflictq, INS_vpconflictq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512CD, LeadingZeroCount, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vplzcntd, INS_vplzcntd, INS_vplzcntq, INS_vplzcntq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512CD, DetectConflicts, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpconflictd, INS_vpconflictd, INS_vpconflictq, INS_vpconflictq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512CD, LeadingZeroCount, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vplzcntd, INS_vplzcntd, INS_vplzcntq, INS_vplzcntq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512CD.VL Intrinsics
-HARDWARE_INTRINSIC(AVX512CD_VL, DetectConflicts, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpconflictd, INS_vpconflictd, INS_vpconflictq, INS_vpconflictq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512CD_VL, LeadingZeroCount, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vplzcntd, INS_vplzcntd, INS_vplzcntq, INS_vplzcntq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512CD_VL, DetectConflicts, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpconflictd, INS_vpconflictd, INS_vpconflictq, INS_vpconflictq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512CD_VL, LeadingZeroCount, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vplzcntd, INS_vplzcntd, INS_vplzcntq, INS_vplzcntq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512DQ Intrinsics
-HARDWARE_INTRINSIC(AVX512DQ, And, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512DQ, AndNot, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ, BroadcastPairScalarToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_vbroadcastf32x2, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ, BroadcastVector128ToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti64x2, INS_vbroadcasti64x2, INS_invalid, INS_vbroadcastf64x2}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ, BroadcastVector256ToVector512, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x8, INS_vbroadcasti32x8, INS_invalid, INS_invalid, INS_vbroadcastf32x8, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector256Single, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2ps, INS_vcvtuqq2ps, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Double, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Int64, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Int64WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512UInt64, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512UInt64WithTruncation, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ, ExtractVector128, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vextracti64x2, INS_vextracti64x2, INS_invalid, INS_vextractf64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ, ExtractVector256, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vextracti32x8, INS_vextracti32x8, INS_invalid, INS_invalid, INS_vextractf32x8, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ, InsertVector128, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vinserti64x2, INS_vinserti64x2, INS_invalid, INS_vinsertf64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ, InsertVector256, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vinserti32x8, INS_vinserti32x8, INS_invalid, INS_invalid, INS_vinsertf32x8, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ, MultiplyLow, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmullq, INS_vpmullq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512DQ, Or, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
-HARDWARE_INTRINSIC(AVX512DQ, Range, 64, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangeps, INS_vrangepd}, HW_Category_IMM, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ, RangeScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangess, INS_vrangesd}, HW_Category_IMM, HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512DQ, Reduce, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreduceps, INS_vreducepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ, ReduceScalar, 16, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreducess, INS_vreducesd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
-HARDWARE_INTRINSIC(AVX512DQ, Xor, 64, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512DQ, And, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andps, INS_andpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512DQ, AndNot, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andnps, INS_andnpd}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ, BroadcastPairScalarToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_vbroadcastf32x2, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ, BroadcastVector128ToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti64x2, INS_vbroadcasti64x2, INS_invalid, INS_vbroadcastf64x2}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ, BroadcastVector256ToVector512, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x8, INS_vbroadcasti32x8, INS_invalid, INS_invalid, INS_vbroadcastf32x8, INS_invalid}, HW_Category_MemoryLoad, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector256Single, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2ps, INS_vcvtuqq2ps, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Double, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Int64, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512Int64WithTruncation, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512UInt64, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ConvertToVector512UInt64WithTruncation, 64, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ, ExtractVector128, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vextracti64x2, INS_vextracti64x2, INS_invalid, INS_vextractf64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ, ExtractVector256, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vextracti32x8, INS_vextracti32x8, INS_invalid, INS_invalid, INS_vextractf32x8, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ, InsertVector128, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vinserti64x2, INS_vinserti64x2, INS_invalid, INS_vinsertf64x2}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ, InsertVector256, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vinserti32x8, INS_vinserti32x8, INS_invalid, INS_invalid, INS_vinsertf32x8, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ, MultiplyLow, 64, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmullq, INS_vpmullq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512DQ, Or, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_orps, INS_orpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512DQ, Range, 64, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangeps, INS_vrangepd}, HW_Category_IMM, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ, RangeScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangess, INS_vrangesd}, HW_Category_IMM, HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512DQ, Reduce, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreduceps, INS_vreducepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ, ReduceScalar, 16, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreducess, INS_vreducesd}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_CopyUpperBits)
+HARDWARE_INTRINSIC(AVX512DQ, Xor, 64, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_xorps, INS_xorpd}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512DQ.VL Intrinsics
-HARDWARE_INTRINSIC(AVX512DQ_VL, BroadcastPairScalarToVector128, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ_VL, BroadcastPairScalarToVector256, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_vbroadcastf32x2, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Double, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Int64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Int64WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Single, -1, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2ps, INS_vcvtuqq2ps, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128UInt64, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128UInt64WithTruncation, 16, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Double, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Int64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Int64WithTruncation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256UInt64, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256UInt64WithTruncation, 32, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
-HARDWARE_INTRINSIC(AVX512DQ_VL, Range, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangeps, INS_vrangepd}, HW_Category_IMM, HW_Flag_NoFlag)
-HARDWARE_INTRINSIC(AVX512DQ_VL, Reduce, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreduceps, INS_vreducepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
-HARDWARE_INTRINSIC(AVX512DQ_VL, MultiplyLow, -1, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmullq, INS_vpmullq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
+HARDWARE_INTRINSIC(AVX512DQ_VL, BroadcastPairScalarToVector128, 16, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ_VL, BroadcastPairScalarToVector256, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vbroadcasti32x2, INS_vbroadcasti32x2, INS_invalid, INS_invalid, INS_vbroadcastf32x2, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Double, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Int64, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Int64WithTruncation, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128Single, -1, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2ps, INS_vcvtuqq2ps, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128UInt64, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector128UInt64WithTruncation, 16, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Double, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtqq2pd, INS_vcvtuqq2pd, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Int64, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2qq, INS_vcvtpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256Int64WithTruncation, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2qq, INS_vcvttpd2qq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256UInt64, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvtps2uqq, INS_vcvtpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, ConvertToVector256UInt64WithTruncation, 32, 1, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vcvttps2uqq, INS_vcvttpd2uqq}, HW_Category_SimpleSIMD, HW_Flag_BaseTypeFromFirstArg)
+HARDWARE_INTRINSIC(AVX512DQ_VL, Range, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vrangeps, INS_vrangepd}, HW_Category_IMM, HW_Flag_NoFlag)
+HARDWARE_INTRINSIC(AVX512DQ_VL, Reduce, -1, 2, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vreduceps, INS_vreducepd}, HW_Category_IMM, HW_Flag_FullRangeIMM)
+HARDWARE_INTRINSIC(AVX512DQ_VL, MultiplyLow, -1, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vpmullq, INS_vpmullq, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_Commutative)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512VBMI Intrinsics
-HARDWARE_INTRINSIC(AVX512VBMI, PermuteVar64x8, 64, 2, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512VBMI, PermuteVar64x8x2, 64, 3, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512VBMI, PermuteVar64x8, 64, 2, false, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512VBMI, PermuteVar64x8x2, 64, 3, false, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVX512VBMI.VL Intrinsics
-HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar16x8, 16, 2, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar16x8x2, 16, 3, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar32x8, 32, 2, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
-HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar32x8x2, 32, 3, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar16x8, 16, 2, false, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar16x8x2, 16, 3, false, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar32x8, 32, 2, false, {INS_vpermb, INS_vpermb, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialImport)
+HARDWARE_INTRINSIC(AVX512VBMI_VL, PermuteVar32x8x2, 32, 3, false, {INS_vpermt2b, INS_vpermt2b, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_PermuteVar2x|HW_Flag_RmwIntrinsic)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AVXVNNI Intrinsics
-HARDWARE_INTRINSIC(AVXVNNI, MultiplyWideningAndAdd, -1, 3, {INS_invalid, INS_vpdpbusd, INS_vpdpwssd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg)
-HARDWARE_INTRINSIC(AVXVNNI, MultiplyWideningAndAddSaturate, -1, 3, {INS_invalid, INS_vpdpbusds, INS_vpdpwssds, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVXVNNI, MultiplyWideningAndAdd, -1, 3, true, {INS_invalid, INS_vpdpbusd, INS_vpdpwssd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg)
+HARDWARE_INTRINSIC(AVXVNNI, MultiplyWideningAndAddSaturate, -1, 3, true, {INS_invalid, INS_vpdpbusds, INS_vpdpwssds, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoFloatingPointUsed|HW_Flag_BaseTypeFromSecondArg)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// AES Intrinsics
-HARDWARE_INTRINSIC(AES, Decrypt, 16, 2, {INS_invalid, INS_aesdec, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AES, DecryptLast, 16, 2, {INS_invalid, INS_aesdeclast, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AES, Encrypt, 16, 2, {INS_invalid, INS_aesenc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AES, EncryptLast, 16, 2, {INS_invalid, INS_aesenclast, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AES, InverseMixColumns, 16, 1, {INS_invalid, INS_aesimc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AES, KeygenAssist, 16, 2, {INS_invalid, INS_aeskeygenassist, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, Decrypt, 16, 2, false, {INS_invalid, INS_aesdec, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, DecryptLast, 16, 2, false, {INS_invalid, INS_aesdeclast, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, Encrypt, 16, 2, false, {INS_invalid, INS_aesenc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, EncryptLast, 16, 2, false, {INS_invalid, INS_aesenclast, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, InverseMixColumns, 16, 1, false, {INS_invalid, INS_aesimc, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AES, KeygenAssist, 16, 2, false, {INS_invalid, INS_aeskeygenassist, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// BMI1 Intrinsics
-HARDWARE_INTRINSIC(BMI1, AndNot, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI1, ExtractLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsi, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI1, GetMaskUpToLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsmsk, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(BMI1, ResetLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(BMI1, TrailingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_tzcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns)
-HARDWARE_INTRINSIC(BMI1, BitFieldExtract, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bextr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1, AndNot, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andn, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1, ExtractLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsi, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1, GetMaskUpToLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsmsk, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(BMI1, ResetLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(BMI1, TrailingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_tzcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(BMI1, BitFieldExtract, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bextr, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// BMI1 Intrinsics
-HARDWARE_INTRINSIC(BMI1_X64, AndNot, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andn, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI1_X64, ExtractLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsi, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI1_X64, GetMaskUpToLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsmsk, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(BMI1_X64, ResetLowestSetBit, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
-HARDWARE_INTRINSIC(BMI1_X64, TrailingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_tzcnt, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns)
-HARDWARE_INTRINSIC(BMI1_X64, BitFieldExtract, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bextr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1_X64, AndNot, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_andn, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1_X64, ExtractLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsi, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI1_X64, GetMaskUpToLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsmsk, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(BMI1_X64, ResetLowestSetBit, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_blsr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed)
+HARDWARE_INTRINSIC(BMI1_X64, TrailingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_tzcnt, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(BMI1_X64, BitFieldExtract, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bextr, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_MultiIns|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// BMI2 Intrinsics
-HARDWARE_INTRINSIC(BMI2, ParallelBitDeposit, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pdep, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2, ParallelBitExtract, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pext, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2, ZeroHighBits, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bzhi, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2, MultiplyNoFlags, 0, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoContainment|HW_Flag_MaybeMemoryStore|HW_Flag_MultiIns|HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2, ParallelBitDeposit, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pdep, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2, ParallelBitExtract, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pext, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2, ZeroHighBits, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bzhi, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2, MultiplyNoFlags, 0, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulx, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoContainment|HW_Flag_MaybeMemoryStore|HW_Flag_MultiIns|HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// BMI2 Intrinsics
-HARDWARE_INTRINSIC(BMI2_X64, ParallelBitDeposit, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pdep, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2_X64, ParallelBitExtract, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pext, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2_X64, ZeroHighBits, 0, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bzhi, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(BMI2_X64, MultiplyNoFlags, 0, -1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulx, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoContainment|HW_Flag_MaybeMemoryStore|HW_Flag_MultiIns|HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2_X64, ParallelBitDeposit, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pdep, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2_X64, ParallelBitExtract, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pext, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2_X64, ZeroHighBits, 0, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_bzhi, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_SpecialImport|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(BMI2_X64, MultiplyNoFlags, 0, -1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_mulx, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoContainment|HW_Flag_MaybeMemoryStore|HW_Flag_MultiIns|HW_Flag_NoFloatingPointUsed|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// FMA Intrinsics
-HARDWARE_INTRINSIC(FMA, MultiplyAdd, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ps, INS_vfmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplyAddNegated, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ps, INS_vfnmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplyAddNegatedScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ss, INS_vfnmadd213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplyAddScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ss, INS_vfmadd213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplyAddSubtract, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmaddsub213ps, INS_vfmaddsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplySubtract, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ps, INS_vfmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplySubtractAdd, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsubadd213ps, INS_vfmsubadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplySubtractNegated, -1, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ps, INS_vfnmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplySubtractScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ss, INS_vfmsub213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
-HARDWARE_INTRINSIC(FMA, MultiplySubtractNegatedScalar, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ss, INS_vfnmsub213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplyAdd, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ps, INS_vfmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplyAddNegated, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ps, INS_vfnmadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplyAddNegatedScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmadd213ss, INS_vfnmadd213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplyAddScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmadd213ss, INS_vfmadd213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplyAddSubtract, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmaddsub213ps, INS_vfmaddsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplySubtract, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ps, INS_vfmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplySubtractAdd, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsubadd213ps, INS_vfmsubadd213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplySubtractNegated, -1, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ps, INS_vfnmsub213pd}, HW_Category_SimpleSIMD, HW_Flag_SpecialCodeGen|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplySubtractScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfmsub213ss, INS_vfmsub213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
+HARDWARE_INTRINSIC(FMA, MultiplySubtractNegatedScalar, 16, 3, true, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_vfnmsub213ss, INS_vfnmsub213sd}, HW_Category_SIMDScalar, HW_Flag_SpecialCodeGen|HW_Flag_CopyUpperBits|HW_Flag_FmaIntrinsic|HW_Flag_RmwIntrinsic)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// LZCNT Intrinsics
-HARDWARE_INTRINSIC(LZCNT, LeadingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_lzcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(LZCNT, LeadingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_lzcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// LZCNT Intrinsics
-HARDWARE_INTRINSIC(LZCNT_X64, LeadingZeroCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_lzcnt, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(LZCNT_X64, LeadingZeroCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_lzcnt, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// PCLMULQDQ Intrinsics
-HARDWARE_INTRINSIC(PCLMULQDQ, CarrylessMultiply, 16, 3, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pclmulqdq, INS_pclmulqdq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(PCLMULQDQ, CarrylessMultiply, 16, 3, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_pclmulqdq, INS_pclmulqdq, INS_invalid, INS_invalid}, HW_Category_IMM, HW_Flag_FullRangeIMM|HW_Flag_NoEvexSemantics)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// POPCNT Intrinsics
-HARDWARE_INTRINSIC(POPCNT, PopCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_popcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(POPCNT, PopCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_popcnt, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// POPCNT Intrinsics
-HARDWARE_INTRINSIC(POPCNT_X64, PopCount, 0, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_popcnt, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
+HARDWARE_INTRINSIC(POPCNT_X64, PopCount, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_popcnt, INS_invalid, INS_invalid}, HW_Category_Scalar, HW_Flag_NoFloatingPointUsed|HW_Flag_NoRMWSemantics|HW_Flag_MultiIns)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// X86Serialize Intrinsics
-HARDWARE_INTRINSIC(X86Serialize, Serialize, 0, 0, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
+HARDWARE_INTRINSIC(X86Serialize, Serialize, 0, 0, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment|HW_Flag_NoRMWSemantics|HW_Flag_SpecialSideEffect_Barrier)
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
-// ISA Function name SIMD size NumArg Instructions Category Flags
-// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
+// ISA Function name SIMD size NumArg EncodesExtraTypeArg Instructions Category Flags
+// {TYP_BYTE, TYP_UBYTE, TYP_SHORT, TYP_USHORT, TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG, TYP_FLOAT, TYP_DOUBLE}
// ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
// Special intrinsics that are generated during lowering
-HARDWARE_INTRINSIC(SSE, COMISS, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE, UCOMISS, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, COMISD, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE2, UCOMISD, 16, 2, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(SSE41, PTEST, 16, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
-HARDWARE_INTRINSIC(AVX, PTEST, 0, 2, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(SSE, COMISS, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE, UCOMISS, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomiss, INS_invalid}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, COMISD, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_comisd}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE2, UCOMISD, 16, 2, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_ucomisd}, HW_Category_SIMDScalar, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(SSE41, PTEST, 16, 2, false, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_invalid, INS_invalid}, HW_Category_SimpleSIMD, HW_Flag_NoRMWSemantics|HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX, PTEST, 0, 2, false, {INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_ptest, INS_vtestps, INS_vtestpd}, HW_Category_SimpleSIMD, HW_Flag_NoEvexSemantics)
+HARDWARE_INTRINSIC(AVX512F, KORTEST, 0, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment)
-HARDWARE_INTRINSIC(AVX512F, MoveMaskSpecial, 64, 1, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_movd, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, CompareEqualSpecial, 64, 2, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, CompareGreaterThanOrEqualSpecial, 64, 2, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, CompareGreaterThanSpecial, 64, 2, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, CompareLessThanOrEqualSpecial, 64, 2, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, CompareLessThanSpecial, 64, 2, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, MoveMaskToVectorSpecial, 64, 1, {INS_vpmovm2b, INS_vpmovm2b, INS_vpmovm2w, INS_vpmovm2w, INS_vpmovm2d, INS_vpmovm2d, INS_vpmovm2q, INS_vpmovm2q, INS_vpmovm2d, INS_vpmovm2q}, HW_Category_Special, HW_Flag_BaseTypeFromFirstArg|HW_Flag_NoContainment|HW_Flag_SpecialCodeGen|HW_Flag_NoRMWSemantics)
-HARDWARE_INTRINSIC(AVX512F, KORTEST, 0, 1, {INS_kortestq, INS_kortestq, INS_kortestd, INS_kortestd, INS_kortestw, INS_kortestw, INS_kortestb, INS_kortestb, INS_kortestw, INS_kortestb}, HW_Category_Special, HW_Flag_NoRMWSemantics)
+HARDWARE_INTRINSIC(AVX512F, CompareEqualMask, -1, 2, true, {INS_vpcmpeqb, INS_vpcmpeqb, INS_vpcmpeqw, INS_vpcmpeqw, INS_vpcmpeqd, INS_vpcmpeqd, INS_vpcmpeqq, INS_vpcmpeqq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, CompareGreaterThanMask, -1, 2, true, {INS_vpcmpgtb, INS_vpcmpub, INS_vpcmpgtw, INS_vpcmpuw, INS_vpcmpgtd, INS_vpcmpud, INS_vpcmpgtq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, CompareGreaterThanOrEqualMask, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, CompareLessThanMask, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, CompareLessThanOrEqualMask, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, CompareNotEqualMask, -1, 2, true, {INS_vpcmpb, INS_vpcmpub, INS_vpcmpw, INS_vpcmpuw, INS_vpcmpd, INS_vpcmpud, INS_vpcmpq, INS_vpcmpuq, INS_vcmpps, INS_vcmppd}, HW_Category_SimpleSIMD, HW_Flag_Commutative|HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, ConvertMaskToVector, -1, 1, true, {INS_vpmovm2b, INS_vpmovm2b, INS_vpmovm2w, INS_vpmovm2w, INS_vpmovm2d, INS_vpmovm2d, INS_vpmovm2q, INS_vpmovm2q, INS_vpmovm2d, INS_vpmovm2q}, HW_Category_SimpleSIMD, HW_Flag_ReturnsPerElementMask)
+HARDWARE_INTRINSIC(AVX512F, MoveMaskSpecial, -1, 1, false, {INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid, INS_invalid}, HW_Category_Special, HW_Flag_NoContainment)
#endif // FEATURE_HW_INTRINSIC
diff --git a/src/coreclr/jit/hwintrinsicxarch.cpp b/src/coreclr/jit/hwintrinsicxarch.cpp
index 379509a17180e6..dadabaddbb9491 100644
--- a/src/coreclr/jit/hwintrinsicxarch.cpp
+++ b/src/coreclr/jit/hwintrinsicxarch.cpp
@@ -553,6 +553,302 @@ bool HWIntrinsicInfo::isScalarIsa(CORINFO_InstructionSet isa)
}
}
+//------------------------------------------------------------------------
+// lookupIval: Gets a the implicit immediate value for the given intrinsic
+//
+// Arguments:
+// comp - The compiler
+// id - The intrinsic for which to get the ival
+// simdBaseType - The base type for the intrinsic
+//
+// Return Value:
+// The immediate value for the given intrinsic or -1 if none exists
+int HWIntrinsicInfo::lookupIval(Compiler* comp, NamedIntrinsic id, var_types simdBaseType)
+{
+ switch (id)
+ {
+ case NI_SSE_CompareEqual:
+ case NI_SSE_CompareScalarEqual:
+ case NI_SSE2_CompareEqual:
+ case NI_SSE2_CompareScalarEqual:
+ case NI_AVX_CompareEqual:
+ case NI_AVX512F_CompareEqualMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ return static_cast(FloatComparisonMode::OrderedEqualNonSignaling);
+ }
+ else
+ {
+ // We can emit `vpcmpeqb`, `vpcmpeqw`, `vpcmpeqd`, or `vpcmpeqq`
+ }
+ break;
+ }
+
+ case NI_SSE_CompareGreaterThan:
+ case NI_SSE_CompareScalarGreaterThan:
+ case NI_SSE2_CompareGreaterThan:
+ case NI_SSE2_CompareScalarGreaterThan:
+ case NI_AVX_CompareGreaterThan:
+ case NI_AVX512F_CompareGreaterThanMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ if (comp->compOpportunisticallyDependsOn(InstructionSet_AVX))
+ {
+ return static_cast(FloatComparisonMode::OrderedGreaterThanSignaling);
+ }
+
+ // CompareGreaterThan is not directly supported in hardware without AVX support.
+ // We will return the inverted case here and lowering will itself swap the ops
+ // to ensure the emitted code remains correct. This simplifies the overall logic
+ // here and for other use cases.
+
+ assert(id != NI_AVX_CompareGreaterThan);
+ return static_cast(FloatComparisonMode::OrderedLessThanSignaling);
+ }
+ else if ((id == NI_AVX512F_CompareGreaterThanMask) && varTypeIsUnsigned(simdBaseType))
+ {
+ // TODO-XARCH-CQ: Allow the other integer paths to use the EVEX encoding
+ return static_cast(IntComparisonMode::GreaterThan);
+ }
+ break;
+ }
+
+ case NI_SSE_CompareLessThan:
+ case NI_SSE_CompareScalarLessThan:
+ case NI_SSE2_CompareLessThan:
+ case NI_SSE2_CompareScalarLessThan:
+ case NI_AVX_CompareLessThan:
+ case NI_AVX512F_CompareLessThanMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ return static_cast(FloatComparisonMode::OrderedLessThanSignaling);
+ }
+ else if (id == NI_AVX512F_CompareLessThanMask)
+ {
+ // TODO-XARCH-CQ: Allow the other integer paths to use the EVEX encoding
+ return static_cast(IntComparisonMode::LessThan);
+ }
+ break;
+ }
+
+ case NI_SSE_CompareGreaterThanOrEqual:
+ case NI_SSE_CompareScalarGreaterThanOrEqual:
+ case NI_SSE2_CompareGreaterThanOrEqual:
+ case NI_SSE2_CompareScalarGreaterThanOrEqual:
+ case NI_AVX_CompareGreaterThanOrEqual:
+ case NI_AVX512F_CompareGreaterThanOrEqualMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ if (comp->compOpportunisticallyDependsOn(InstructionSet_AVX))
+ {
+ return static_cast(FloatComparisonMode::OrderedGreaterThanOrEqualSignaling);
+ }
+
+ // CompareGreaterThanOrEqual is not directly supported in hardware without AVX support.
+ // We will return the inverted case here and lowering will itself swap the ops
+ // to ensure the emitted code remains correct. This simplifies the overall logic
+ // here and for other use cases.
+
+ assert(id != NI_AVX_CompareGreaterThanOrEqual);
+ return static_cast(FloatComparisonMode::OrderedLessThanOrEqualSignaling);
+ }
+ else
+ {
+ assert(id == NI_AVX512F_CompareGreaterThanOrEqualMask);
+ return static_cast(IntComparisonMode::GreaterThanOrEqual);
+ }
+ break;
+ }
+
+ case NI_SSE_CompareLessThanOrEqual:
+ case NI_SSE_CompareScalarLessThanOrEqual:
+ case NI_SSE2_CompareLessThanOrEqual:
+ case NI_SSE2_CompareScalarLessThanOrEqual:
+ case NI_AVX_CompareLessThanOrEqual:
+ case NI_AVX512F_CompareLessThanOrEqualMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ return static_cast(FloatComparisonMode::OrderedLessThanOrEqualSignaling);
+ }
+ else
+ {
+ assert(id == NI_AVX512F_CompareLessThanOrEqualMask);
+ return static_cast(IntComparisonMode::LessThanOrEqual);
+ }
+ break;
+ }
+
+ case NI_SSE_CompareNotEqual:
+ case NI_SSE_CompareScalarNotEqual:
+ case NI_SSE2_CompareNotEqual:
+ case NI_SSE2_CompareScalarNotEqual:
+ case NI_AVX_CompareNotEqual:
+ case NI_AVX512F_CompareNotEqualMask:
+ {
+ if (varTypeIsFloating(simdBaseType))
+ {
+ return static_cast(FloatComparisonMode::UnorderedNotEqualNonSignaling);
+ }
+ else
+ {
+ assert(id == NI_AVX512F_CompareNotEqualMask);
+ return static_cast(IntComparisonMode::NotEqual);
+ }
+ break;
+ }
+
+ case NI_SSE_CompareNotGreaterThan:
+ case NI_SSE_CompareScalarNotGreaterThan:
+ case NI_SSE2_CompareNotGreaterThan:
+ case NI_SSE2_CompareScalarNotGreaterThan:
+ case NI_AVX_CompareNotGreaterThan:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+
+ if (comp->compOpportunisticallyDependsOn(InstructionSet_AVX))
+ {
+ return static_cast(FloatComparisonMode::UnorderedNotGreaterThanSignaling);
+ }
+
+ // CompareNotGreaterThan is not directly supported in hardware without AVX support.
+ // We will return the inverted case here and lowering will itself swap the ops
+ // to ensure the emitted code remains correct. This simplifies the overall logic
+ // here and for other use cases.
+
+ assert(id != NI_AVX_CompareNotGreaterThan);
+ return static_cast(FloatComparisonMode::UnorderedNotLessThanSignaling);
+ }
+
+ case NI_SSE_CompareNotLessThan:
+ case NI_SSE_CompareScalarNotLessThan:
+ case NI_SSE2_CompareNotLessThan:
+ case NI_SSE2_CompareScalarNotLessThan:
+ case NI_AVX_CompareNotLessThan:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatComparisonMode::UnorderedNotLessThanSignaling);
+ }
+
+ case NI_SSE_CompareNotGreaterThanOrEqual:
+ case NI_SSE_CompareScalarNotGreaterThanOrEqual:
+ case NI_SSE2_CompareNotGreaterThanOrEqual:
+ case NI_SSE2_CompareScalarNotGreaterThanOrEqual:
+ case NI_AVX_CompareNotGreaterThanOrEqual:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+
+ if (comp->compOpportunisticallyDependsOn(InstructionSet_AVX))
+ {
+ return static_cast(FloatComparisonMode::UnorderedNotGreaterThanOrEqualSignaling);
+ }
+
+ // CompareNotGreaterThanOrEqual is not directly supported in hardware without AVX support.
+ // We will return the inverted case here and lowering will itself swap the ops
+ // to ensure the emitted code remains correct. This simplifies the overall logic
+ // here and for other use cases.
+
+ assert(id != NI_AVX_CompareNotGreaterThanOrEqual);
+ return static_cast(FloatComparisonMode::UnorderedNotLessThanOrEqualSignaling);
+ }
+
+ case NI_SSE_CompareNotLessThanOrEqual:
+ case NI_SSE_CompareScalarNotLessThanOrEqual:
+ case NI_SSE2_CompareNotLessThanOrEqual:
+ case NI_SSE2_CompareScalarNotLessThanOrEqual:
+ case NI_AVX_CompareNotLessThanOrEqual:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatComparisonMode::UnorderedNotLessThanOrEqualSignaling);
+ }
+
+ case NI_SSE_CompareOrdered:
+ case NI_SSE_CompareScalarOrdered:
+ case NI_SSE2_CompareOrdered:
+ case NI_SSE2_CompareScalarOrdered:
+ case NI_AVX_CompareOrdered:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatComparisonMode::OrderedNonSignaling);
+ }
+
+ case NI_SSE_CompareUnordered:
+ case NI_SSE_CompareScalarUnordered:
+ case NI_SSE2_CompareUnordered:
+ case NI_SSE2_CompareScalarUnordered:
+ case NI_AVX_CompareUnordered:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatComparisonMode::UnorderedNonSignaling);
+ }
+
+ case NI_SSE41_Ceiling:
+ case NI_SSE41_CeilingScalar:
+ case NI_AVX_Ceiling:
+ {
+ FALLTHROUGH;
+ }
+
+ case NI_SSE41_RoundToPositiveInfinity:
+ case NI_SSE41_RoundToPositiveInfinityScalar:
+ case NI_AVX_RoundToPositiveInfinity:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatRoundingMode::ToPositiveInfinity);
+ }
+
+ case NI_SSE41_Floor:
+ case NI_SSE41_FloorScalar:
+ case NI_AVX_Floor:
+ {
+ FALLTHROUGH;
+ }
+
+ case NI_SSE41_RoundToNegativeInfinity:
+ case NI_SSE41_RoundToNegativeInfinityScalar:
+ case NI_AVX_RoundToNegativeInfinity:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatRoundingMode::ToNegativeInfinity);
+ }
+
+ case NI_SSE41_RoundCurrentDirection:
+ case NI_SSE41_RoundCurrentDirectionScalar:
+ case NI_AVX_RoundCurrentDirection:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatRoundingMode::CurrentDirection);
+ }
+
+ case NI_SSE41_RoundToNearestInteger:
+ case NI_SSE41_RoundToNearestIntegerScalar:
+ case NI_AVX_RoundToNearestInteger:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatRoundingMode::ToNearestInteger);
+ }
+
+ case NI_SSE41_RoundToZero:
+ case NI_SSE41_RoundToZeroScalar:
+ case NI_AVX_RoundToZero:
+ {
+ assert(varTypeIsFloating(simdBaseType));
+ return static_cast(FloatRoundingMode::ToZero);
+ }
+
+ default:
+ {
+ break;
+ }
+ }
+
+ return -1;
+}
+
//------------------------------------------------------------------------
// impNonConstFallback: convert certain SSE2/AVX2 shift intrinsic to its semantic alternative when the imm-arg is
// not a compile-time constant
@@ -783,10 +1079,8 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
return impSpecialIntrinsic(NI_Vector128_ToVector256, clsHnd, method, sig, simdBaseJitType, retType,
simdSize);
}
- else
+ else if (vectorTByteLength == XMM_REGSIZE_BYTES)
{
- assert(vectorTByteLength == XMM_REGSIZE_BYTES);
-
// We fold away the cast here, as it only exists to satisfy
// the type system. It is safe to do this here since the retNode type
// and the signature return type are both the same TYP_SIMD.
@@ -795,6 +1089,10 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
SetOpLclRelatedToSIMDIntrinsic(retNode);
assert(retNode->gtType == getSIMDTypeForSize(getSIMDTypeSizeInBytes(sig->retTypeSigClass)));
}
+ else
+ {
+ assert(vectorTByteLength == 0);
+ }
break;
}
@@ -919,10 +1217,8 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
break;
}
- else
+ else if (vectorTByteLength == XMM_REGSIZE_BYTES)
{
- assert(vectorTByteLength == XMM_REGSIZE_BYTES);
-
if (compExactlyDependsOn(InstructionSet_AVX))
{
// We support Vector256 but Vector is only 16-bytes, so we should
@@ -941,6 +1237,10 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
}
}
}
+ else
+ {
+ assert(vectorTByteLength == 0);
+ }
break;
}
@@ -969,10 +1269,8 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
}
break;
}
- else
+ else if (vectorTByteLength == XMM_REGSIZE_BYTES)
{
- assert(vectorTByteLength == XMM_REGSIZE_BYTES);
-
if (compExactlyDependsOn(InstructionSet_AVX512F))
{
// We support Vector512 but Vector is only 16-bytes, so we should
@@ -991,6 +1289,10 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
}
}
}
+ else
+ {
+ assert(vectorTByteLength == 0);
+ }
break;
}
@@ -2865,10 +3167,9 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
// These intrinsics are "special import" because the non-AVX path isn't directly
// hardware supported. Instead, they start with "swapped operands" and we fix that here.
- FloatComparisonMode comparison =
- static_cast(HWIntrinsicInfo::lookupIval(intrinsic, true));
- retNode = gtNewSimdHWIntrinsicNode(TYP_SIMD16, op1, op2, gtNewIconNode(static_cast(comparison)),
- NI_AVX_CompareScalar, simdBaseJitType, simdSize);
+ int ival = HWIntrinsicInfo::lookupIval(this, intrinsic, simdBaseType);
+ retNode = gtNewSimdHWIntrinsicNode(TYP_SIMD16, op1, op2, gtNewIconNode(ival), NI_AVX_CompareScalar,
+ simdBaseJitType, simdSize);
}
else
{
@@ -2925,10 +3226,9 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
// These intrinsics are "special import" because the non-AVX path isn't directly
// hardware supported. Instead, they start with "swapped operands" and we fix that here.
- FloatComparisonMode comparison =
- static_cast(HWIntrinsicInfo::lookupIval(intrinsic, true));
- retNode = gtNewSimdHWIntrinsicNode(TYP_SIMD16, op1, op2, gtNewIconNode(static_cast(comparison)),
- NI_AVX_CompareScalar, simdBaseJitType, simdSize);
+ int ival = HWIntrinsicInfo::lookupIval(this, intrinsic, simdBaseType);
+ retNode = gtNewSimdHWIntrinsicNode(TYP_SIMD16, op1, op2, gtNewIconNode(ival), NI_AVX_CompareScalar,
+ simdBaseJitType, simdSize);
}
else
{
diff --git a/src/coreclr/jit/importercalls.cpp b/src/coreclr/jit/importercalls.cpp
index 7f55bdd92ef8d7..4afb8c55535495 100644
--- a/src/coreclr/jit/importercalls.cpp
+++ b/src/coreclr/jit/importercalls.cpp
@@ -5584,25 +5584,25 @@ void Compiler::addFatPointerCandidate(GenTreeCall* call)
// pickGDV: Use profile information to pick a GDV candidate for a call site.
//
// Arguments:
-// call - the call
-// ilOffset - exact IL offset of the call
-// isInterface - whether or not the call target is defined on an interface
-// classGuess - [out] the class to guess for (mutually exclusive with methodGuess)
-// methodGuess - [out] the method to guess for (mutually exclusive with classGuess)
-// likelihood - [out] an estimate of the likelihood that the guess will succeed
+// call - the call
+// ilOffset - exact IL offset of the call
+// isInterface - whether or not the call target is defined on an interface
+// classGuesses - [out] the classes to guess for (mutually exclusive with methodGuess)
+// methodGuesses - [out] the methods to guess for (mutually exclusive with classGuess)
+// candidatesCount - [out] number of guesses
+// likelihoods - [out] estimates of the likelihoods that the guesses will succeed
//
void Compiler::pickGDV(GenTreeCall* call,
IL_OFFSET ilOffset,
bool isInterface,
- CORINFO_CLASS_HANDLE* classGuess,
- CORINFO_METHOD_HANDLE* methodGuess,
- unsigned* likelihood)
+ CORINFO_CLASS_HANDLE* classGuesses,
+ CORINFO_METHOD_HANDLE* methodGuesses,
+ int* candidatesCount,
+ unsigned* likelihoods)
{
- *classGuess = NO_CLASS_HANDLE;
- *methodGuess = NO_METHOD_HANDLE;
- *likelihood = 0;
+ *candidatesCount = 0;
- const int maxLikelyClasses = 32;
+ const int maxLikelyClasses = MAX_GDV_TYPE_CHECKS;
LikelyClassMethodRecord likelyClasses[maxLikelyClasses];
unsigned numberOfClasses = 0;
if (call->IsVirtualStub() || call->IsVirtualVtable())
@@ -5611,7 +5611,7 @@ void Compiler::pickGDV(GenTreeCall* call,
getLikelyClasses(likelyClasses, maxLikelyClasses, fgPgoSchema, fgPgoSchemaCount, fgPgoData, ilOffset);
}
- const int maxLikelyMethods = 32;
+ const int maxLikelyMethods = MAX_GDV_TYPE_CHECKS;
LikelyClassMethodRecord likelyMethods[maxLikelyMethods];
unsigned numberOfMethods = 0;
@@ -5702,16 +5702,26 @@ void Compiler::pickGDV(GenTreeCall* call,
unsigned index = static_cast(random->Next(static_cast(numberOfClasses + numberOfMethods)));
if (index < numberOfClasses)
{
- *classGuess = (CORINFO_CLASS_HANDLE)likelyClasses[index].handle;
- *likelihood = 100;
- JITDUMP("Picked random class for GDV: %p (%s)\n", *classGuess, eeGetClassName(*classGuess));
+ classGuesses[0] = (CORINFO_CLASS_HANDLE)likelyClasses[index].handle;
+ likelihoods[0] = 100;
+ *candidatesCount = 1;
+ // TODO: report multiple random candidates. For now we don't do it because with the current impl
+ // we might give up on all candidates if one of them is not inlinable, so we don't want to reduce
+ // testing coverage.
+ //
+ JITDUMP("Picked random class for GDV: %p (%s)\n", classGuesses[0], eeGetClassName(classGuesses[0]));
return;
}
else
{
- *methodGuess = (CORINFO_METHOD_HANDLE)likelyMethods[index - numberOfClasses].handle;
- *likelihood = 100;
- JITDUMP("Picked random method for GDV: %p (%s)\n", *methodGuess, eeGetMethodFullName(*methodGuess));
+ methodGuesses[0] = (CORINFO_METHOD_HANDLE)likelyMethods[index - numberOfClasses].handle;
+ likelihoods[0] = 100;
+ *candidatesCount = 1;
+ // TODO: report multiple random candidates. For now we don't do it because with the current impl
+ // we might give up on all candidates if one of them is not inlinable, so we don't want to reduce
+ // testing coverage.
+ //
+ JITDUMP("Picked random method for GDV: %p (%s)\n", methodGuesses[0], eeGetMethodFullName(methodGuesses[0]));
return;
}
}
@@ -5720,25 +5730,68 @@ void Compiler::pickGDV(GenTreeCall* call,
// Prefer class guess as it is cheaper
if (numberOfClasses > 0)
{
- unsigned likelihoodThreshold = isInterface ? 25 : 30;
- if (likelyClasses[0].likelihood >= likelihoodThreshold)
+ const int maxNumberOfGuesses = min(MAX_GDV_TYPE_CHECKS, JitConfig.JitGuardedDevirtualizationMaxTypeChecks());
+ if (maxNumberOfGuesses == 0)
{
- *classGuess = (CORINFO_CLASS_HANDLE)likelyClasses[0].handle;
- *likelihood = likelyClasses[0].likelihood;
+ // DOTNET_JitGuardedDevirtualizationMaxTypeChecks=0 means we don't want to do any guarded devirtualization
+ // Although, we expect users to disable GDV by setting DOTNET_JitEnableGuardedDevirtualization=0
return;
}
- JITDUMP("Not guessing for class; likelihood is below %s call threshold %u\n",
- isInterface ? "interface" : "virtual", likelihoodThreshold);
+ assert((maxNumberOfGuesses > 0) && (maxNumberOfGuesses <= MAX_GDV_TYPE_CHECKS));
+
+ unsigned likelihoodThreshold;
+ if (maxNumberOfGuesses == 1)
+ {
+ // We're allowed to make only a single guess - it means we want to work only with dominating types
+ likelihoodThreshold = isInterface ? 25 : 30;
+ }
+ else if (maxNumberOfGuesses == 2)
+ {
+ // Two guesses - slightly relax the thresholds
+ likelihoodThreshold = isInterface ? 15 : 20;
+ }
+ else
+ {
+ // We're allowed to make more than 2 guesses - pick all types with likelihood >= 10%
+ likelihoodThreshold = 10;
+ }
+
+ // We have 'maxNumberOfGuesses' number of classes available
+ // and we're allowed to make 'maxNumberOfGuesses' number of guesses
+ // Iterate over the available classes to find classes with likelihoods bigger than
+ // a specific threshold
+ //
+ assert(*candidatesCount == 0);
+ unsigned totalGuesses = min((unsigned)maxNumberOfGuesses, numberOfClasses);
+ for (unsigned guessIdx = 0; guessIdx < totalGuesses; guessIdx++)
+ {
+ if (likelyClasses[guessIdx].likelihood >= likelihoodThreshold)
+ {
+ classGuesses[guessIdx] = (CORINFO_CLASS_HANDLE)likelyClasses[guessIdx].handle;
+ likelihoods[guessIdx] = likelyClasses[guessIdx].likelihood;
+ *candidatesCount = *candidatesCount + 1;
+ JITDUMP("Accepting type %s with likelihood %u as a candidate\n", eeGetClassName(classGuesses[guessIdx]),
+ likelihoods[guessIdx])
+ }
+ else
+ {
+ // The candidates are sorted by likelihood so the rest of the
+ // guesses will have even lower likelihoods
+ break;
+ }
+ }
}
if (numberOfMethods > 0)
{
+ // For method guessing we only support a single target for now
unsigned likelihoodThreshold = 30;
if (likelyMethods[0].likelihood >= likelihoodThreshold)
{
- *methodGuess = (CORINFO_METHOD_HANDLE)likelyMethods[0].handle;
- *likelihood = likelyMethods[0].likelihood;
+ methodGuesses[0] = (CORINFO_METHOD_HANDLE)likelyMethods[0].handle;
+ likelihoods[0] = likelyMethods[0].likelihood;
+ *candidatesCount = 1;
return;
}
@@ -5862,9 +5915,10 @@ void Compiler::considerGuardedDevirtualization(GenTreeCall* call,
bool hasPgoData = true;
- CORINFO_CLASS_HANDLE likelyClass = NO_CLASS_HANDLE;
- CORINFO_METHOD_HANDLE likelyMethod = NO_METHOD_HANDLE;
- unsigned likelihood = 0;
+ CORINFO_CLASS_HANDLE likelyClasses[MAX_GDV_TYPE_CHECKS] = {};
+ CORINFO_METHOD_HANDLE likelyMethodes[MAX_GDV_TYPE_CHECKS] = {};
+ unsigned likelihoods[MAX_GDV_TYPE_CHECKS] = {};
+ int candidatesCount = 0;
// We currently only get likely class guesses when there is PGO data
// with class profiles.
@@ -5875,8 +5929,10 @@ void Compiler::considerGuardedDevirtualization(GenTreeCall* call,
}
else
{
- pickGDV(call, ilOffset, isInterface, &likelyClass, &likelyMethod, &likelihood);
- if ((likelyClass == NO_CLASS_HANDLE) && (likelyMethod == NO_METHOD_HANDLE))
+ pickGDV(call, ilOffset, isInterface, likelyClasses, likelyMethodes, &candidatesCount, likelihoods);
+ assert((unsigned)candidatesCount <= MAX_GDV_TYPE_CHECKS);
+ assert((unsigned)candidatesCount <= (unsigned)JitConfig.JitGuardedDevirtualizationMaxTypeChecks());
+ if (candidatesCount == 0)
{
hasPgoData = false;
}
@@ -5933,7 +5989,9 @@ void Compiler::considerGuardedDevirtualization(GenTreeCall* call,
if (!info.compCompHnd->resolveVirtualMethod(&dvInfo))
{
JITDUMP("Can't figure out which method would be invoked, sorry\n");
- return;
+ // Maybe other candidates will be resolved.
+ // Although, we no longer can remove the fallback (we never do it currently anyway)
+ break;
}
CORINFO_METHOD_HANDLE exactMethod = dvInfo.devirtualizedMethod;
@@ -5964,103 +6022,119 @@ void Compiler::considerGuardedDevirtualization(GenTreeCall* call,
return;
}
- uint32_t likelyClassAttribs = 0;
- if (likelyClass != NO_CLASS_HANDLE)
+ // Iterate over the guesses
+ for (int candidateId = 0; candidateId < candidatesCount; candidateId++)
{
- likelyClassAttribs = info.compCompHnd->getClassAttribs(likelyClass);
+ CORINFO_CLASS_HANDLE likelyClass = likelyClasses[candidateId];
+ CORINFO_METHOD_HANDLE likelyMethod = likelyMethodes[candidateId];
+ unsigned likelihood = likelihoods[candidateId];
- if ((likelyClassAttribs & CORINFO_FLG_ABSTRACT) != 0)
+ uint32_t likelyClassAttribs = 0;
+ if (likelyClass != NO_CLASS_HANDLE)
{
- // We may see an abstract likely class, if we have a stale profile.
- // No point guessing for this.
- //
- JITDUMP("Not guessing for class; abstract (stale profile)\n");
- return;
- }
+ likelyClassAttribs = info.compCompHnd->getClassAttribs(likelyClass);
- // Figure out which method will be called.
- //
- CORINFO_DEVIRTUALIZATION_INFO dvInfo;
- dvInfo.virtualMethod = baseMethod;
- dvInfo.objClass = likelyClass;
- dvInfo.context = *pContextHandle;
- dvInfo.exactContext = *pContextHandle;
- dvInfo.pResolvedTokenVirtualMethod = nullptr;
+ if ((likelyClassAttribs & CORINFO_FLG_ABSTRACT) != 0)
+ {
+ // We may see an abstract likely class, if we have a stale profile.
+ // No point guessing for this.
+ //
+ JITDUMP("Not guessing for class; abstract (stale profile)\n");
- const bool canResolve = info.compCompHnd->resolveVirtualMethod(&dvInfo);
+ // Continue checking other candidates, maybe some of them aren't stale.
+ break;
+ }
- if (!canResolve)
- {
- JITDUMP("Can't figure out which method would be invoked, sorry\n");
- return;
- }
+ // Figure out which method will be called.
+ //
+ CORINFO_DEVIRTUALIZATION_INFO dvInfo;
+ dvInfo.virtualMethod = baseMethod;
+ dvInfo.objClass = likelyClass;
+ dvInfo.context = *pContextHandle;
+ dvInfo.exactContext = *pContextHandle;
+ dvInfo.pResolvedTokenVirtualMethod = nullptr;
- likelyMethod = dvInfo.devirtualizedMethod;
- }
+ const bool canResolve = info.compCompHnd->resolveVirtualMethod(&dvInfo);
- uint32_t likelyMethodAttribs = info.compCompHnd->getMethodAttribs(likelyMethod);
+ if (!canResolve)
+ {
+ JITDUMP("Can't figure out which method would be invoked, sorry\n");
- if (likelyClass == NO_CLASS_HANDLE)
- {
- // For method GDV do a few more checks that we get for free in the
- // resolve call above for class-based GDV.
- if ((likelyMethodAttribs & CORINFO_FLG_STATIC) != 0)
- {
- assert((fgPgoSource != ICorJitInfo::PgoSource::Dynamic) || call->IsDelegateInvoke());
- JITDUMP("Cannot currently handle devirtualizing static delegate calls, sorry\n");
- return;
+ // Continue checking other candidates, maybe some of them will succeed.
+ break;
+ }
+
+ likelyMethod = dvInfo.devirtualizedMethod;
}
- CORINFO_CLASS_HANDLE definingClass = info.compCompHnd->getMethodClass(likelyMethod);
- likelyClassAttribs = info.compCompHnd->getClassAttribs(definingClass);
+ uint32_t likelyMethodAttribs = info.compCompHnd->getMethodAttribs(likelyMethod);
- // For instance methods on value classes we need an extended check to
- // check for the unboxing stub. This is NYI.
- // Note: For dynamic PGO likelyMethod above will be the unboxing stub
- // which would fail GDV for other reasons.
- // However, with static profiles or textual PGO input it is still
- // possible that likelyMethod is not the unboxing stub. So we do need
- // this explicit check.
- if ((likelyClassAttribs & CORINFO_FLG_VALUECLASS) != 0)
+ if (likelyClass == NO_CLASS_HANDLE)
{
- JITDUMP("Cannot currently handle devirtualizing delegate calls on value types, sorry\n");
- return;
- }
+ // We don't support multiple candidates for method guessing yet.
+ assert(candidateId == 0);
- // Verify that the call target and args look reasonable so that the JIT
- // does not blow up during inlining/call morphing.
- //
- // NOTE: Once we want to support devirtualization of delegate calls to
- // static methods and remove the check above we will start failing here
- // for delegates pointing to static methods that have the first arg
- // bound. For example:
- //
- // public static void E(this C c) ...
- // Action a = new C().E;
- //
- // The delegate instance looks exactly like one pointing to an instance
- // method in this case and the call will have zero args while the
- // signature has 1 arg.
- //
- if (!isCompatibleMethodGDV(call, likelyMethod))
- {
- JITDUMP("Target for method-based GDV is incompatible (stale profile?)\n");
- assert((fgPgoSource != ICorJitInfo::PgoSource::Dynamic) && "Unexpected stale profile in dynamic PGO data");
- return;
+ // For method GDV do a few more checks that we get for free in the
+ // resolve call above for class-based GDV.
+ if ((likelyMethodAttribs & CORINFO_FLG_STATIC) != 0)
+ {
+ assert((fgPgoSource != ICorJitInfo::PgoSource::Dynamic) || call->IsDelegateInvoke());
+ JITDUMP("Cannot currently handle devirtualizing static delegate calls, sorry\n");
+ break;
+ }
+
+ CORINFO_CLASS_HANDLE definingClass = info.compCompHnd->getMethodClass(likelyMethod);
+ likelyClassAttribs = info.compCompHnd->getClassAttribs(definingClass);
+
+ // For instance methods on value classes we need an extended check to
+ // check for the unboxing stub. This is NYI.
+ // Note: For dynamic PGO likelyMethod above will be the unboxing stub
+ // which would fail GDV for other reasons.
+ // However, with static profiles or textual PGO input it is still
+ // possible that likelyMethod is not the unboxing stub. So we do need
+ // this explicit check.
+ if ((likelyClassAttribs & CORINFO_FLG_VALUECLASS) != 0)
+ {
+ JITDUMP("Cannot currently handle devirtualizing delegate calls on value types, sorry\n");
+ break;
+ }
+
+ // Verify that the call target and args look reasonable so that the JIT
+ // does not blow up during inlining/call morphing.
+ //
+ // NOTE: Once we want to support devirtualization of delegate calls to
+ // static methods and remove the check above we will start failing here
+ // for delegates pointing to static methods that have the first arg
+ // bound. For example:
+ //
+ // public static void E(this C c) ...
+ // Action a = new C().E;
+ //
+ // The delegate instance looks exactly like one pointing to an instance
+ // method in this case and the call will have zero args while the
+ // signature has 1 arg.
+ //
+ if (!isCompatibleMethodGDV(call, likelyMethod))
+ {
+ JITDUMP("Target for method-based GDV is incompatible (stale profile?)\n");
+ assert((fgPgoSource != ICorJitInfo::PgoSource::Dynamic) &&
+ "Unexpected stale profile in dynamic PGO data");
+ break;
+ }
}
- }
#ifdef DEBUG
- char buffer[256];
- JITDUMP("%s call would invoke method %s\n",
- isInterface ? "interface" : call->IsDelegateInvoke() ? "delegate" : "virtual",
- eeGetMethodFullName(likelyMethod, true, true, buffer, sizeof(buffer)));
+ char buffer[256];
+ JITDUMP("%s call would invoke method %s\n",
+ isInterface ? "interface" : call->IsDelegateInvoke() ? "delegate" : "virtual",
+ eeGetMethodFullName(likelyMethod, true, true, buffer, sizeof(buffer)));
#endif
- // Add this as a potential candidate.
- //
- addGuardedDevirtualizationCandidate(call, likelyMethod, likelyClass, likelyMethodAttribs, likelyClassAttribs,
- likelihood);
+ // Add this as a potential candidate.
+ //
+ addGuardedDevirtualizationCandidate(call, likelyMethod, likelyClass, likelyMethodAttribs, likelyClassAttribs,
+ likelihood);
+ }
}
//------------------------------------------------------------------------
@@ -6204,6 +6278,18 @@ void Compiler::impMarkInlineCandidate(GenTree* callNode,
CORINFO_CALL_INFO* callInfo,
IL_OFFSET ilOffset)
{
+ if (!opts.OptEnabled(CLFLG_INLINING))
+ {
+ /* XXX Mon 8/18/2008
+ * This assert is misleading. The caller does not ensure that we have CLFLG_INLINING set before
+ * calling impMarkInlineCandidate. However, if this assert trips it means that we're an inlinee and
+ * CLFLG_MINOPT is set. That doesn't make a lot of sense. If you hit this assert, work back and
+ * figure out why we did not set MAXOPT for this compile.
+ */
+ assert(!compIsForInlining());
+ return;
+ }
+
GenTreeCall* call = callNode->AsCall();
// Call might not have an inline candidate info yet (will be set by impMarkInlineCandidateHelper)
@@ -6257,7 +6343,7 @@ void Compiler::impMarkInlineCandidate(GenTree* callNode,
JITDUMP("Revoking guarded devirtualization candidacy for call [%06u]: target method can't be inlined\n",
dspTreeID(call));
- call->ClearGuardedDevirtualizationCandidate();
+ call->ClearInlineInfo();
}
//------------------------------------------------------------------------
@@ -6293,17 +6379,7 @@ void Compiler::impMarkInlineCandidateHelper(GenTreeCall* call,
// Let the strategy know there's another call
impInlineRoot()->m_inlineStrategy->NoteCall();
- if (!opts.OptEnabled(CLFLG_INLINING))
- {
- /* XXX Mon 8/18/2008
- * This assert is misleading. The caller does not ensure that we have CLFLG_INLINING set before
- * calling impMarkInlineCandidate. However, if this assert trips it means that we're an inlinee and
- * CLFLG_MINOPT is set. That doesn't make a lot of sense. If you hit this assert, work back and
- * figure out why we did not set MAXOPT for this compile.
- */
- assert(!compIsForInlining());
- return;
- }
+ assert(opts.OptEnabled(CLFLG_INLINING));
// Don't inline if not optimizing root method
if (opts.compDbgCode)
diff --git a/src/coreclr/jit/indirectcalltransformer.cpp b/src/coreclr/jit/indirectcalltransformer.cpp
index 15042ce6d744e3..bb2f316bb327d6 100644
--- a/src/coreclr/jit/indirectcalltransformer.cpp
+++ b/src/coreclr/jit/indirectcalltransformer.cpp
@@ -534,12 +534,9 @@ class IndirectCallTransformer
return call;
}
- //------------------------------------------------------------------------
- // ClearFlag: clear guarded devirtualization candidate flag from the original call.
- //
virtual void ClearFlag()
{
- origCall->ClearGuardedDevirtualizationCandidate();
+ // We remove the GDV flag from the call in the CreateElse
}
virtual UINT8 GetChecksCount()
@@ -587,9 +584,18 @@ class IndirectCallTransformer
prevCheckBlock->bbJumpDest = checkBlock;
compiler->fgAddRefPred(checkBlock, prevCheckBlock);
- // Weight for the new secondary check is the difference between the previous check and the thenBlock.
- checkBlock->inheritWeightPercentage(prevCheckBlock,
- 100 - origCall->GetGDVCandidateInfo(checkIdx)->likelihood);
+ // Calculate the total likelihood for this check as a sum of likelihoods
+ // of all previous candidates (thenBlocks)
+ unsigned checkLikelihood = 100;
+ for (uint8_t previousCandidate = 0; previousCandidate < checkIdx; previousCandidate++)
+ {
+ checkLikelihood -= origCall->GetGDVCandidateInfo(previousCandidate)->likelihood;
+ }
+
+ // Make sure we didn't overflow
+ assert(checkLikelihood <= 100);
+
+ checkBlock->inheritWeightPercentage(currBlock, checkLikelihood);
}
// Find last arg with a side effect. All args with any effect
@@ -839,6 +845,7 @@ class IndirectCallTransformer
// special candidate helper and we need to use the new 'this'.
GenTreeCall* call = compiler->gtCloneCandidateCall(origCall);
call->gtArgs.GetThisArg()->SetEarlyNode(compiler->gtNewLclvNode(thisTemp, TYP_REF));
+
call->SetIsGuarded();
JITDUMP("Direct call [%06u] in block " FMT_BB "\n", compiler->dspTreeID(call), block->bbNum);
@@ -994,12 +1001,16 @@ class IndirectCallTransformer
// Make sure it didn't overflow
assert(elseLikelihood <= 100);
+ // Remove everything related to inlining from the original call
+ origCall->ClearInlineInfo();
+
elseBlock->inheritWeightPercentage(currBlock, elseLikelihood);
GenTreeCall* call = origCall;
Statement* newStmt = compiler->gtNewStmt(call, stmt->GetDebugInfo());
call->gtFlags &= ~GTF_CALL_INLINE_CANDIDATE;
+
call->SetIsGuarded();
JITDUMP("Residual call [%06u] moved to block " FMT_BB "\n", compiler->dspTreeID(call), elseBlock->bbNum);
diff --git a/src/coreclr/jit/instr.cpp b/src/coreclr/jit/instr.cpp
index b942ddd6d878d9..93c4e601bb7811 100644
--- a/src/coreclr/jit/instr.cpp
+++ b/src/coreclr/jit/instr.cpp
@@ -335,6 +335,22 @@ bool CodeGenInterface::instIsFP(instruction ins)
#endif
}
+#if defined(TARGET_XARCH)
+/*****************************************************************************
+ *
+ * Returns non-zero if the given CPU instruction is an embedded broadcast
+ * compatible instruction.
+ */
+
+// static inline
+bool CodeGenInterface::instIsEmbeddedBroadcastCompatible(instruction ins)
+{
+ assert((unsigned)ins < ArrLen(instInfo));
+
+ return (instInfo[ins] & INS_Flags_EmbeddedBroadcastSupported) != 0;
+}
+#endif // TARGET_XARCH
+
/*****************************************************************************
*
* Generate a set instruction.
@@ -749,7 +765,7 @@ void CodeGen::inst_RV_SH(
// logic for determining what "kind" of operand "op" is.
//
// Arguments:
-// op - The operand node for which to obtain the descriptor
+// op - The operand node for which to obtain the descriptor.
//
// Return Value:
// The operand descriptor for "op".
@@ -795,11 +811,69 @@ CodeGen::OperandDesc CodeGen::genOperandDesc(GenTree* op)
assert(op->OperIsHWIntrinsic());
#if defined(FEATURE_HW_INTRINSICS)
- GenTreeHWIntrinsic* hwintrinsic = op->AsHWIntrinsic();
- NamedIntrinsic intrinsicId = hwintrinsic->GetHWIntrinsicId();
-
+ GenTreeHWIntrinsic* hwintrinsic = op->AsHWIntrinsic();
+ NamedIntrinsic intrinsicId = hwintrinsic->GetHWIntrinsicId();
+ var_types simdBaseType = hwintrinsic->GetSimdBaseType();
switch (intrinsicId)
{
+ case NI_AVX_BroadcastScalarToVector128:
+ case NI_AVX_BroadcastScalarToVector256:
+ {
+ // we have the assumption that AVX_BroadcastScalarToVector*
+ // only take the memory address as the operand.
+ assert(hwintrinsic->isContained());
+ assert(hwintrinsic->OperIsMemoryLoad());
+ assert(hwintrinsic->GetOperandCount() == 1);
+ assert(varTypeIsFloating(simdBaseType));
+ GenTree* hwintrinsicChild = hwintrinsic->Op(1);
+ assert(hwintrinsicChild->isContained());
+ if (hwintrinsicChild->OperIs(GT_LCL_ADDR, GT_CLS_VAR_ADDR, GT_CNS_INT, GT_LEA))
+ {
+ addr = hwintrinsic->Op(1);
+ break;
+ }
+ else
+ {
+ assert(hwintrinsicChild->OperIs(GT_LCL_VAR));
+ return OperandDesc(simdBaseType, hwintrinsicChild);
+ }
+ }
+
+ case NI_SSE3_MoveAndDuplicate:
+ case NI_AVX2_BroadcastScalarToVector128:
+ case NI_AVX2_BroadcastScalarToVector256:
+ case NI_AVX512F_BroadcastScalarToVector512:
+ {
+ assert(hwintrinsic->isContained());
+ if (intrinsicId == NI_SSE3_MoveAndDuplicate)
+ {
+ assert(simdBaseType == TYP_DOUBLE);
+ }
+ // If broadcast node is contained, should mean that we have some forms like
+ // Broadcast -> CreateScalarUnsafe -> Scalar.
+ // If so, directly emit scalar.
+ // In the codes below, we specially handle the `Broadcast -> CNS_INT` form and
+ // handle other cases recursively.
+ GenTree* hwintrinsicChild = hwintrinsic->Op(1);
+ assert(hwintrinsicChild->isContained());
+ if (hwintrinsicChild->OperIs(GT_CNS_INT))
+ {
+ // a special case is when the operand of CreateScalarUnsafe is in integer type,
+ // CreateScalarUnsafe node will be fold, so we directly match a pattern of
+ // broadcast -> LCL_VAR(TYP_(U)INT)
+ ssize_t scalarValue = hwintrinsicChild->AsIntCon()->IconValue();
+ UNATIVE_OFFSET cnum = emit->emitDataConst(&scalarValue, genTypeSize(simdBaseType),
+ genTypeSize(simdBaseType), simdBaseType);
+ return OperandDesc(compiler->eeFindJitDataOffs(cnum));
+ }
+ else
+ {
+ // If the operand of broadcast is not a constant integer,
+ // we handle all the other cases recursively.
+ return genOperandDesc(hwintrinsicChild);
+ }
+ break;
+ }
case NI_Vector128_CreateScalarUnsafe:
case NI_Vector256_CreateScalarUnsafe:
case NI_Vector512_CreateScalarUnsafe:
@@ -865,8 +939,10 @@ CodeGen::OperandDesc CodeGen::genOperandDesc(GenTree* op)
return OperandDesc(emit->emitFltOrDblConst(op->AsDblCon()->DconValue(), emitTypeSize(op)));
case GT_CNS_INT:
+ {
assert(op->isContainedIntOrIImmed());
return OperandDesc(op->AsIntCon()->IconValue(), op->AsIntCon()->ImmedValNeedsReloc(compiler));
+ }
case GT_CNS_VEC:
{
@@ -1096,19 +1172,49 @@ void CodeGen::inst_RV_TT_IV(instruction ins, emitAttr attr, regNumber reg1, GenT
}
}
+//------------------------------------------------------------------------
+// IsEmbeddedBroadcastEnabled: determine if embedded broadcast can be enabled
+//
+// Arguments:
+// ins -- The instruction being emitted
+// op -- The second operand of the instruction.
+//
+#if defined(TARGET_XARCH) && defined(FEATURE_HW_INTRINSICS)
+bool CodeGenInterface::IsEmbeddedBroadcastEnabled(instruction ins, GenTree* op)
+{
+ // To enable embedded broadcast, we need 3 things,
+ // 1. EVEX enabled.
+ // 2. Embedded broadcast compatible intrinsics
+ // 3. A contained broadcast scalar node
+ if (!GetEmitter()->UseEvexEncoding())
+ {
+ return false;
+ }
+ if (!instIsEmbeddedBroadcastCompatible(ins))
+ {
+ return false;
+ }
+ if (!op->isContained() || !op->OperIsHWIntrinsic())
+ {
+ return false;
+ }
+
+ return op->AsHWIntrinsic()->OperIsBroadcastScalar();
+}
+#endif // TARGET_XARCH && FEATURE_HW_INTRINSICS
+
//------------------------------------------------------------------------
// inst_RV_RV_TT: Generates an instruction that takes 2 operands:
// a register operand and an operand that may be in memory or register
// the result is returned in register
//
// Arguments:
-// ins -- The instruction being emitted
-// size -- The emit size attribute
-// targetReg -- The target register
-// op1Reg -- The first operand register
-// op2 -- The second operand, which may be a memory node or a node producing a register
-// isRMW -- true if the instruction is RMW; otherwise, false
-//
+// ins -- The instruction being emitted
+// size -- The emit size attribute
+// targetReg -- The target register
+// op1Reg -- The first operand register
+// op2 -- The second operand, which may be a memory node or a node producing a register
+// isRMW -- true if the instruction is RMW; otherwise, false
void CodeGen::inst_RV_RV_TT(
instruction ins, emitAttr size, regNumber targetReg, regNumber op1Reg, GenTree* op2, bool isRMW)
{
@@ -1118,15 +1224,25 @@ void CodeGen::inst_RV_RV_TT(
// TODO-XArch-CQ: Commutative operations can have op1 be contained
// TODO-XArch-CQ: Non-VEX encoded instructions can have both ops contained
+ insOpts instOptions = INS_OPTS_NONE;
+#if defined(TARGET_XARCH) && defined(FEATURE_HW_INTRINSICS)
+ bool IsEmbBroadcast = CodeGenInterface::IsEmbeddedBroadcastEnabled(ins, op2);
+ if (IsEmbBroadcast)
+ {
+ instOptions = INS_OPTS_EVEX_b;
+ }
+#endif // TARGET_XARCH && FEATURE_HW_INTRINSICS
OperandDesc op2Desc = genOperandDesc(op2);
switch (op2Desc.GetKind())
{
case OperandKind::ClsVar:
- emit->emitIns_SIMD_R_R_C(ins, size, targetReg, op1Reg, op2Desc.GetFieldHnd(), 0);
+ {
+ emit->emitIns_SIMD_R_R_C(ins, size, targetReg, op1Reg, op2Desc.GetFieldHnd(), 0, instOptions);
break;
-
+ }
case OperandKind::Local:
- emit->emitIns_SIMD_R_R_S(ins, size, targetReg, op1Reg, op2Desc.GetVarNum(), op2Desc.GetLclOffset());
+ emit->emitIns_SIMD_R_R_S(ins, size, targetReg, op1Reg, op2Desc.GetVarNum(), op2Desc.GetLclOffset(),
+ instOptions);
break;
case OperandKind::Indir:
@@ -1135,7 +1251,7 @@ void CodeGen::inst_RV_RV_TT(
// temporary GT_IND to generate code with.
GenTreeIndir indirForm;
GenTreeIndir* indir = op2Desc.GetIndirForm(&indirForm);
- emit->emitIns_SIMD_R_R_A(ins, size, targetReg, op1Reg, indir);
+ emit->emitIns_SIMD_R_R_A(ins, size, targetReg, op1Reg, indir, instOptions);
}
break;
diff --git a/src/coreclr/jit/instr.h b/src/coreclr/jit/instr.h
index df8658195f8da9..bbc8089376ae58 100644
--- a/src/coreclr/jit/instr.h
+++ b/src/coreclr/jit/instr.h
@@ -146,8 +146,7 @@ enum insFlags : uint64_t
// Avx
INS_Flags_IsDstDstSrcAVXInstruction = 1ULL << 26,
INS_Flags_IsDstSrcSrcAVXInstruction = 1ULL << 27,
- INS_Flags_IsMskSrcSrcEvexInstruction = 1ULL << 28,
- INS_Flags_Is3OperandInstructionMask = (INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_IsDstSrcSrcAVXInstruction | INS_Flags_IsMskSrcSrcEvexInstruction),
+ INS_Flags_Is3OperandInstructionMask = (INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_IsDstSrcSrcAVXInstruction),
// w and s bits
INS_FLAGS_Has_Wbit = 1ULL << 29,
@@ -180,10 +179,20 @@ enum insFlags : uint64_t
KInstruction = 1ULL << 41,
+ // EVEX feature: embedded broadcast
+ INS_Flags_EmbeddedBroadcastSupported = 1ULL << 42,
+
// TODO-Cleanup: Remove this flag and its usage from TARGET_XARCH
INS_FLAGS_DONT_CARE = 0x00ULL,
};
+enum insOpts: unsigned
+{
+ INS_OPTS_NONE,
+
+ INS_OPTS_EVEX_b
+};
+
#elif defined(TARGET_ARM) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
// TODO-Cleanup: Move 'insFlags' under TARGET_ARM
enum insFlags: unsigned
@@ -350,7 +359,7 @@ enum insOpts : unsigned
INS_OPTS_JALR, // see ::emitIns_J_R().
INS_OPTS_J, // see ::emitIns_J().
INS_OPTS_J_cond, // see ::emitIns_J_cond_la().
- INS_OPTS_I, // see ::emitIns_I_la().
+ INS_OPTS_I, // see ::emitLoadImmediate().
INS_OPTS_C, // see ::emitIns_Call().
INS_OPTS_RELOC, // see ::emitIns_R_AI().
};
diff --git a/src/coreclr/jit/instrsxarch.h b/src/coreclr/jit/instrsxarch.h
index 0bb638bfea69bd..161df4485e0d98 100644
--- a/src/coreclr/jit/instrsxarch.h
+++ b/src/coreclr/jit/instrsxarch.h
@@ -191,7 +191,7 @@ INSTMUL(imul_15, "imul", IUM_RD, BAD_CODE, 0x4400003868,
INST3(FIRST_SSE_INSTRUCTION, "FIRST_SSE_INSTRUCTION", IUM_WR, BAD_CODE, BAD_CODE, BAD_CODE, INS_TT_NONE, INS_FLAGS_None)
// SSE
-INST3(addps, "addps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x58), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed singles
+INST3(addps, "addps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x58), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_EmbeddedBroadcastSupported) // Add packed singles
INST3(addss, "addss", IUM_WR, BAD_CODE, BAD_CODE, SSEFLT(0x58), INS_TT_TUPLE1_SCALAR, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add scalar singles
INST3(andnps, "andnps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x55), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // And-Not packed singles
INST3(andps, "andps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x54), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // AND packed singles
@@ -240,7 +240,7 @@ INST3(unpcklps, "unpcklps", IUM_WR, BAD_CODE, BAD_CODE,
INST3(xorps, "xorps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x57), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // XOR packed singles
// SSE2
-INST3(addpd, "addpd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x58), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed doubles
+INST3(addpd, "addpd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x58), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_EmbeddedBroadcastSupported) // Add packed doubles
INST3(addsd, "addsd", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x58), INS_TT_TUPLE1_SCALAR, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add scalar doubles
INST3(andnpd, "andnpd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x55), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // And-Not packed doubles
INST3(andpd, "andpd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x54), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // AND packed doubles
@@ -290,8 +290,8 @@ INST3(packssdw, "packssdw", IUM_WR, BAD_CODE, BAD_CODE,
INST3(packsswb, "packsswb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x63), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Pack (narrow) short to byte with saturation
INST3(packuswb, "packuswb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x67), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Pack (narrow) short to unsigned byte with saturation
INST3(paddb, "paddb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xFC), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed byte integers
-INST3(paddd, "paddd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xFE), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed double-word (32-bit) integers
-INST3(paddq, "paddq", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xD4), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed quad-word (64-bit) integers
+INST3(paddd, "paddd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xFE), INS_TT_FULL, Input_32Bit | REX_W0_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_EmbeddedBroadcastSupported) // Add packed double-word (32-bit) integers
+INST3(paddq, "paddq", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xD4), INS_TT_FULL, Input_64Bit | REX_W1_EVEX | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction | INS_Flags_EmbeddedBroadcastSupported) // Add packed quad-word (64-bit) integers
INST3(paddsb, "paddsb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xEC), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed signed byte integers and saturate the results
INST3(paddsw, "paddsw", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xED), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed signed word integers and saturate the results
INST3(paddusb, "paddusb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xDC), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_VEX | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Add packed unsigned byte integers and saturate the results
@@ -608,19 +608,28 @@ INST3(LAST_AVX_INSTRUCTION, "LAST_AVX_INSTRUCTION", IUM_WR, BAD_CODE, BAD_CODE,
INST3(FIRST_AVX512_INSTRUCTION, "FIRST_AVX512_INSTRUCTION", IUM_WR, BAD_CODE, BAD_CODE, BAD_CODE, INS_TT_NONE, INS_FLAGS_None)
// AVX512F
-INST3(kmovw_gpr, "kmovw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction)
-INST3(kmovw_msk, "kmovw", IUM_WR, PCKFLT(0x91), BAD_CODE, PCKFLT(0x90), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction)
-INST3(kortestw, "kortestw", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x98), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction)
+INST3(kandw, "kandw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x41), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical AND masks
+INST3(kandnw, "kandnw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x42), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical AND NOT masks
+INST3(kmovw_gpr, "kmovw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(kmovw_msk, "kmovw", IUM_WR, PCKFLT(0x91), BAD_CODE, PCKFLT(0x90), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(knotw, "knotw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x44), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // NOT mask register
+INST3(korw, "korw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x45), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical OR masks
+INST3(kortestw, "kortestw", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x98), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // OR masks and set flags
+INST3(kshiftlw, "kshiftlw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x32), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Shift left mask registers
+INST3(kshiftrw, "kshiftrw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x30), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Shift right mask registers
+INST3(kunpckbw, "kunpckbw", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x4B), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Unpack for mask registers
+INST3(kxnorw, "kxnorw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x46), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical XNOR masks
+INST3(kxorw, "kxorw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x47), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical XOR masks
INST3(valignd, "alignd", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x03), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Align doubleword vectors
INST3(valignq, "alignq", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x03), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Align quadword vectors
INST3(vbroadcastf64x2, "broadcastf64x2", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x1A), INS_TT_TUPLE2, Input_64Bit | REX_W1 | Encoding_EVEX) // Broadcast packed float values read from memory to entire register
INST3(vbroadcasti64x2, "broadcasti64x2", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x5A), INS_TT_TUPLE2, Input_64Bit | REX_W1 | Encoding_EVEX) // Broadcast packed integer values read from memory to entire register
INST3(vbroadcastf64x4, "broadcastf64x4", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x1B), INS_TT_TUPLE2, Input_64Bit | REX_W1 | Encoding_EVEX) // Broadcast packed float values read from memory to entire register
INST3(vbroadcasti64x4, "broadcasti64x4", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x5B), INS_TT_TUPLE2, Input_64Bit | REX_W1 | Encoding_EVEX) // Broadcast packed integer values read from memory to entire register
-INST3(vcmpps, "cmpps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0xC2), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // compare packed singles
-INST3(vcmpss, "cmpss", IUM_WR, BAD_CODE, BAD_CODE, SSEFLT(0xC2), INS_TT_TUPLE1_SCALAR, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // compare scalar singles
-INST3(vcmppd, "cmppd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xC2), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // compare packed doubles
-INST3(vcmpsd, "cmpsd", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0xC2), INS_TT_TUPLE1_SCALAR, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // compare scalar doubles
+INST3(vcmpps, "cmpps", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0xC2), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // compare packed singles
+INST3(vcmpss, "cmpss", IUM_WR, BAD_CODE, BAD_CODE, SSEFLT(0xC2), INS_TT_TUPLE1_SCALAR, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // compare scalar singles
+INST3(vcmppd, "cmppd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xC2), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // compare packed doubles
+INST3(vcmpsd, "cmpsd", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0xC2), INS_TT_TUPLE1_SCALAR, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // compare scalar doubles
INST3(vcvtpd2udq, "cvtpd2udq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x79), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX) // cvt packed doubles to unsigned DWORDs
INST3(vcvtps2udq, "cvtps2udq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x79), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX) // cvt packed singles to unsigned DWORDs
INST3(vcvtsd2usi, "cvtsd2usi", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x79), INS_TT_TUPLE1_FIXED, Input_64Bit | REX_WX | Encoding_EVEX) // cvt scalar double to unsigned DWORD/QWORD
@@ -658,10 +667,10 @@ INST3(vpandq, "pandq", IUM_WR, BAD_CODE, BAD_
INST3(vpandnq, "pandnq", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0xDF), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed bit-wise AND NOT of two xmm regs
INST3(vpbroadcastd_gpr, "pbroadcastd", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x7C), INS_TT_TUPLE1_SCALAR, Input_32Bit | REX_W0 | Encoding_EVEX) // Broadcast int32 value from gpr to entire register
INST3(vpbroadcastq_gpr, "pbroadcastq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x7C), INS_TT_TUPLE1_SCALAR, Input_64Bit | REX_W1 | Encoding_EVEX) // Broadcast int64 value from gpr to entire register
-INST3(vpcmpeqd, "pcmpeqd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x76), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 32-bit integers for equality
-INST3(vpcmpgtd, "pcmpgtd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x66), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 32-bit signed integers for greater than
-INST3(vpcmpeqq, "pcmpeqq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x29), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 64-bit integers for equality
-INST3(vpcmpgtq, "pcmpgtq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x37), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 64-bit integers for equality
+INST3(vpcmpeqd, "pcmpeqd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x76), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 32-bit integers for equality
+INST3(vpcmpgtd, "pcmpgtd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x66), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 32-bit signed integers for greater than
+INST3(vpcmpeqq, "pcmpeqq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x29), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 64-bit integers for equality
+INST3(vpcmpgtq, "pcmpgtq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x37), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 64-bit integers for equality
INST3(vpermq_reg, "permq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x36), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Permute 64-bit of input register
INST3(vpermpd_reg, "permpd", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x16), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Permute 64-bit of input register
INST3(vpermi2d, "permi2d", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x76), INS_TT_FULL, Input_32Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Full Permute From Two Tables Overwriting the Index
@@ -735,25 +744,47 @@ INST3(vshufi32x4, "shufi32x4", IUM_WR, BAD_CODE, BAD_
INST3(vshufi64x2, "shufi64x2", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x43), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Shuffle packed values at 128-bit granularity
// AVX512BW
-INST3(kmovd_gpr, "kmovd", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction)
-INST3(kmovd_msk, "kmovd", IUM_WR, PCKDBL(0x91), BAD_CODE, PCKDBL(0x90), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction)
-INST3(kmovq_gpr, "kmovq", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x92), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction)
-INST3(kmovq_msk, "kmovq", IUM_WR, PCKFLT(0x91), BAD_CODE, PCKFLT(0x90), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction)
-INST3(kortestd, "kortestd", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x98), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction)
-INST3(kortestq, "kortestq", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x98), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction)
+INST3(kaddd, "kaddd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x4A), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Add two masks
+INST3(kaddq, "kaddq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x4A), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Add two masks
+INST3(kandd, "kandd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x41), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical AND masks
+INST3(kandq, "kandq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x41), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical AND masks
+INST3(kandnd, "kandnd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x42), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical AND NOT masks
+INST3(kandnq, "kandnq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x42), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical AND NOT masks
+INST3(kmovd_gpr, "kmovd", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(kmovd_msk, "kmovd", IUM_WR, PCKDBL(0x91), BAD_CODE, PCKDBL(0x90), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(kmovq_gpr, "kmovq", IUM_WR, BAD_CODE, BAD_CODE, SSEDBL(0x92), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(kmovq_msk, "kmovq", IUM_WR, PCKFLT(0x91), BAD_CODE, PCKFLT(0x90), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(knotd, "knotd", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x44), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // NOT mask register
+INST3(knotq, "knotq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x44), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // NOT mask register
+INST3(kord, "kord", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x45), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical OR masks
+INST3(korq, "korq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x45), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical OR masks
+INST3(kortestd, "kortestd", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x98), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // OR masks and set flags
+INST3(kortestq, "kortestq", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x98), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // OR masks and set flags
+INST3(kshiftld, "kshiftld", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x33), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Shift left mask registers
+INST3(kshiftlq, "kshiftlq", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x33), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Shift left mask registers
+INST3(kshiftrd, "kshiftrd", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x31), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Shift right mask registers
+INST3(kshiftrq, "kshiftrq", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x31), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Shift right mask registers
+INST3(ktestd, "ktestd", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x99), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // Packed bit test masks and set flags
+INST3(ktestq, "ktestq", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x99), INS_TT_NONE, REX_W1 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // Packed bit test masks and set flags
+INST3(kunpckdq, "kunpckdq", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x4B), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Unpack for mask registers
+INST3(kunpckwd, "kunpckwd", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x4B), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Unpack for mask registers
+INST3(kxnord, "kxnord", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x46), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical XNOR masks
+INST3(kxnorq, "kxnorq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x46), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical XNOR masks
+INST3(kxord, "kxord", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x47), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical XOR masks
+INST3(kxorq, "kxorq", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x47), INS_TT_NONE, REX_W1 | Encoding_VEX | KInstruction) // Bitwise logical XOR masks
INST3(vdbpsadbw, "dbpsadbw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x42), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Double block packed Sum-Absolute-Differences (SAD) on unsigned bytes
INST3(vmovdqu8, "movdqu8", IUM_WR, SSEFLT(0x7F), BAD_CODE, SSEFLT(0x6F), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX)
INST3(vmovdqu16, "movdqu16", IUM_WR, SSEFLT(0x7F), BAD_CODE, SSEFLT(0x6F), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX)
INST3(vpbroadcastb_gpr, "pbroadcastb", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x7A), INS_TT_TUPLE1_SCALAR, Input_8Bit | REX_W0 | Encoding_EVEX) // Broadcast int8 value from gpr to entire register
INST3(vpbroadcastw_gpr, "pbroadcastw", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x7B), INS_TT_TUPLE1_SCALAR, Input_16Bit | REX_W0 | Encoding_EVEX) // Broadcast int16 value from gpr to entire register
-INST3(vpcmpb, "pcmpb", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3F), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction)
-INST3(vpcmpeqb, "pcmpeqb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x74), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 8-bit integers for equality
-INST3(vpcmpeqw, "pcmpeqw", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x75), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 16-bit integers for equality
-INST3(vpcmpgtb, "pcmpgtb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x64), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 8-bit signed integers for greater than
-INST3(vpcmpgtw, "pcmpgtw", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x65), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction) // Packed compare 16-bit signed integers for greater than
-INST3(vpcmpw, "pcmpw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3F), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction)
-INST3(vpcmpub, "pcmpub", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3E), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction)
-INST3(vpcmpuw, "pcmpuw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3E), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsMskSrcSrcEvexInstruction)
+INST3(vpcmpb, "pcmpb", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3F), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction)
+INST3(vpcmpeqb, "pcmpeqb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x74), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 8-bit integers for equality
+INST3(vpcmpeqw, "pcmpeqw", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x75), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 16-bit integers for equality
+INST3(vpcmpgtb, "pcmpgtb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x64), INS_TT_FULL_MEM, Input_8Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 8-bit signed integers for greater than
+INST3(vpcmpgtw, "pcmpgtw", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x65), INS_TT_FULL_MEM, Input_16Bit | REX_WIG | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction) // Packed compare 16-bit signed integers for greater than
+INST3(vpcmpw, "pcmpw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3F), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction)
+INST3(vpcmpub, "pcmpub", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3E), INS_TT_FULL_MEM, Input_8Bit | REX_W0 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction)
+INST3(vpcmpuw, "pcmpuw", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x3E), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstDstSrcAVXInstruction)
INST3(vpermw, "permw", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x8D), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Permute Packed Doublewords Elements
INST3(vpermi2w, "permi2w", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x75), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Full Permute From Two Tables Overwriting the Index
INST3(vpermt2w, "permt2w", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x7D), INS_TT_FULL_MEM, Input_16Bit | REX_W1 | Encoding_EVEX | INS_Flags_IsDstSrcSrcAVXInstruction) // Full Permute From Two Tables Overwriting one Table
@@ -775,9 +806,21 @@ INST3(vplzcntd, "plzcntd", IUM_WR, BAD_CODE, BAD_
INST3(vplzcntq, "plzcntq", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x44), INS_TT_FULL, Input_64Bit | REX_W1 | Encoding_EVEX) // Count the number of leading zero bits for packed qword values
// AVX512DQ
-INST3(kortestb, "kortestb", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x98), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction)
-INST3(kmovb_gpr, "kmovb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction)
-INST3(kmovb_msk, "kmovb", IUM_WR, PCKDBL(0x91), BAD_CODE, PCKDBL(0x90), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction)
+INST3(kaddb, "kaddb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x4A), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Add two masks
+INST3(kaddw, "kaddw", IUM_WR, BAD_CODE, BAD_CODE, PCKFLT(0x4A), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Add two masks
+INST3(kandb, "kandb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x41), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical AND masks
+INST3(kandnb, "kandnb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x42), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical AND NOT masks
+INST3(kmovb_gpr, "kmovb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x92), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(kmovb_msk, "kmovb", IUM_WR, PCKDBL(0x91), BAD_CODE, PCKDBL(0x90), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Move from and to mask registers
+INST3(knotb, "knotb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x44), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // NOT mask register
+INST3(korb, "korb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x45), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical OR masks
+INST3(kortestb, "kortestb", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x98), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // OR masks and set flags
+INST3(kshiftlb, "kshiftlb", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x32), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Shift left mask registers
+INST3(kshiftrb, "kshiftrb", IUM_WR, BAD_CODE, BAD_CODE, SSE3A(0x30), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Shift right mask registers
+INST3(ktestb, "ktestb", IUM_RD, BAD_CODE, BAD_CODE, PCKDBL(0x99), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // Packed bit test masks and set flags
+INST3(ktestw, "ktestw", IUM_RD, BAD_CODE, BAD_CODE, PCKFLT(0x99), INS_TT_NONE, REX_W0 | Encoding_VEX | Resets_OF | Resets_SF | Writes_ZF | Resets_AF | Resets_PF | Writes_CF | KInstruction) // Packed bit test masks and set flags
+INST3(kxnorb, "kxnorb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x46), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical XNOR masks
+INST3(kxorb, "kxorb", IUM_WR, BAD_CODE, BAD_CODE, PCKDBL(0x47), INS_TT_NONE, REX_W0 | Encoding_VEX | KInstruction) // Bitwise logical XOR masks
INST3(vbroadcastf32x2, "broadcastf32x2", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x19), INS_TT_TUPLE2, Input_32Bit | REX_W0 | Encoding_EVEX) // Broadcast packed float values read from memory to entire register
INST3(vbroadcasti32x2, "broadcasti32x2", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x59), INS_TT_TUPLE2, Input_32Bit | REX_W0 | Encoding_EVEX) // Broadcast packed integer values read from memory to entire register
INST3(vbroadcastf32x8, "broadcastf32x8", IUM_WR, BAD_CODE, BAD_CODE, SSE38(0x1B), INS_TT_TUPLE8, Input_32Bit | REX_W0 | Encoding_EVEX) // Broadcast packed float values read from memory to entire register
diff --git a/src/coreclr/jit/jitconfigvalues.h b/src/coreclr/jit/jitconfigvalues.h
index 6ab73d6abde378..8657940e5a00c5 100644
--- a/src/coreclr/jit/jitconfigvalues.h
+++ b/src/coreclr/jit/jitconfigvalues.h
@@ -528,7 +528,7 @@ CONFIG_INTEGER(JitEnableGuardedDevirtualization, W("JitEnableGuardedDevirtualiza
#define MAX_GDV_TYPE_CHECKS 5
// Number of types to probe for polymorphic virtual call-sites to devirtualize them,
// Max number is MAX_GDV_TYPE_CHECKS defined above ^
-CONFIG_INTEGER(JitGuardedDevirtualizationMaxTypeChecks, W("JitGuardedDevirtualizationMaxTypeChecks"), 3)
+CONFIG_INTEGER(JitGuardedDevirtualizationMaxTypeChecks, W("JitGuardedDevirtualizationMaxTypeChecks"), 1)
// Various policies for GuardedDevirtualization
CONFIG_INTEGER(JitGuardedDevirtualizationChainLikelihood, W("JitGuardedDevirtualizationChainLikelihood"), 0x4B) // 75
@@ -539,11 +539,11 @@ CONFIG_INTEGER(JitRandomGuardedDevirtualization, W("JitRandomGuardedDevirtualiza
#endif // DEBUG
// Enable insertion of patchpoints into Tier0 methods, switching to optimized where needed.
-#if defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#if defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
CONFIG_INTEGER(TC_OnStackReplacement, W("TC_OnStackReplacement"), 1)
#else
CONFIG_INTEGER(TC_OnStackReplacement, W("TC_OnStackReplacement"), 0)
-#endif // defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#endif // defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// Initial patchpoint counter value used by jitted code
CONFIG_INTEGER(TC_OnStackReplacement_InitialCounter, W("TC_OnStackReplacement_InitialCounter"), 1000)
// Enable partial compilation for Tier0 methods
diff --git a/src/coreclr/jit/jitee.h b/src/coreclr/jit/jitee.h
index 7c3089833bc332..ac51df952ab420 100644
--- a/src/coreclr/jit/jitee.h
+++ b/src/coreclr/jit/jitee.h
@@ -60,7 +60,7 @@ class JitFlags
JIT_FLAG_BBINSTR_IF_LOOPS = 32, // JIT must instrument current method if it has loops
JIT_FLAG_PUBLISH_SECRET_PARAM = 33, // JIT must place stub secret param into local 0. (used by IL stubs)
JIT_FLAG_UNUSED13 = 34,
- JIT_FLAG_SAMPLING_JIT_BACKGROUND = 35, // JIT is being invoked as a result of stack sampling for hot methods in the background
+ JIT_FLAG_UNUSED14 = 35,
JIT_FLAG_USE_PINVOKE_HELPERS = 36, // The JIT should use the PINVOKE_{BEGIN,END} helpers instead of emitting inline transitions
JIT_FLAG_REVERSE_PINVOKE = 37, // The JIT should insert REVERSE_PINVOKE_{ENTER,EXIT} helpers into method prolog/epilog
JIT_FLAG_TRACK_TRANSITIONS = 38, // The JIT should insert the helper variants that track transitions.
@@ -192,7 +192,6 @@ class JitFlags
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_BBOPT, JIT_FLAG_BBOPT);
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_FRAMED, JIT_FLAG_FRAMED);
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_PUBLISH_SECRET_PARAM, JIT_FLAG_PUBLISH_SECRET_PARAM);
- FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND, JIT_FLAG_SAMPLING_JIT_BACKGROUND);
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_USE_PINVOKE_HELPERS, JIT_FLAG_USE_PINVOKE_HELPERS);
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_REVERSE_PINVOKE, JIT_FLAG_REVERSE_PINVOKE);
FLAGS_EQUAL(CORJIT_FLAGS::CORJIT_FLAG_TIER0, JIT_FLAG_TIER0);
diff --git a/src/coreclr/jit/lclvars.cpp b/src/coreclr/jit/lclvars.cpp
index 454d65172543bc..80d9493daca427 100644
--- a/src/coreclr/jit/lclvars.cpp
+++ b/src/coreclr/jit/lclvars.cpp
@@ -1757,7 +1757,8 @@ bool Compiler::StructPromotionHelper::CanPromoteStructType(CORINFO_CLASS_HANDLE
#if defined(FEATURE_SIMD)
// getMaxVectorByteLength() represents the size of the largest primitive type that we can struct promote.
- const unsigned maxSize = MAX_NumOfFieldsInPromotableStruct * compiler->getMaxVectorByteLength();
+ const unsigned maxSize =
+ MAX_NumOfFieldsInPromotableStruct * max(compiler->getMaxVectorByteLength(), sizeof(double));
#else // !FEATURE_SIMD
// sizeof(double) represents the size of the largest primitive type that we can struct promote.
const unsigned maxSize = MAX_NumOfFieldsInPromotableStruct * sizeof(double);
@@ -5286,7 +5287,7 @@ void Compiler::lvaFixVirtualFrameOffsets()
if (opts.IsOSR())
{
-#if defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#if defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
// Stack offset includes Tier0 frame.
//
JITDUMP("--- delta bump %d for OSR + Tier0 frame\n", info.compPatchpointInfo->TotalFrameSize());
@@ -5391,7 +5392,7 @@ void Compiler::lvaFixVirtualFrameOffsets()
#endif // FEATURE_FIXED_OUT_ARGS
-#if defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
+#if defined(TARGET_ARM64) || defined(TARGET_RISCV64)
// We normally add alignment below the locals between them and the outgoing
// arg space area. When we store fp/lr(ra) at the bottom, however, this will
// be below the alignment. So we should not apply the alignment adjustment to
@@ -5403,7 +5404,14 @@ void Compiler::lvaFixVirtualFrameOffsets()
{
lvaTable[lvaRetAddrVar].SetStackOffset(REGSIZE_BYTES);
}
-#endif // TARGET_ARM64 || TARGET_LOONGARCH64 || TARGET_RISCV64
+#elif defined(TARGET_LOONGARCH64)
+ assert(codeGen->isFramePointerUsed());
+ if (lvaRetAddrVar != BAD_VAR_NUM)
+ {
+ // For LoongArch64, the RA is below the fp. see the `genPushCalleeSavedRegisters`
+ lvaTable[lvaRetAddrVar].SetStackOffset(-REGSIZE_BYTES);
+ }
+#endif // !TARGET_LOONGARCH64
}
#ifdef TARGET_ARM
@@ -6189,13 +6197,17 @@ void Compiler::lvaAssignVirtualFrameOffsetsToLocals()
stkOffs -= (compCalleeRegsPushed - 2) * REGSIZE_BYTES;
}
-#elif defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
+#elif defined(TARGET_LOONGARCH64)
+
+ assert(compCalleeRegsPushed >= 2);
+
+#elif defined(TARGET_RISCV64)
// Subtract off FP and RA.
assert(compCalleeRegsPushed >= 2);
stkOffs -= (compCalleeRegsPushed - 2) * REGSIZE_BYTES;
-#else // !TARGET_LOONGARCH64 !TARGET_RISCV64
+#else // !TARGET_RISCV64
#ifdef TARGET_ARM
// On ARM32 LR is part of the pushed registers and is always stored at the
// top.
@@ -6206,7 +6218,7 @@ void Compiler::lvaAssignVirtualFrameOffsetsToLocals()
#endif
stkOffs -= compCalleeRegsPushed * REGSIZE_BYTES;
-#endif // !TARGET_LOONGARCH64 !TARGET_RISCV64
+#endif // !TARGET_RISCV64
// (2) Account for the remainder of the frame
//
@@ -6928,10 +6940,10 @@ void Compiler::lvaAssignVirtualFrameOffsetsToLocals()
}
#endif // TARGET_ARM64
-#if defined(TARGET_LOONGARCH64) || defined(TARGET_RISCV64)
+#if defined(TARGET_RISCV64)
assert(isFramePointerUsed()); // Note that currently we always have a frame pointer
stkOffs -= 2 * REGSIZE_BYTES;
-#endif // TARGET_LOONGARCH64 || TARGET_RISCV64
+#endif // TARGET_RISCV64
#if FEATURE_FIXED_OUT_ARGS
if (lvaOutgoingArgSpaceSize > 0)
@@ -6949,9 +6961,14 @@ void Compiler::lvaAssignVirtualFrameOffsetsToLocals()
}
#endif // FEATURE_FIXED_OUT_ARGS
+#ifdef TARGET_LOONGARCH64
+ // For LoongArch64, CalleeSavedRegs are at bottom.
+ int pushedCount = 0;
+#else
// compLclFrameSize equals our negated virtual stack offset minus the pushed registers and return address
// and the pushed frame pointer register which for some strange reason isn't part of 'compCalleeRegsPushed'.
int pushedCount = compCalleeRegsPushed;
+#endif
#ifdef TARGET_ARM64
if (info.compIsVarArgs)
@@ -7803,7 +7820,7 @@ unsigned Compiler::lvaFrameSize(FrameLayoutState curState)
if (compFloatingPointUsed)
compCalleeRegsPushed += CNT_CALLEE_SAVED_FLOAT;
- compCalleeRegsPushed++; // we always push LR/RA. See genPushCalleeSavedRegisters
+ compCalleeRegsPushed++; // we always push LR or RA. See genPushCalleeSavedRegisters
#elif defined(TARGET_AMD64)
if (compFloatingPointUsed)
{
@@ -7840,7 +7857,7 @@ unsigned Compiler::lvaFrameSize(FrameLayoutState curState)
{
calleeSavedRegMaxSz += CALLEE_SAVED_FLOAT_MAXSZ;
}
- calleeSavedRegMaxSz += REGSIZE_BYTES; // we always push LR/RA. See genPushCalleeSavedRegisters
+ calleeSavedRegMaxSz += REGSIZE_BYTES; // we always push LR or RA. See genPushCalleeSavedRegisters
#endif
result = compLclFrameSize + calleeSavedRegMaxSz;
@@ -7927,7 +7944,7 @@ int Compiler::lvaToCallerSPRelativeOffset(int offset, bool isFpBased, bool forRo
offset += codeGen->genCallerSPtoInitialSPdelta();
}
-#if defined(TARGET_AMD64) || defined(TARGET_ARM64)
+#if defined(TARGET_AMD64) || defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
if (forRootFrame && opts.IsOSR())
{
const PatchpointInfo* const ppInfo = info.compPatchpointInfo;
@@ -7946,7 +7963,7 @@ int Compiler::lvaToCallerSPRelativeOffset(int offset, bool isFpBased, bool forRo
//
const int adjustment = ppInfo->TotalFrameSize() + REGSIZE_BYTES;
-#elif defined(TARGET_ARM64)
+#elif defined(TARGET_ARM64) || defined(TARGET_LOONGARCH64)
const int adjustment = ppInfo->TotalFrameSize();
#endif
diff --git a/src/coreclr/jit/likelyclass.cpp b/src/coreclr/jit/likelyclass.cpp
index ed877fdc5da8fa..86f4784b7e6a81 100644
--- a/src/coreclr/jit/likelyclass.cpp
+++ b/src/coreclr/jit/likelyclass.cpp
@@ -121,6 +121,11 @@ static unsigned getLikelyClassesOrMethods(LikelyClassMethodRecord*
int32_t ilOffset,
bool types)
{
+ if (maxLikelyClasses == 0)
+ {
+ return 0;
+ }
+
ICorJitInfo::PgoInstrumentationKind histogramKind =
types ? ICorJitInfo::PgoInstrumentationKind::HandleHistogramTypes
: ICorJitInfo::PgoInstrumentationKind::HandleHistogramMethods;
@@ -235,6 +240,12 @@ static unsigned getLikelyClassesOrMethods(LikelyClassMethodRecord*
}
}
+ if (knownHandles == 0)
+ {
+ // We don't have known handles
+ return 0;
+ }
+
// sort by m_count (descending)
jitstd::sort(sortedEntries, sortedEntries + knownHandles,
[](const LikelyClassMethodHistogramEntry& h1,
@@ -244,12 +255,27 @@ static unsigned getLikelyClassesOrMethods(LikelyClassMethodRecord*
const UINT32 numberOfClasses = min(knownHandles, maxLikelyClasses);
+ UINT32 totalLikelihood = 0;
for (size_t hIdx = 0; hIdx < numberOfClasses; hIdx++)
{
LikelyClassMethodHistogramEntry const hc = sortedEntries[hIdx];
pLikelyEntries[hIdx].handle = hc.m_handle;
pLikelyEntries[hIdx].likelihood = hc.m_count * 100 / h.m_totalCount;
+ totalLikelihood += pLikelyEntries[hIdx].likelihood;
+ }
+
+ assert(totalLikelihood <= 100);
+
+ // Distribute the rounding error and just apply it to the first entry.
+ // Assume that there is no error If we have unknown handles.
+ if (numberOfClasses == h.m_totalCount)
+ {
+ assert(numberOfClasses > 0);
+ assert(totalLikelihood > 0);
+ pLikelyEntries[0].likelihood += 100 - totalLikelihood;
+ assert(pLikelyEntries[0].likelihood <= 100);
}
+
return numberOfClasses;
}
}
diff --git a/src/coreclr/jit/lower.h b/src/coreclr/jit/lower.h
index aabb28592803be..2afbeb1b23df54 100644
--- a/src/coreclr/jit/lower.h
+++ b/src/coreclr/jit/lower.h
@@ -110,6 +110,9 @@ class Lowering final : public Phase
#ifdef FEATURE_HW_INTRINSICS
void ContainCheckHWIntrinsicAddr(GenTreeHWIntrinsic* node, GenTree* addr);
void ContainCheckHWIntrinsic(GenTreeHWIntrinsic* node);
+#ifdef TARGET_XARCH
+ void TryFoldCnsVecForEmbeddedBroadcast(GenTreeHWIntrinsic* parentNode, GenTreeVecCon* childNode);
+#endif // TARGET_XARCH
#endif // FEATURE_HW_INTRINSICS
#ifdef DEBUG
@@ -352,11 +355,11 @@ class Lowering final : public Phase
GenTree* LowerHWIntrinsic(GenTreeHWIntrinsic* node);
void LowerHWIntrinsicCC(GenTreeHWIntrinsic* node, NamedIntrinsic newIntrinsicId, GenCondition condition);
GenTree* LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cmpOp);
- GenTree* LowerHWIntrinsicCmpOpWithKReg(GenTreeHWIntrinsic* node);
GenTree* LowerHWIntrinsicCreate(GenTreeHWIntrinsic* node);
GenTree* LowerHWIntrinsicDot(GenTreeHWIntrinsic* node);
#if defined(TARGET_XARCH)
void LowerFusedMultiplyAdd(GenTreeHWIntrinsic* node);
+ GenTree* LowerHWIntrinsicWithAvx512Mask(GenTreeHWIntrinsic* node);
GenTree* LowerHWIntrinsicToScalar(GenTreeHWIntrinsic* node);
GenTree* LowerHWIntrinsicGetElement(GenTreeHWIntrinsic* node);
GenTree* LowerHWIntrinsicCndSel(GenTreeHWIntrinsic* node);
diff --git a/src/coreclr/jit/lowerriscv64.cpp b/src/coreclr/jit/lowerriscv64.cpp
index 8bd0a8003fe34f..17b1ca46b37543 100644
--- a/src/coreclr/jit/lowerriscv64.cpp
+++ b/src/coreclr/jit/lowerriscv64.cpp
@@ -140,7 +140,7 @@ GenTree* Lowering::LowerJTrue(GenTreeOp* jtrue)
if (op->OperIsCompare() && !varTypeIsFloating(op->gtGetOp1()))
{
- // We do not expect any other relops on LA64
+ // We do not expect any other relops on RISCV64
assert(op->OperIs(GT_EQ, GT_NE, GT_LT, GT_LE, GT_GE, GT_GT));
cond = GenCondition::FromRelop(op);
diff --git a/src/coreclr/jit/lowerxarch.cpp b/src/coreclr/jit/lowerxarch.cpp
index 4b516a5b0ce36a..24a581adf6d431 100644
--- a/src/coreclr/jit/lowerxarch.cpp
+++ b/src/coreclr/jit/lowerxarch.cpp
@@ -894,12 +894,20 @@ void Lowering::LowerHWIntrinsicCC(GenTreeHWIntrinsic* node, NamedIntrinsic newIn
}
break;
- case NI_AVX512F_KORTEST:
case NI_SSE41_PTEST:
case NI_AVX_PTEST:
+ {
// If we need the Carry flag then we can't swap operands.
canSwapOperands = (cc == nullptr) || cc->gtCondition.Is(GenCondition::EQ, GenCondition::NE);
break;
+ }
+
+ case NI_AVX512F_KORTEST:
+ {
+ // TODO-XARCH-AVX512 remove the KORTEST check when its promoted to 2 proper arguments
+ assert(HWIntrinsicInfo::lookupNumArgs(newIntrinsicId) == 1);
+ break;
+ }
default:
unreached();
@@ -1166,28 +1174,16 @@ GenTree* Lowering::LowerHWIntrinsic(GenTreeHWIntrinsic* node)
case NI_Vector128_op_Equality:
case NI_Vector256_op_Equality:
+ case NI_Vector512_op_Equality:
{
return LowerHWIntrinsicCmpOp(node, GT_EQ);
}
case NI_Vector128_op_Inequality:
case NI_Vector256_op_Inequality:
- {
- return LowerHWIntrinsicCmpOp(node, GT_NE);
- }
-
- case NI_Vector512_GreaterThanAll:
- case NI_Vector512_GreaterThanAny:
- case NI_Vector512_GreaterThanOrEqualAll:
- case NI_Vector512_GreaterThanOrEqualAny:
- case NI_Vector512_LessThanAll:
- case NI_Vector512_LessThanAny:
- case NI_Vector512_LessThanOrEqualAll:
- case NI_Vector512_LessThanOrEqualAny:
- case NI_Vector512_op_Equality:
case NI_Vector512_op_Inequality:
{
- return LowerHWIntrinsicCmpOpWithKReg(node);
+ return LowerHWIntrinsicCmpOp(node, GT_NE);
}
case NI_Vector128_ToScalar:
@@ -1614,6 +1610,32 @@ GenTree* Lowering::LowerHWIntrinsic(GenTreeHWIntrinsic* node)
LowerFusedMultiplyAdd(node);
break;
+ case NI_AVX512F_CompareEqual:
+ case NI_AVX512F_CompareGreaterThan:
+ case NI_AVX512F_CompareGreaterThanOrEqual:
+ case NI_AVX512F_CompareLessThan:
+ case NI_AVX512F_CompareLessThanOrEqual:
+ case NI_AVX512F_CompareNotEqual:
+ case NI_AVX512F_VL_CompareGreaterThan:
+ case NI_AVX512F_VL_CompareGreaterThanOrEqual:
+ case NI_AVX512F_VL_CompareLessThan:
+ case NI_AVX512F_VL_CompareLessThanOrEqual:
+ case NI_AVX512F_VL_CompareNotEqual:
+ case NI_AVX512BW_CompareEqual:
+ case NI_AVX512BW_CompareGreaterThan:
+ case NI_AVX512BW_CompareGreaterThanOrEqual:
+ case NI_AVX512BW_CompareLessThan:
+ case NI_AVX512BW_CompareLessThanOrEqual:
+ case NI_AVX512BW_CompareNotEqual:
+ case NI_AVX512BW_VL_CompareGreaterThan:
+ case NI_AVX512BW_VL_CompareGreaterThanOrEqual:
+ case NI_AVX512BW_VL_CompareLessThan:
+ case NI_AVX512BW_VL_CompareLessThanOrEqual:
+ case NI_AVX512BW_VL_CompareNotEqual:
+ {
+ return LowerHWIntrinsicWithAvx512Mask(node);
+ }
+
default:
break;
}
@@ -1638,7 +1660,8 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
var_types simdType = Compiler::getSIMDTypeForSize(simdSize);
assert((intrinsicId == NI_Vector128_op_Equality) || (intrinsicId == NI_Vector128_op_Inequality) ||
- (intrinsicId == NI_Vector256_op_Equality) || (intrinsicId == NI_Vector256_op_Inequality));
+ (intrinsicId == NI_Vector256_op_Equality) || (intrinsicId == NI_Vector256_op_Inequality) ||
+ (intrinsicId == NI_Vector512_op_Equality) || (intrinsicId == NI_Vector512_op_Inequality));
assert(varTypeIsSIMD(simdType));
assert(varTypeIsArithmetic(simdBaseType));
@@ -1655,8 +1678,9 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
GenTree* op2 = node->Op(2);
GenCondition cmpCnd = (cmpOp == GT_EQ) ? GenCondition::EQ : GenCondition::NE;
- if (!varTypeIsFloating(simdBaseType) && op2->IsVectorZero() &&
- comp->compOpportunisticallyDependsOn(InstructionSet_SSE41))
+ if (!varTypeIsFloating(simdBaseType) && (simdSize != 64) && op2->IsVectorZero() &&
+ comp->compOpportunisticallyDependsOn(InstructionSet_SSE41) &&
+ !op1->OperIsHWIntrinsic(NI_AVX512F_ConvertMaskToVector))
{
// On SSE4.1 or higher we can optimize comparisons against zero to
// just use PTEST. We can't support it for floating-point, however,
@@ -1681,14 +1705,269 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
}
else
{
+ assert(simdSize == 16);
+
// TODO-Review: LowerHWIntrinsicCC resets the id again, so why is this needed?
node->ChangeHWIntrinsicId(NI_SSE41_TestZ);
LowerHWIntrinsicCC(node, NI_SSE41_PTEST, cmpCnd);
}
- return node->gtNext;
+ return LowerNode(node);
+ }
+
+ // TODO-XARCH-AVX512: We should handle TYP_SIMD12 here under the EVEX path, but doing
+ // so will require us to account for the unused 4th element.
+
+ if ((simdType != TYP_SIMD12) && comp->IsBaselineVector512IsaSupported())
+ {
+ // The EVEX encoded versions of the comparison instructions all return a kmask
+ //
+ // For the comparisons against zero that we normally optimize to use `PTEST` we
+ // have to make a decision to use EVEX and emit 2 instructions (vpcmp + kortest)
+ // or to continue emitting PTEST and hope that the register allocator isn't limited
+ // by it not supporting the extended register set.
+ //
+ // Ideally we'd opt to not use PTEST when EVEX is available, This would be done so we can
+ // best take advantage of EVEX exclusive features such as embedded broadcast and the
+ // 16 additional registers. In many cases this allows for overall denser codegen where
+ // we are doing more in the same number of bytes, even though the individual instruction
+ // is 1-2 bytes larger. Even though there may be cases where continuing to use PTEST for select-
+ // 128/256-bit code paths would still be beneficial, the additional complexity required
+ // to detect and account for those differences is not likely to be worth the tradeoff.
+ //
+ // TODO-XARCH-AVX512: Given the above don't emit the PTEST path above when AVX-512 is available
+ // This will require exposing `NI_AVX512F_TestZ` so that we can keep codegen optimized to just
+ // `vptestm` followed by `kortest`. This will be one instruction more than just `vptest` but
+ // it has the advantages detailed above.
+ //
+ // For other comparisons, using EVEX allows us to avoid leaving the SIMD domain, avoids
+ // needing to use a general-purpose register, and allows us to generate less instructions.
+
+ GenTree* nextNode = node->gtNext;
+
+ NamedIntrinsic maskIntrinsicId = NI_AVX512F_CompareEqualMask;
+ uint32_t count = simdSize / genTypeSize(simdBaseType);
+
+ // KORTEST does a bitwise or on the result and sets ZF if it is zero and CF if it is all
+ // bits set. Because of this, when we have at least 8 elements to compare we can use a
+ // normal comparison alongside CF.
+ //
+ // That is, if the user wants `x == y`, we can keep it as `mask = (x == y)` and then emit
+ // `kortest mask, mask` and check `CF == 1`. This will be true if all elements matched and
+ // false otherwise. Things work out nicely and we keep readable disasm.
+ //
+ // Likewise, if the user wants `x != y`, we can keep it as `mask = (x != y)` and then emit
+ // `kortest mask, mask` and check `ZF != 0`. This will be true if any elements mismatched.
+ //
+ // However, if we have less than 8 elements then we have to change it up since we have less
+ // than 8 bits in the output mask and unused bits will be set to 0. This occurs for 32-bit
+ // for Vector128 and and 64-bit elements when using either Vector128 or Vector256.
+ //
+ // To account for this, we will invert the comparison being done. So if the user wants
+ // `x == y`, we will instead emit `mask = (x != y)`, we will still emit `kortest mask, mask`,
+ // but we will then check for `ZF == 0`. This works since that equates to all elements being equal
+ //
+ // Likewise for `x != y` we will instead emit `mask = (x == y)`, then `kortest mask, mask`,
+ // and will then check for `CF == 0` which equates to one or more elements not being equal
+
+ // The scenarios we have to for a full mask are:
+ // * No matches: 0000_0000 - ZF == 1, CF == 0
+ // * Partial matches: 0000_1111 - ZF == 0, CF == 0
+ // * All matches: 1111_1111 - ZF == 0, CF == 1
+ //
+ // The scenarios we have to for a partial mask are:
+ // * No matches: 0000_0000 - ZF == 1, CF == 0
+ // * Partial matches: 0000_0011 - ZF == 0, CF == 0
+ // * All matches: 0000_1111 - ZF == 0, CF == 0
+ //
+ // When we have less than a full mask worth of elements, we need to account for the upper
+ // bits being implicitly zero. To do that, we may need to invert the comparison.
+ //
+ // By inverting the comparison we'll get:
+ // * All matches: 0000_0000 - ZF == 1, CF == 0
+ // * Partial matches: 0000_0011 - ZF == 0, CF == 0
+ // * No matches: 0000_1111 - ZF == 0, CF == 0
+ //
+ // This works since the upper bits are implicitly zero and so by inverting matches also become
+ // zero, which in turn means that `AllBitsSet` will become `Zero` and other cases become non-zero
+
+ if (op1->OperIsHWIntrinsic(NI_AVX512F_ConvertMaskToVector) && op2->IsCnsVec())
+ {
+ // We want to specially handle the common cases of `mask op Zero` and `mask op AllBitsSet`
+ //
+ // These get created for the various `gtNewSimdCmpOpAnyNode` and `gtNewSimdCmpOpAllNode`
+ // scenarios and we want to ensure they still get "optimal" codegen. To handle that, we
+ // simply consume the mask directly and preserve the intended comparison by tweaking the
+ // compare condition passed down into `KORTEST`
+
+ GenTreeHWIntrinsic* maskNode = op1->AsHWIntrinsic()->Op(1)->AsHWIntrinsic();
+ assert(maskNode->TypeIs(TYP_MASK));
+
+ bool isHandled = false;
+ GenTreeVecCon* vecCon = op2->AsVecCon();
+
+ if (vecCon->IsZero())
+ {
+ // We have `mask == Zero` which is the same as checking that nothing in the mask
+ // is set. This scenario can be handled by `kortest` and then checking that `ZF == 1`
+ //
+ // -or-
+ //
+ // We have `mask != Zero` which is the same as checking that something in the mask
+ // is set. This scenario can be handled by `kortest` and then checking that `ZF == 0`
+ //
+ // Since this is the default state for `CompareEqualMask` + `GT_EQ`/`GT_NE`, there is nothing
+ // for us to change. This also applies to cases where we have less than a full mask of
+ // elements since the upper mask bits are implicitly zero.
+
+ isHandled = true;
+ }
+ else if (vecCon->IsAllBitsSet())
+ {
+ // We have `mask == AllBitsSet` which is the same as checking that everything in the mask
+ // is set. This scenario can be handled by `kortest` and then checking that `CF == 1` for
+ // a full mask and checking `ZF == 1` for a partial mask using an inverted comparison
+ //
+ // -or-
+ //
+ // We have `mask != AllBitsSet` which is the same as checking that something in the mask
+ // is set. This scenario can be handled by `kortest` and then checking that `CF == 0` for
+ // a full mask and checking `ZF != 0` for a partial mask using an inverted comparison
+
+ if (count < 8)
+ {
+ assert((count == 1) || (count == 2) || (count == 4));
+
+ switch (maskNode->GetHWIntrinsicId())
+ {
+ case NI_AVX512F_CompareEqualMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareNotEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_CompareGreaterThanMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareLessThanOrEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_CompareGreaterThanOrEqualMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareLessThanMask;
+ break;
+ }
+
+ case NI_AVX512F_CompareLessThanMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareGreaterThanOrEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_CompareLessThanOrEqualMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareGreaterThanMask;
+ break;
+ }
+
+ case NI_AVX512F_CompareNotEqualMask:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareEqualMask;
+ break;
+ }
+
+ default:
+ {
+ unreached();
+ }
+ }
+
+ maskNode->ChangeHWIntrinsicId(maskIntrinsicId);
+ }
+ else if (cmpOp == GT_EQ)
+ {
+ cmpCnd = GenCondition::C;
+ }
+ else
+ {
+ cmpCnd = GenCondition::NC;
+ }
+ isHandled = true;
+ }
+
+ if (isHandled)
+ {
+ LIR::Use use;
+ if (BlockRange().TryGetUse(node, &use))
+ {
+ use.ReplaceWith(maskNode);
+ }
+ else
+ {
+ maskNode->SetUnusedValue();
+ }
+
+ BlockRange().Remove(op2);
+ BlockRange().Remove(op1);
+ BlockRange().Remove(node);
+
+ node = maskNode;
+ }
+ }
+
+ if (node->gtType != TYP_MASK)
+ {
+ // We have `x == y` or `x != y` both of which where we want to find `AllBitsSet` in the mask since
+ // we can directly do the relevant comparison. Given the above tables then when we have a full mask
+ // we can simply check against `CF == 1` for `op_Equality` and `ZF == 0` for `op_Inequality`.
+ //
+ // For a partial mask, we need to invert the `op_Equality` comparisons which means that we now need
+ // to check for `ZF == 1` (we're looking for `AllBitsSet`, that is `all match`). For `op_Inequality`
+ // we can keep things as is since we're looking for `any match` and just want to check `ZF == 0`
+
+ if (count < 8)
+ {
+ assert((count == 1) || (count == 2) || (count == 4));
+ maskIntrinsicId = NI_AVX512F_CompareNotEqualMask;
+ }
+ else
+ {
+ assert((count == 8) || (count == 16) || (count == 32) || (count == 64));
+
+ if (cmpOp == GT_EQ)
+ {
+ cmpCnd = GenCondition::C;
+ }
+ else
+ {
+ maskIntrinsicId = NI_AVX512F_CompareNotEqualMask;
+ }
+ }
+
+ node->gtType = TYP_MASK;
+ node->ChangeHWIntrinsicId(maskIntrinsicId);
+
+ LowerNode(node);
+ }
+
+ LIR::Use use;
+ if (BlockRange().TryGetUse(node, &use))
+ {
+ GenTreeHWIntrinsic* cc;
+
+ cc = comp->gtNewSimdHWIntrinsicNode(simdType, node, NI_AVX512F_KORTEST, simdBaseJitType, simdSize);
+ BlockRange().InsertBefore(nextNode, cc);
+
+ use.ReplaceWith(cc);
+ LowerHWIntrinsicCC(cc, NI_AVX512F_KORTEST, cmpCnd);
+
+ nextNode = cc->gtNext;
+ }
+ return nextNode;
}
+ assert(simdSize != 64);
+
NamedIntrinsic cmpIntrinsic;
CorInfoType cmpJitType;
NamedIntrinsic mskIntrinsic;
@@ -1728,11 +2007,11 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
case TYP_ULONG:
{
mskJitType = CORINFO_TYPE_UBYTE;
+ cmpJitType = simdBaseJitType;
if (simdSize == 32)
{
cmpIntrinsic = NI_AVX2_CompareEqual;
- cmpJitType = simdBaseJitType;
mskIntrinsic = NI_AVX2_MoveMask;
mskConstant = -1;
}
@@ -1743,7 +2022,6 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
if (comp->compOpportunisticallyDependsOn(InstructionSet_SSE41))
{
cmpIntrinsic = NI_SSE41_CompareEqual;
- cmpJitType = simdBaseJitType;
}
else
{
@@ -1855,80 +2133,6 @@ GenTree* Lowering::LowerHWIntrinsicCmpOp(GenTreeHWIntrinsic* node, genTreeOps cm
return LowerNode(node);
}
-//----------------------------------------------------------------------------------------------
-// Lowering::LowerHWIntrinsicCmpOpWithKReg: Lowers a Vector512 comparison intrinsic
-//
-// Arguments:
-// node - The hardware intrinsic node.
-//
-GenTree* Lowering::LowerHWIntrinsicCmpOpWithKReg(GenTreeHWIntrinsic* node)
-{
- NamedIntrinsic intrinsicId = node->GetHWIntrinsicId();
- CorInfoType simdBaseJitType = node->GetSimdBaseJitType();
- var_types simdBaseType = node->GetSimdBaseType();
- unsigned simdSize = node->GetSimdSize();
- var_types simdType = Compiler::getSIMDTypeForSize(simdSize);
-
- assert((intrinsicId == NI_Vector512_GreaterThanAll) || (intrinsicId == NI_Vector512_GreaterThanOrEqualAll) ||
- (intrinsicId == NI_Vector512_LessThanAll) || (intrinsicId == NI_Vector512_LessThanOrEqualAll) ||
- (intrinsicId == NI_Vector512_op_Equality) || (intrinsicId == NI_Vector512_op_Inequality));
-
- assert(varTypeIsSIMD(simdType));
- assert(varTypeIsArithmetic(simdBaseType));
- assert(simdSize == 64);
- assert(node->gtType == TYP_BOOL);
-
- NamedIntrinsic newIntrinsicId = NI_Illegal;
- switch (intrinsicId)
- {
- case NI_Vector512_GreaterThanAll:
- {
- newIntrinsicId = NI_AVX512F_CompareGreaterThanSpecial;
- break;
- }
- case NI_Vector512_GreaterThanOrEqualAll:
- {
- newIntrinsicId = NI_AVX512F_CompareGreaterThanOrEqualSpecial;
- break;
- }
- case NI_Vector512_LessThanAll:
- {
- newIntrinsicId = NI_AVX512F_CompareLessThanSpecial;
- break;
- }
- case NI_Vector512_LessThanOrEqualAll:
- {
- newIntrinsicId = NI_AVX512F_CompareLessThanOrEqualSpecial;
- break;
- }
- case NI_Vector512_op_Equality:
- case NI_Vector512_op_Inequality:
- {
- newIntrinsicId = NI_AVX512F_CompareEqualSpecial;
- break;
- }
-
- default:
- {
- assert(false);
- break;
- }
- }
-
- GenTree* op1 = node->Op(1);
- GenTree* op2 = node->Op(2);
-
- GenTree* cmp = comp->gtNewSimdHWIntrinsicNode(TYP_MASK, op1, op2, newIntrinsicId, simdBaseJitType, simdSize);
- BlockRange().InsertBefore(node, cmp);
- LowerNode(cmp);
-
- node->ResetHWIntrinsicId(NI_AVX512F_KORTEST, cmp);
- GenCondition cmpCnd = (intrinsicId != NI_Vector512_op_Inequality) ? GenCondition::C : GenCondition::NC;
- LowerHWIntrinsicCC(node, NI_AVX512F_KORTEST, cmpCnd);
-
- return node->gtNext;
-}
-
//----------------------------------------------------------------------------------------------
// Lowering::LowerHWIntrinsicCndSel: Lowers a Vector128 or Vector256 Conditional Select call
//
@@ -2312,12 +2516,13 @@ GenTree* Lowering::LowerHWIntrinsicCreate(GenTreeHWIntrinsic* node)
node->ResetHWIntrinsicId(NI_AVX512BW_BroadcastScalarToVector512, tmp1);
break;
}
+
case TYP_INT:
case TYP_UINT:
- case TYP_LONG:
- case TYP_ULONG:
case TYP_FLOAT:
case TYP_DOUBLE:
+ case TYP_LONG:
+ case TYP_ULONG:
{
assert(comp->compIsaSupportedDebugOnly(InstructionSet_AVX512F));
node->ResetHWIntrinsicId(NI_AVX512F_BroadcastScalarToVector512, tmp1);
@@ -2353,7 +2558,6 @@ GenTree* Lowering::LowerHWIntrinsicCreate(GenTreeHWIntrinsic* node)
LowerNode(tmp1);
node->ResetHWIntrinsicId(NI_AVX2_BroadcastScalarToVector256, tmp1);
-
return LowerNode(node);
}
@@ -2428,7 +2632,6 @@ GenTree* Lowering::LowerHWIntrinsicCreate(GenTreeHWIntrinsic* node)
// return Avx2.BroadcastScalarToVector128(tmp1);
node->ChangeHWIntrinsicId(NI_AVX2_BroadcastScalarToVector128, tmp1);
-
return LowerNode(node);
}
@@ -4826,6 +5029,125 @@ GenTree* Lowering::LowerHWIntrinsicDot(GenTreeHWIntrinsic* node)
}
}
+//----------------------------------------------------------------------------------------------
+// Lowering::LowerHWIntrinsicWithAvx512Mask: Lowers a HWIntrinsic node that utilizes the AVX512 KMASK registers
+//
+// Arguments:
+// node - The hardware intrinsic node.
+//
+GenTree* Lowering::LowerHWIntrinsicWithAvx512Mask(GenTreeHWIntrinsic* node)
+{
+ NamedIntrinsic intrinsicId = node->GetHWIntrinsicId();
+ CorInfoType simdBaseJitType = node->GetSimdBaseJitType();
+ var_types simdBaseType = node->GetSimdBaseType();
+ unsigned simdSize = node->GetSimdSize();
+ var_types simdType = Compiler::getSIMDTypeForSize(simdSize);
+
+ assert(varTypeIsSIMD(simdType));
+ assert(varTypeIsArithmetic(simdBaseType));
+ assert(simdSize != 0);
+
+ NamedIntrinsic maskIntrinsicId;
+
+ switch (intrinsicId)
+ {
+ case NI_AVX512F_CompareEqual:
+ case NI_AVX512BW_CompareEqual:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_VL_CompareGreaterThan:
+ case NI_AVX512BW_VL_CompareGreaterThan:
+ {
+ assert(varTypeIsUnsigned(simdBaseType));
+ FALLTHROUGH;
+ }
+
+ case NI_AVX512F_CompareGreaterThan:
+ case NI_AVX512BW_CompareGreaterThan:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareGreaterThanMask;
+ break;
+ }
+
+ case NI_AVX512F_VL_CompareGreaterThanOrEqual:
+ case NI_AVX512BW_VL_CompareGreaterThanOrEqual:
+ {
+ assert(!varTypeIsFloating(simdBaseType));
+ FALLTHROUGH;
+ }
+
+ case NI_AVX512F_CompareGreaterThanOrEqual:
+ case NI_AVX512BW_CompareGreaterThanOrEqual:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareGreaterThanOrEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_VL_CompareLessThan:
+ case NI_AVX512BW_VL_CompareLessThan:
+ {
+ assert(varTypeIsUnsigned(simdBaseType));
+ FALLTHROUGH;
+ }
+
+ case NI_AVX512F_CompareLessThan:
+ case NI_AVX512BW_CompareLessThan:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareLessThanMask;
+ break;
+ }
+
+ case NI_AVX512F_VL_CompareLessThanOrEqual:
+ case NI_AVX512BW_VL_CompareLessThanOrEqual:
+ {
+ assert(!varTypeIsFloating(simdBaseType));
+ FALLTHROUGH;
+ }
+
+ case NI_AVX512F_CompareLessThanOrEqual:
+ case NI_AVX512BW_CompareLessThanOrEqual:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareLessThanOrEqualMask;
+ break;
+ }
+
+ case NI_AVX512F_VL_CompareNotEqual:
+ case NI_AVX512BW_VL_CompareNotEqual:
+ {
+ assert(!varTypeIsFloating(simdBaseType));
+ FALLTHROUGH;
+ }
+
+ case NI_AVX512F_CompareNotEqual:
+ case NI_AVX512BW_CompareNotEqual:
+ {
+ maskIntrinsicId = NI_AVX512F_CompareNotEqualMask;
+ break;
+ }
+
+ default:
+ {
+ unreached();
+ }
+ }
+
+ node->gtType = TYP_MASK;
+ node->ChangeHWIntrinsicId(maskIntrinsicId);
+
+ LIR::Use use;
+ if (BlockRange().TryGetUse(node, &use))
+ {
+ GenTree* maskToVector =
+ comp->gtNewSimdHWIntrinsicNode(simdType, node, NI_AVX512F_ConvertMaskToVector, simdBaseJitType, simdSize);
+ BlockRange().InsertAfter(node, maskToVector);
+ use.ReplaceWith(maskToVector);
+ }
+ return LowerNode(node);
+}
+
//----------------------------------------------------------------------------------------------
// Lowering::LowerHWIntrinsicToScalar: Lowers a Vector128 or Vector256 ToScalar call
//
@@ -7578,7 +7900,6 @@ bool Lowering::IsContainableHWIntrinsicOp(GenTreeHWIntrinsic* parentNode, GenTre
canBeContained = !vecCon->IsAllBitsSet() && !vecCon->IsZero();
}
}
-
return canBeContained;
}
@@ -7601,7 +7922,7 @@ bool Lowering::IsContainableHWIntrinsicOp(GenTreeHWIntrinsic* parentNode, GenTre
GenTree* op1 = hwintrinsic->Op(1);
- if (IsSafeToContainMem(parentNode, hwintrinsic, op1))
+ if (IsInvariantInRange(op1, parentNode, hwintrinsic))
{
if (op1->isContained())
{
@@ -7699,6 +8020,51 @@ bool Lowering::IsContainableHWIntrinsicOp(GenTreeHWIntrinsic* parentNode, GenTre
return false;
}
+ case NI_SSE3_MoveAndDuplicate:
+ case NI_AVX2_BroadcastScalarToVector128:
+ case NI_AVX2_BroadcastScalarToVector256:
+ case NI_AVX512F_BroadcastScalarToVector512:
+ {
+ // make the broadcast node containable when embedded broadcast can be enabled.
+ if (intrinsicId == NI_SSE3_MoveAndDuplicate)
+ {
+ // NI_SSE3_MoveAndDuplicate is for Vector128 only.
+ assert(hwintrinsic->GetSimdBaseType() == TYP_DOUBLE);
+ }
+
+ if (comp->compOpportunisticallyDependsOn(InstructionSet_AVX512F_VL) &&
+ parentNode->OperIsEmbBroadcastCompatible())
+ {
+ GenTree* broadcastOperand = hwintrinsic->Op(1);
+
+ if (broadcastOperand->OperIsHWIntrinsic())
+ {
+ GenTreeHWIntrinsic* hwintrinsicOperand = broadcastOperand->AsHWIntrinsic();
+
+ if (hwintrinsicOperand->OperIsCreateScalarUnsafe())
+ {
+ // CreateScalarUnsafe can contain non-memory operands such as enregistered
+ // locals, so we want to check if its operand is containable instead. This
+ // will result in such enregistered locals returning `false`.
+ broadcastOperand = hwintrinsicOperand->Op(1);
+ }
+ }
+
+ bool childSupportsRegOptional;
+ if (IsContainableHWIntrinsicOp(hwintrinsic, broadcastOperand, &childSupportsRegOptional))
+ {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ case NI_AVX_BroadcastScalarToVector128:
+ case NI_AVX_BroadcastScalarToVector256:
+ {
+ return parentNode->OperIsEmbBroadcastCompatible();
+ }
+
default:
{
assert(!childNode->isContainableHWIntrinsic());
@@ -7707,6 +8073,153 @@ bool Lowering::IsContainableHWIntrinsicOp(GenTreeHWIntrinsic* parentNode, GenTre
}
}
+//----------------------------------------------------------------------------------------------
+// TryFoldCnsVecForEmbeddedBroadcast:
+// Unfold the eligible constant vector when embedded broadcast is
+// available.
+//
+// Arguments:
+// parentNode - The hardware intrinsic node
+// childNode - The operand node to try contain
+//
+void Lowering::TryFoldCnsVecForEmbeddedBroadcast(GenTreeHWIntrinsic* parentNode, GenTreeVecCon* childNode)
+{
+ assert(!childNode->IsAllBitsSet());
+ assert(!childNode->IsZero());
+ var_types simdType = parentNode->TypeGet();
+ var_types simdBaseType = parentNode->GetSimdBaseType();
+ CorInfoType simdBaseJitType = parentNode->GetSimdBaseJitType();
+ bool isCreatedFromScalar = true;
+ int elementCount = GenTreeVecCon::ElementCount(genTypeSize(simdType), simdBaseType);
+ switch (simdBaseType)
+ {
+ case TYP_FLOAT:
+ case TYP_INT:
+ case TYP_UINT:
+ {
+ uint32_t firstElement = static_cast(childNode->gtSimdVal.u32[0]);
+ for (int i = 1; i < elementCount; i++)
+ {
+ uint32_t elementToCheck = static_cast(childNode->gtSimdVal.u32[i]);
+ if (firstElement != elementToCheck)
+ {
+ isCreatedFromScalar = false;
+ break;
+ }
+ }
+ break;
+ }
+
+ case TYP_DOUBLE:
+#if defined(TARGET_AMD64)
+ case TYP_LONG:
+ case TYP_ULONG:
+#endif // TARGET_AMD64
+ {
+ uint64_t firstElement = static_cast(childNode->gtSimdVal.u64[0]);
+ for (int i = 1; i < elementCount; i++)
+ {
+ uint64_t elementToCheck = static_cast(childNode->gtSimdVal.u64[i]);
+ if (firstElement != elementToCheck)
+ {
+ isCreatedFromScalar = false;
+ break;
+ }
+ }
+ break;
+ }
+
+ default:
+ isCreatedFromScalar = false;
+ break;
+ }
+ if (isCreatedFromScalar)
+ {
+ NamedIntrinsic broadcastName = NI_AVX2_BroadcastScalarToVector128;
+ if (simdType == TYP_SIMD32)
+ {
+ broadcastName = NI_AVX2_BroadcastScalarToVector256;
+ }
+ else if (simdType == TYP_SIMD64)
+ {
+ broadcastName = NI_AVX512F_BroadcastScalarToVector512;
+ }
+ else
+ {
+ assert(simdType == TYP_SIMD16);
+ }
+ GenTree* constScalar = nullptr;
+ switch (simdBaseType)
+ {
+ case TYP_FLOAT:
+ {
+ float scalar = static_cast(childNode->gtSimdVal.f32[0]);
+ constScalar = comp->gtNewDconNode(scalar, simdBaseType);
+ break;
+ }
+ case TYP_DOUBLE:
+ {
+ double scalar = static_cast(childNode->gtSimdVal.f64[0]);
+ constScalar = comp->gtNewDconNode(scalar, simdBaseType);
+ break;
+ }
+ case TYP_INT:
+ {
+ int32_t scalar = static_cast(childNode->gtSimdVal.i32[0]);
+ constScalar = comp->gtNewIconNode(scalar, simdBaseType);
+ break;
+ }
+ case TYP_UINT:
+ {
+ uint32_t scalar = static_cast(childNode->gtSimdVal.u32[0]);
+ constScalar = comp->gtNewIconNode(scalar, TYP_INT);
+ break;
+ }
+#if defined(TARGET_AMD64)
+ case TYP_LONG:
+ {
+ int64_t scalar = static_cast(childNode->gtSimdVal.i64[0]);
+ constScalar = comp->gtNewIconNode(scalar, simdBaseType);
+ break;
+ }
+ case TYP_ULONG:
+ {
+ uint64_t scalar = static_cast(childNode->gtSimdVal.u64[0]);
+ constScalar = comp->gtNewIconNode(scalar, TYP_LONG);
+ break;
+ }
+#endif // TARGET_AMD64
+ default:
+ unreached();
+ }
+ GenTreeHWIntrinsic* createScalar =
+ comp->gtNewSimdHWIntrinsicNode(TYP_SIMD16, constScalar, NI_Vector128_CreateScalarUnsafe, simdBaseJitType,
+ 16);
+ GenTreeHWIntrinsic* broadcastNode = comp->gtNewSimdHWIntrinsicNode(simdType, createScalar, broadcastName,
+ simdBaseJitType, genTypeSize(simdType));
+ BlockRange().InsertBefore(childNode, broadcastNode);
+ BlockRange().InsertBefore(broadcastNode, createScalar);
+ BlockRange().InsertBefore(createScalar, constScalar);
+ LIR::Use use;
+ BlockRange().TryGetUse(childNode, &use);
+ use.ReplaceWith(broadcastNode);
+ BlockRange().Remove(childNode);
+ LowerNode(createScalar);
+ LowerNode(broadcastNode);
+ if (varTypeIsFloating(simdBaseType))
+ {
+ MakeSrcContained(broadcastNode, createScalar);
+ }
+ else if (constScalar->TypeIs(TYP_INT, TYP_UINT, TYP_LONG, TYP_ULONG))
+ {
+ MakeSrcContained(broadcastNode, constScalar);
+ }
+ MakeSrcContained(parentNode, broadcastNode);
+ return;
+ }
+ MakeSrcContained(parentNode, childNode);
+}
+
//----------------------------------------------------------------------------------------------
// ContainCheckHWIntrinsicAddr: Perform containment analysis for an address operand of a hardware
// intrinsic node.
@@ -8015,13 +8528,29 @@ void Lowering::ContainCheckHWIntrinsic(GenTreeHWIntrinsic* node)
if (IsContainableHWIntrinsicOp(node, op2, &supportsOp2RegOptional))
{
- MakeSrcContained(node, op2);
+ if (op2->OperIs(GT_CNS_VEC) && comp->compOpportunisticallyDependsOn(InstructionSet_AVX512F) &&
+ node->OperIsEmbBroadcastCompatible())
+ {
+ TryFoldCnsVecForEmbeddedBroadcast(node, op2->AsVecCon());
+ }
+ else
+ {
+ MakeSrcContained(node, op2);
+ }
}
else if ((isCommutative || (intrinsicId == NI_BMI2_MultiplyNoFlags) ||
(intrinsicId == NI_BMI2_X64_MultiplyNoFlags)) &&
IsContainableHWIntrinsicOp(node, op1, &supportsOp1RegOptional))
{
- MakeSrcContained(node, op1);
+ if (op1->OperIs(GT_CNS_VEC) && comp->compOpportunisticallyDependsOn(InstructionSet_AVX512F) &&
+ node->OperIsEmbBroadcastCompatible())
+ {
+ TryFoldCnsVecForEmbeddedBroadcast(node, op1->AsVecCon());
+ }
+ else
+ {
+ MakeSrcContained(node, op1);
+ }
// Swap the operands here to make the containment checks in codegen significantly simpler
std::swap(node->Op(1), node->Op(2));
diff --git a/src/coreclr/jit/lsraxarch.cpp b/src/coreclr/jit/lsraxarch.cpp
index 765de402b90e23..08ae13c6cff2f3 100644
--- a/src/coreclr/jit/lsraxarch.cpp
+++ b/src/coreclr/jit/lsraxarch.cpp
@@ -1166,7 +1166,7 @@ int LinearScan::BuildCall(GenTreeCall* call)
// The return value will be on the X87 stack, and we will need to move it.
dstCandidates = allRegs(registerType);
#else // !TARGET_X86
- dstCandidates = RBM_FLOATRET;
+ dstCandidates = RBM_FLOATRET;
#endif // !TARGET_X86
}
else
@@ -1378,12 +1378,10 @@ int LinearScan::BuildBlockStore(GenTreeBlk* blkNode)
{
case GenTreeBlk::BlkOpKindUnroll:
{
-#ifdef TARGET_AMD64
- const bool canUse16BytesSimdMov = !blkNode->IsOnHeapAndContainsReferences();
- const bool willUseSimdMov = canUse16BytesSimdMov && (size >= 16);
-#else
- const bool willUseSimdMov = (size >= 16);
-#endif
+ const bool canUse16BytesSimdMov =
+ !blkNode->IsOnHeapAndContainsReferences() && compiler->IsBaselineSimdIsaSupported();
+ const bool willUseSimdMov = canUse16BytesSimdMov && (size >= XMM_REGSIZE_BYTES);
+
if (willUseSimdMov)
{
buildInternalFloatRegisterDefForNode(blkNode, internalFloatRegCandidates());
@@ -1440,8 +1438,26 @@ int LinearScan::BuildBlockStore(GenTreeBlk* blkNode)
break;
case GenTreeBlk::BlkOpKindUnroll:
- if ((size % XMM_REGSIZE_BYTES) != 0)
+ {
+ unsigned regSize = compiler->roundDownSIMDSize(size);
+ unsigned remainder = size;
+
+ if ((size >= regSize) && (regSize > 0))
+ {
+ // We need a float temporary if we're doing SIMD operations
+
+ buildInternalFloatRegisterDefForNode(blkNode, internalFloatRegCandidates());
+ SetContainsAVXFlags(size);
+
+ remainder %= regSize;
+ }
+
+ if ((remainder > 0) && ((regSize == 0) || (isPow2(remainder) && (remainder <= REGSIZE_BYTES))))
{
+ // We need an int temporary if we're not doing SIMD operations
+ // or if are but the remainder is a power of 2 and less than the
+ // size of a register
+
regMaskTP regMask = availableIntRegs;
#ifdef TARGET_X86
if ((size & 1) != 0)
@@ -1453,13 +1469,8 @@ int LinearScan::BuildBlockStore(GenTreeBlk* blkNode)
#endif
internalIntDef = buildInternalIntRegisterDefForNode(blkNode, regMask);
}
-
- if (size >= XMM_REGSIZE_BYTES)
- {
- buildInternalFloatRegisterDefForNode(blkNode, internalFloatRegCandidates());
- SetContainsAVXFlags(size);
- }
break;
+ }
case GenTreeBlk::BlkOpKindUnrollMemmove:
{
diff --git a/src/coreclr/jit/namedintrinsiclist.h b/src/coreclr/jit/namedintrinsiclist.h
index 50fb3e3ea214bc..6bc31d315b5c36 100644
--- a/src/coreclr/jit/namedintrinsiclist.h
+++ b/src/coreclr/jit/namedintrinsiclist.h
@@ -126,11 +126,11 @@ enum NamedIntrinsic : unsigned short
#ifdef FEATURE_HW_INTRINSICS
NI_HW_INTRINSIC_START,
#if defined(TARGET_XARCH)
-#define HARDWARE_INTRINSIC(isa, name, size, numarg, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+#define HARDWARE_INTRINSIC(isa, name, size, numarg, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
NI_##isa##_##name,
#include "hwintrinsiclistxarch.h"
#elif defined(TARGET_ARM64)
-#define HARDWARE_INTRINSIC(isa, name, size, numarg, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+#define HARDWARE_INTRINSIC(isa, name, size, numarg, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
NI_##isa##_##name,
#include "hwintrinsiclistarm64.h"
#endif // !defined(TARGET_XARCH) && !defined(TARGET_ARM64)
diff --git a/src/coreclr/jit/simd.cpp b/src/coreclr/jit/simd.cpp
index 97b7d06e51ebd0..2e10faae2f0188 100644
--- a/src/coreclr/jit/simd.cpp
+++ b/src/coreclr/jit/simd.cpp
@@ -299,7 +299,10 @@ CorInfoType Compiler::getBaseJitTypeAndSizeOfSIMDType(CORINFO_CLASS_HANDLE typeH
JITDUMP(" Found Vector<%s>\n", varTypeName(JitType2PreciseVarType(simdBaseJitType)));
size = getVectorTByteLength();
- assert(size != 0);
+ if (size == 0)
+ {
+ return CORINFO_TYPE_UNDEF;
+ }
break;
}
diff --git a/src/coreclr/jit/simdashwintrinsic.cpp b/src/coreclr/jit/simdashwintrinsic.cpp
index 0cb64b37428e37..7f71f1c62ecf8b 100644
--- a/src/coreclr/jit/simdashwintrinsic.cpp
+++ b/src/coreclr/jit/simdashwintrinsic.cpp
@@ -219,8 +219,14 @@ SimdAsHWIntrinsicClassId SimdAsHWIntrinsicInfo::lookupClassId(Compiler* comp,
}
#endif // TARGET_XARCH
- assert(vectorTByteLength == 16);
- return SimdAsHWIntrinsicClassId::VectorT128;
+ if (vectorTByteLength == 16)
+ {
+ return SimdAsHWIntrinsicClassId::VectorT128;
+ }
+ else
+ {
+ return SimdAsHWIntrinsicClassId::Unknown;
+ }
}
break;
}
diff --git a/src/coreclr/jit/ssabuilder.cpp b/src/coreclr/jit/ssabuilder.cpp
index b78786a7fa7de1..774e51ee8f9872 100644
--- a/src/coreclr/jit/ssabuilder.cpp
+++ b/src/coreclr/jit/ssabuilder.cpp
@@ -143,7 +143,7 @@ SsaBuilder::SsaBuilder(Compiler* pCompiler)
//
// Return Value:
// The number of nodes visited while performing DFS on the graph.
-
+//
int SsaBuilder::TopologicalSort(BasicBlock** postOrder, int count)
{
Compiler* comp = m_pCompiler;
@@ -164,15 +164,14 @@ int SsaBuilder::TopologicalSort(BasicBlock** postOrder, int count)
unsigned index = 0;
while (true)
{
- bool isEHsucc = successors.IsNextEHSuccessor();
- BasicBlock* succ = successors.NextSuccessor(comp);
+ BasicBlock* succ = successors.NextSuccessor(comp);
if (succ == nullptr)
{
break;
}
- printf("%s%s" FMT_BB, (index++ ? ", " : ""), (isEHsucc ? "[EH]" : ""), succ->bbNum);
+ printf("%s" FMT_BB, (index++ ? ", " : ""), succ->bbNum);
}
printf("]\n");
}
@@ -210,7 +209,7 @@ int SsaBuilder::TopologicalSort(BasicBlock** postOrder, int count)
DBG_SSA_JITDUMP("[SsaBuilder::TopologicalSort] postOrder[%d] = " FMT_BB "\n", postIndex, block->bbNum);
postOrder[postIndex] = block;
block->bbPostorderNum = postIndex;
- postIndex += 1;
+ postIndex++;
}
}
diff --git a/src/coreclr/jit/valuenum.cpp b/src/coreclr/jit/valuenum.cpp
index 573fc0c0d054a3..da623ad9ae63f1 100644
--- a/src/coreclr/jit/valuenum.cpp
+++ b/src/coreclr/jit/valuenum.cpp
@@ -8852,7 +8852,65 @@ void ValueNumStore::vnDumpZeroObj(Compiler* comp, VNFuncApp* zeroObj)
#endif // DEBUG
// Static fields, methods.
-static UINT8 vnfOpAttribs[VNF_COUNT];
+
+#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic, extra) \
+ static_assert((arity) >= 0 || !(extra), "valuenumfuncs.h has EncodesExtraTypeArg==true and arity<0 for " #vnf);
+#include "valuenumfuncs.h"
+
+#ifdef FEATURE_HW_INTRINSICS
+
+#define HARDWARE_INTRINSIC(isa, name, size, argCount, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+ static_assert((size) != 0 || !(extra), \
+ "hwintrinsicslist.h has EncodesExtraTypeArg==true and size==0 for " #isa " " #name);
+#if defined(TARGET_XARCH)
+#include "hwintrinsiclistxarch.h"
+#elif defined(TARGET_ARM64)
+#include "hwintrinsiclistarm64.h"
+#else
+#error Unsupported platform
+#endif
+
+#endif // FEATURE_HW_INTRINSICS
+
+/* static */ constexpr uint8_t ValueNumStore::GetOpAttribsForArity(genTreeOps oper, GenTreeOperKind kind)
+{
+ return ((GenTree::StaticOperIs(oper, GT_SELECT) ? 3 : (((kind & GTK_UNOP) >> 1) | ((kind & GTK_BINOP) >> 1)))
+ << VNFOA_ArityShift) &
+ VNFOA_ArityMask;
+}
+
+/* static */ constexpr uint8_t ValueNumStore::GetOpAttribsForGenTree(genTreeOps oper,
+ bool commute,
+ bool illegalAsVNFunc,
+ GenTreeOperKind kind)
+{
+ return GetOpAttribsForArity(oper, kind) | (static_cast(commute) << VNFOA_CommutativeShift) |
+ (static_cast(illegalAsVNFunc) << VNFOA_IllegalGenTreeOpShift);
+}
+
+/* static */ constexpr uint8_t ValueNumStore::GetOpAttribsForFunc(int arity,
+ bool commute,
+ bool knownNonNull,
+ bool sharedStatic)
+{
+ return (static_cast(commute) << VNFOA_CommutativeShift) |
+ (static_cast(knownNonNull) << VNFOA_KnownNonNullShift) |
+ (static_cast(sharedStatic) << VNFOA_SharedStaticShift) |
+ ((static_cast(arity & ~(arity >> 31)) << VNFOA_ArityShift) & VNFOA_ArityMask);
+}
+
+const uint8_t ValueNumStore::s_vnfOpAttribs[VNF_COUNT] = {
+#define GTNODE(en, st, cm, ivn, ok) \
+ GetOpAttribsForGenTree(static_cast(GT_##en), cm, ivn, static_cast(ok)),
+#include "gtlist.h"
+
+ 0, // VNF_Boundary
+
+#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic, extra) \
+ GetOpAttribsForFunc((arity) + static_cast(extra), commute, knownNonNull, sharedStatic),
+#include "valuenumfuncs.h"
+};
+
static genTreeOps genTreeOpsIllegalAsVNFunc[] = {GT_IND, // When we do heap memory.
GT_NULLCHECK, GT_QMARK, GT_COLON, GT_LOCKADD, GT_XADD, GT_XCHG,
GT_CMPXCHG, GT_LCLHEAP, GT_BOX, GT_XORR, GT_XAND, GT_STORE_DYN_BLK,
@@ -8869,15 +8927,10 @@ static genTreeOps genTreeOpsIllegalAsVNFunc[] = {GT_IND, // When we do heap memo
// These control-flow operations need no values.
GT_JTRUE, GT_RETURN, GT_SWITCH, GT_RETFILT, GT_CKFINITE};
-UINT8* ValueNumStore::s_vnfOpAttribs = nullptr;
-
-void ValueNumStore::InitValueNumStoreStatics()
+void ValueNumStore::ValidateValueNumStoreStatics()
{
- // Make sure we have the constants right...
- assert(unsigned(VNFOA_Arity1) == (1 << VNFOA_ArityShift));
- assert(VNFOA_ArityMask == (VNFOA_MaxArity << VNFOA_ArityShift));
-
- s_vnfOpAttribs = &vnfOpAttribs[0];
+#if DEBUG
+ uint8_t arr[VNF_COUNT] = {};
for (unsigned i = 0; i < GT_COUNT; i++)
{
genTreeOps gtOper = static_cast(i);
@@ -8895,11 +8948,11 @@ void ValueNumStore::InitValueNumStoreStatics()
arity = 3;
}
- vnfOpAttribs[i] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask);
+ arr[i] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask);
if (GenTree::OperIsCommutative(gtOper))
{
- vnfOpAttribs[i] |= VNFOA_Commutative;
+ arr[i] |= VNFOA_Commutative;
}
}
@@ -8907,25 +8960,24 @@ void ValueNumStore::InitValueNumStoreStatics()
int vnfNum = VNF_Boundary + 1; // The macro definition below will update this after using it.
-#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic) \
+#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic, extra) \
if (commute) \
- vnfOpAttribs[vnfNum] |= VNFOA_Commutative; \
+ arr[vnfNum] |= VNFOA_Commutative; \
if (knownNonNull) \
- vnfOpAttribs[vnfNum] |= VNFOA_KnownNonNull; \
+ arr[vnfNum] |= VNFOA_KnownNonNull; \
if (sharedStatic) \
- vnfOpAttribs[vnfNum] |= VNFOA_SharedStatic; \
+ arr[vnfNum] |= VNFOA_SharedStatic; \
if (arity > 0) \
- vnfOpAttribs[vnfNum] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask); \
+ arr[vnfNum] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask); \
vnfNum++;
#include "valuenumfuncs.h"
-#undef ValueNumFuncDef
assert(vnfNum == VNF_COUNT);
#define ValueNumFuncSetArity(vnfNum, arity) \
- vnfOpAttribs[vnfNum] &= ~VNFOA_ArityMask; /* clear old arity value */ \
- vnfOpAttribs[vnfNum] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask) /* set the new arity value */
+ arr[vnfNum] &= ~VNFOA_ArityMask; /* clear old arity value */ \
+ arr[vnfNum] |= ((arity << VNFOA_ArityShift) & VNFOA_ArityMask) /* set the new arity value */
#ifdef FEATURE_HW_INTRINSICS
@@ -8939,7 +8991,7 @@ void ValueNumStore::InitValueNumStoreStatics()
// These HW_Intrinsic's have an extra VNF_SimdType arg.
//
VNFunc func = VNFunc(VNF_HWI_FIRST + (id - NI_HW_INTRINSIC_START - 1));
- unsigned oldArity = VNFuncArity(func);
+ unsigned oldArity = (arr[func] & VNFOA_ArityMask) >> VNFOA_ArityShift;
unsigned newArity = oldArity + 1;
ValueNumFuncSetArity(func, newArity);
@@ -8948,7 +9000,7 @@ void ValueNumStore::InitValueNumStoreStatics()
if (HWIntrinsicInfo::IsCommutative(id))
{
VNFunc func = VNFunc(VNF_HWI_FIRST + (id - NI_HW_INTRINSIC_START - 1));
- vnfOpAttribs[func] |= VNFOA_Commutative;
+ arr[func] |= VNFOA_Commutative;
}
}
@@ -8958,17 +9010,23 @@ void ValueNumStore::InitValueNumStoreStatics()
for (unsigned i = 0; i < ArrLen(genTreeOpsIllegalAsVNFunc); i++)
{
- vnfOpAttribs[genTreeOpsIllegalAsVNFunc[i]] |= VNFOA_IllegalGenTreeOp;
+ arr[genTreeOpsIllegalAsVNFunc[i]] |= VNFOA_IllegalGenTreeOp;
}
+
+ assert(ArrLen(arr) == ArrLen(s_vnfOpAttribs));
+ for (unsigned i = 0; i < ArrLen(arr); i++)
+ {
+ assert(arr[i] == s_vnfOpAttribs[i]);
+ }
+#endif // DEBUG
}
#ifdef DEBUG
// Define the name array.
-#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic) #vnf,
+#define ValueNumFuncDef(vnf, arity, commute, knownNonNull, sharedStatic, extra) #vnf,
const char* ValueNumStore::VNFuncNameArr[] = {
#include "valuenumfuncs.h"
-#undef ValueNumFuncDef
};
/* static */ const char* ValueNumStore::VNFuncName(VNFunc vnf)
@@ -11335,6 +11393,23 @@ void Compiler::fgValueNumberIntrinsic(GenTree* tree)
else
{
assert(intrinsic->gtIntrinsicName == NI_System_Object_GetType);
+
+ // Try to fold obj.GetType() if we know the exact type of obj.
+ bool isExact = false;
+ bool isNonNull = false;
+ CORINFO_CLASS_HANDLE cls = gtGetClassHandle(tree->gtGetOp1(), &isExact, &isNonNull);
+ if ((cls != NO_CLASS_HANDLE) && isExact && isNonNull)
+ {
+ CORINFO_OBJECT_HANDLE typeObj = info.compCompHnd->getRuntimeTypePointer(cls);
+ if (typeObj != nullptr)
+ {
+ setMethodHasFrozenObjects();
+ ValueNum handleVN = vnStore->VNForHandle((ssize_t)typeObj, GTF_ICON_OBJ_HDL);
+ intrinsic->gtVNPair = vnStore->VNPWithExc(ValueNumPair(handleVN, handleVN), arg0VNPx);
+ return;
+ }
+ }
+
intrinsic->gtVNPair =
vnStore->VNPWithExc(vnStore->VNPairForFunc(intrinsic->TypeGet(), VNF_ObjGetType, arg0VNP), arg0VNPx);
}
diff --git a/src/coreclr/jit/valuenum.h b/src/coreclr/jit/valuenum.h
index 94ba28b9ab02b1..e63f4dad58fd4b 100644
--- a/src/coreclr/jit/valuenum.h
+++ b/src/coreclr/jit/valuenum.h
@@ -167,7 +167,7 @@ enum VNFunc
{
// Implicitly, elements of genTreeOps here.
VNF_Boundary = GT_COUNT,
-#define ValueNumFuncDef(nm, arity, commute, knownNonNull, sharedStatic) VNF_##nm,
+#define ValueNumFuncDef(nm, arity, commute, knownNonNull, sharedStatic, extra) VNF_##nm,
#include "valuenumfuncs.h"
VNF_COUNT
};
@@ -275,10 +275,21 @@ class ValueNumStore
VNFOA_SharedStatic = 0x40, // 1 iff this VNF is represent one of the shared static jit helpers
};
- static const unsigned VNFOA_ArityShift = 2;
- static const unsigned VNFOA_ArityBits = 3;
- static const unsigned VNFOA_MaxArity = (1 << VNFOA_ArityBits) - 1; // Max arity we can represent.
- static const unsigned VNFOA_ArityMask = (VNFOA_Arity4 | VNFOA_Arity2 | VNFOA_Arity1);
+ static const unsigned VNFOA_IllegalGenTreeOpShift = 0;
+ static const unsigned VNFOA_CommutativeShift = 1;
+ static const unsigned VNFOA_ArityShift = 2;
+ static const unsigned VNFOA_ArityBits = 3;
+ static const unsigned VNFOA_MaxArity = (1 << VNFOA_ArityBits) - 1; // Max arity we can represent.
+ static const unsigned VNFOA_ArityMask = (VNFOA_Arity4 | VNFOA_Arity2 | VNFOA_Arity1);
+ static const unsigned VNFOA_KnownNonNullShift = 5;
+ static const unsigned VNFOA_SharedStaticShift = 6;
+
+ static_assert(unsigned(VNFOA_IllegalGenTreeOp) == (1 << VNFOA_IllegalGenTreeOpShift));
+ static_assert(unsigned(VNFOA_Commutative) == (1 << VNFOA_CommutativeShift));
+ static_assert(unsigned(VNFOA_Arity1) == (1 << VNFOA_ArityShift));
+ static_assert(VNFOA_ArityMask == (VNFOA_MaxArity << VNFOA_ArityShift));
+ static_assert(unsigned(VNFOA_KnownNonNull) == (1 << VNFOA_KnownNonNullShift));
+ static_assert(unsigned(VNFOA_SharedStatic) == (1 << VNFOA_SharedStaticShift));
// These enum constants are used to encode the cast operation in the lowest bits by VNForCastOper
enum VNFCastAttrib
@@ -289,8 +300,14 @@ class ValueNumStore
VCA_ReservedBits = 0x01, // i.e. (VCA_UnsignedSrc)
};
- // An array of length GT_COUNT, mapping genTreeOp values to their VNFOpAttrib.
- static UINT8* s_vnfOpAttribs;
+ // Helpers and an array of length GT_COUNT, mapping genTreeOp values to their VNFOpAttrib.
+ static constexpr uint8_t GetOpAttribsForArity(genTreeOps oper, GenTreeOperKind kind);
+ static constexpr uint8_t GetOpAttribsForGenTree(genTreeOps oper,
+ bool commute,
+ bool illegalAsVNFunc,
+ GenTreeOperKind kind);
+ static constexpr uint8_t GetOpAttribsForFunc(int arity, bool commute, bool knownNonNull, bool sharedStatic);
+ static const uint8_t s_vnfOpAttribs[];
// Returns "true" iff gtOper is a legal value number function.
// (Requires InitValueNumStoreStatics to have been run.)
@@ -383,8 +400,8 @@ class ValueNumStore
GenTreeFlags GetFoldedArithOpResultHandleFlags(ValueNum vn);
public:
- // Initializes any static variables of ValueNumStore.
- static void InitValueNumStoreStatics();
+ // Validate that the new initializer for s_vnfOpAttribs matches the old code.
+ static void ValidateValueNumStoreStatics();
// Initialize an empty ValueNumStore.
ValueNumStore(Compiler* comp, CompAllocator allocator);
diff --git a/src/coreclr/jit/valuenumfuncs.h b/src/coreclr/jit/valuenumfuncs.h
index 1c6c72bb87fa06..de0c347591a8d2 100644
--- a/src/coreclr/jit/valuenumfuncs.h
+++ b/src/coreclr/jit/valuenumfuncs.h
@@ -3,180 +3,180 @@
// Defines the functions understood by the value-numbering system.
// ValueNumFuncDef(, , , ,
-// )
+// , )
// clang-format off
-ValueNumFuncDef(MemOpaque, 1, false, false, false) // Args: 0: loop num
-ValueNumFuncDef(MapSelect, 2, false, false, false) // Args: 0: map, 1: key.
-ValueNumFuncDef(MapStore, 4, false, false, false) // Args: 0: map, 1: index (e. g. field handle), 2: value being stored, 3: loop num.
-ValueNumFuncDef(MapPhysicalStore, 3, false, false, false) // Args: 0: map, 1: "physical selector": offset and size, 2: value being stored
-ValueNumFuncDef(BitCast, 2, false, false, false) // Args: 0: VN of the arg, 1: VN of the target type
-ValueNumFuncDef(ZeroObj, 1, false, false, false) // Args: 0: VN of the class handle.
-ValueNumFuncDef(PhiDef, 3, false, false, false) // Args: 0: local var # (or -1 for memory), 1: SSA #, 2: VN of definition.
-ValueNumFuncDef(PhiMemoryDef, 2, false, false, false) // Args: 0: VN for basic block pointer, 1: VN of definition
-ValueNumFuncDef(Phi, 2, false, false, false) // A phi function. Only occurs as arg of PhiDef or PhiMemoryDef. Arguments are SSA numbers of var being defined.
-
-ValueNumFuncDef(PtrToLoc, 2, false, true, false) // Pointer (byref) to a local variable. Args: VN's of: 0: local's number, 1: offset.
-ValueNumFuncDef(PtrToArrElem, 4, false, false, false) // Pointer (byref) to an array element. Args: 0: array elem type eq class var_types value, VN's of: 1: array, 2: index, 3: offset.
-ValueNumFuncDef(PtrToStatic, 3, false, true, false) // Pointer (byref) to a static variable (or possibly a field thereof, if the static variable is a struct).
- // Args: 0: (VN of) the box's address if the static is "boxed",
- // 1: (VN of) the field sequence,
- // 2: (VN of) offset for the constituent struct fields
-
-ValueNumFuncDef(MDArrLength, 2, false, false, false) // MD array len, Args: 0: array, 1: dimension
-ValueNumFuncDef(MDArrLowerBound, 2, false, false, false) // MD array lower bound, Args: 0: array, 1: dimension
-
-ValueNumFuncDef(InitVal, 1, false, false, false) // An input arg, or init val of a local Args: 0: a constant VN.
-
-ValueNumFuncDef(Cast, 2, false, false, false) // VNF_Cast: Cast Operation changes the representations size and unsigned-ness.
- // Args: 0: Source for the cast operation.
- // 1: Constant integer representing the operation .
- // Use VNForCastOper() to construct.
-ValueNumFuncDef(CastOvf, 2, false, false, false) // Same as a VNF_Cast but also can throw an overflow exception.
-
-ValueNumFuncDef(CastClass, 2, false, false, false) // Args: 0: Handle of class being cast to, 1: object being cast.
-ValueNumFuncDef(IsInstanceOf, 2, false, false, false) // Args: 0: Handle of class being queried, 1: object being queried.
-ValueNumFuncDef(ReadyToRunCastClass, 2, false, false, false) // Args: 0: Helper stub address, 1: object being cast.
-ValueNumFuncDef(ReadyToRunIsInstanceOf, 2, false, false, false) // Args: 0: Helper stub address, 1: object being queried.
-ValueNumFuncDef(TypeHandleToRuntimeType, 1, false, false, false) // Args: 0: TypeHandle to translate
-ValueNumFuncDef(TypeHandleToRuntimeTypeHandle, 1, false, false, false) // Args: 0: TypeHandle to translate
-
-ValueNumFuncDef(LdElemA, 3, false, false, false) // Args: 0: array value; 1: index value; 2: type handle of element.
-
-ValueNumFuncDef(ByrefExposedLoad, 3, false, false, false) // Args: 0: type handle/id, 1: pointer value; 2: ByrefExposed heap value
-
-ValueNumFuncDef(GetRefanyVal, 2, false, false, false) // Args: 0: type handle; 1: typedref value. Returns the value (asserting that the type is right).
-
-ValueNumFuncDef(GetClassFromMethodParam, 1, false, true, false) // Args: 0: method generic argument.
-ValueNumFuncDef(GetSyncFromClassHandle, 1, false, true, false) // Args: 0: class handle.
-ValueNumFuncDef(LoopCloneChoiceAddr, 0, false, true, false)
+ValueNumFuncDef(MemOpaque, 1, false, false, false, false) // Args: 0: loop num
+ValueNumFuncDef(MapSelect, 2, false, false, false, false) // Args: 0: map, 1: key.
+ValueNumFuncDef(MapStore, 4, false, false, false, false) // Args: 0: map, 1: index (e. g. field handle), 2: value being stored, 3: loop num.
+ValueNumFuncDef(MapPhysicalStore, 3, false, false, false, false) // Args: 0: map, 1: "physical selector": offset and size, 2: value being stored
+ValueNumFuncDef(BitCast, 2, false, false, false, false) // Args: 0: VN of the arg, 1: VN of the target type
+ValueNumFuncDef(ZeroObj, 1, false, false, false, false) // Args: 0: VN of the class handle.
+ValueNumFuncDef(PhiDef, 3, false, false, false, false) // Args: 0: local var # (or -1 for memory), 1: SSA #, 2: VN of definition.
+ValueNumFuncDef(PhiMemoryDef, 2, false, false, false, false) // Args: 0: VN for basic block pointer, 1: VN of definition
+ValueNumFuncDef(Phi, 2, false, false, false, false) // A phi function. Only occurs as arg of PhiDef or PhiMemoryDef. Arguments are SSA numbers of var being defined.
+
+ValueNumFuncDef(PtrToLoc, 2, false, true, false, false) // Pointer (byref) to a local variable. Args: VN's of: 0: local's number, 1: offset.
+ValueNumFuncDef(PtrToArrElem, 4, false, false, false, false) // Pointer (byref) to an array element. Args: 0: array elem type eq class var_types value, VN's of: 1: array, 2: index, 3: offset.
+ValueNumFuncDef(PtrToStatic, 3, false, true, false, false) // Pointer (byref) to a static variable (or possibly a field thereof, if the static variable is a struct).
+ // Args: 0: (VN of) the box's address if the static is "boxed",
+ // 1: (VN of) the field sequence,
+ // 2: (VN of) offset for the constituent struct fields
+
+ValueNumFuncDef(MDArrLength, 2, false, false, false, false) // MD array len, Args: 0: array, 1: dimension
+ValueNumFuncDef(MDArrLowerBound, 2, false, false, false, false) // MD array lower bound, Args: 0: array, 1: dimension
+
+ValueNumFuncDef(InitVal, 1, false, false, false, false) // An input arg, or init val of a local Args: 0: a constant VN.
+
+ValueNumFuncDef(Cast, 2, false, false, false, false) // VNF_Cast: Cast Operation changes the representations size and unsigned-ness.
+ // Args: 0: Source for the cast operation.
+ // 1: Constant integer representing the operation .
+ // Use VNForCastOper() to construct.
+ValueNumFuncDef(CastOvf, 2, false, false, false, false) // Same as a VNF_Cast but also can throw an overflow exception.
+
+ValueNumFuncDef(CastClass, 2, false, false, false, false) // Args: 0: Handle of class being cast to, 1: object being cast.
+ValueNumFuncDef(IsInstanceOf, 2, false, false, false, false) // Args: 0: Handle of class being queried, 1: object being queried.
+ValueNumFuncDef(ReadyToRunCastClass, 2, false, false, false, false) // Args: 0: Helper stub address, 1: object being cast.
+ValueNumFuncDef(ReadyToRunIsInstanceOf, 2, false, false, false, false) // Args: 0: Helper stub address, 1: object being queried.
+ValueNumFuncDef(TypeHandleToRuntimeType, 1, false, false, false, false) // Args: 0: TypeHandle to translate
+ValueNumFuncDef(TypeHandleToRuntimeTypeHandle, 1, false, false, false, false) // Args: 0: TypeHandle to translate
+
+ValueNumFuncDef(LdElemA, 3, false, false, false, false) // Args: 0: array value; 1: index value; 2: type handle of element.
+
+ValueNumFuncDef(ByrefExposedLoad, 3, false, false, false, false) // Args: 0: type handle/id, 1: pointer value; 2: ByrefExposed heap value
+
+ValueNumFuncDef(GetRefanyVal, 2, false, false, false, false) // Args: 0: type handle; 1: typedref value. Returns the value (asserting that the type is right).
+
+ValueNumFuncDef(GetClassFromMethodParam, 1, false, true, false, false) // Args: 0: method generic argument.
+ValueNumFuncDef(GetSyncFromClassHandle, 1, false, true, false, false) // Args: 0: class handle.
+ValueNumFuncDef(LoopCloneChoiceAddr, 0, false, true, false, false)
// How we represent values of expressions with exceptional side effects:
-ValueNumFuncDef(ValWithExc, 2, false, false, false) // Args: 0: value number from normal execution; 1: VN for set of possible exceptions.
-ValueNumFuncDef(ExcSetCons, 2, false, false, false) // Args: 0: exception; 1: exception set (including EmptyExcSet). Invariant: "car"s are always in ascending order.
+ValueNumFuncDef(ValWithExc, 2, false, false, false, false) // Args: 0: value number from normal execution; 1: VN for set of possible exceptions.
+ValueNumFuncDef(ExcSetCons, 2, false, false, false, false) // Args: 0: exception; 1: exception set (including EmptyExcSet). Invariant: "car"s are always in ascending order.
// Various functions that are used to indicate that an exceptions may occur
// Curremtly when the execution is always thrown, the value VNForVoid() is used as Arg0 by OverflowExc and DivideByZeroExc
//
-ValueNumFuncDef(NullPtrExc, 1, false, false, false) // Null pointer exception check. Args: 0: address value, throws when it is null
-ValueNumFuncDef(ArithmeticExc, 2, false, false, false) // Arithmetic exception check, ckfinite and integer division overflow, Args: 0: expression value,
-ValueNumFuncDef(OverflowExc, 1, false, false, false) // Integer overflow check. used for checked add,sub and mul Args: 0: expression value, throws when it overflows
-ValueNumFuncDef(ConvOverflowExc, 2, false, false, false) // Cast conversion overflow check. Args: 0: input value; 1: var_types of the target type
+ValueNumFuncDef(NullPtrExc, 1, false, false, false, false) // Null pointer exception check. Args: 0: address value, throws when it is null
+ValueNumFuncDef(ArithmeticExc, 2, false, false, false, false) // Arithmetic exception check, ckfinite and integer division overflow, Args: 0: expression value,
+ValueNumFuncDef(OverflowExc, 1, false, false, false, false) // Integer overflow check. used for checked add,sub and mul Args: 0: expression value, throws when it overflows
+ValueNumFuncDef(ConvOverflowExc, 2, false, false, false, false) // Cast conversion overflow check. Args: 0: input value; 1: var_types of the target type
// - (shifted left one bit; low bit encode whether source is unsigned.)
-ValueNumFuncDef(DivideByZeroExc, 1, false, false, false) // Division by zero check. Args: 0: divisor value, throws when it is zero
-ValueNumFuncDef(IndexOutOfRangeExc, 2, false, false, false) // Array bounds check, Args: 0: array length; 1: index value, throws when the bounds check fails.
-ValueNumFuncDef(InvalidCastExc, 2, false, false, false) // CastClass check, Args: 0: ref value being cast; 1: handle of type being cast to, throws when the cast fails.
-ValueNumFuncDef(NewArrOverflowExc, 1, false, false, false) // Raises Integer overflow when Arg 0 is negative
-ValueNumFuncDef(HelperMultipleExc, 0, false, false, false) // Represents one or more different exceptions that could be thrown by a Jit Helper method
-
-ValueNumFuncDef(FltRound, 1, false, false, false)
-ValueNumFuncDef(DblRound, 1, false, false, false)
-
-ValueNumFuncDef(Abs, 1, false, false, false)
-ValueNumFuncDef(Acos, 1, false, false, false)
-ValueNumFuncDef(Acosh, 1, false, false, false)
-ValueNumFuncDef(Asin, 1, false, false, false)
-ValueNumFuncDef(Asinh, 1, false, false, false)
-ValueNumFuncDef(Atan, 1, false, false, false)
-ValueNumFuncDef(Atanh, 1, false, false, false)
-ValueNumFuncDef(Atan2, 2, false, false, false)
-ValueNumFuncDef(Cbrt, 1, false, false, false)
-ValueNumFuncDef(Ceiling, 1, false, false, false)
-ValueNumFuncDef(Cos, 1, false, false, false)
-ValueNumFuncDef(Cosh, 1, false, false, false)
-ValueNumFuncDef(Exp, 1, false, false, false)
-ValueNumFuncDef(Floor, 1, false, false, false)
-ValueNumFuncDef(FMod, 2, false, false, false)
-ValueNumFuncDef(ILogB, 1, false, false, false)
-ValueNumFuncDef(Log, 1, false, false, false)
-ValueNumFuncDef(Log2, 1, false, false, false)
-ValueNumFuncDef(Log10, 1, false, false, false)
-ValueNumFuncDef(Max, 2, false, false, false)
-ValueNumFuncDef(Min, 2, false, false, false)
-ValueNumFuncDef(Pow, 2, false, false, false)
-ValueNumFuncDef(RoundDouble, 1, false, false, false)
-ValueNumFuncDef(RoundInt32, 1, false, false, false)
-ValueNumFuncDef(RoundSingle, 1, false, false, false)
-ValueNumFuncDef(Sin, 1, false, false, false)
-ValueNumFuncDef(Sinh, 1, false, false, false)
-ValueNumFuncDef(Sqrt, 1, false, false, false)
-ValueNumFuncDef(Tan, 1, false, false, false)
-ValueNumFuncDef(Tanh, 1, false, false, false)
-ValueNumFuncDef(Truncate, 1, false, false, false)
-
-ValueNumFuncDef(ManagedThreadId, 0, false, false, false)
-
-ValueNumFuncDef(ObjGetType, 1, false, true, false)
-ValueNumFuncDef(GetgenericsGcstaticBase, 1, false, true, true)
-ValueNumFuncDef(GetgenericsNongcstaticBase, 1, false, true, true)
-ValueNumFuncDef(GetsharedGcstaticBase, 2, false, true, true)
-ValueNumFuncDef(GetsharedNongcstaticBase, 2, false, true, true)
-ValueNumFuncDef(GetsharedGcstaticBaseNoctor, 1, false, true, true)
-ValueNumFuncDef(GetsharedNongcstaticBaseNoctor, 1, false, true, true)
-ValueNumFuncDef(ReadyToRunStaticBaseGC, 1, false, true, true)
-ValueNumFuncDef(ReadyToRunStaticBaseNonGC, 1, false, true, true)
-ValueNumFuncDef(ReadyToRunStaticBaseThread, 1, false, true, true)
-ValueNumFuncDef(ReadyToRunStaticBaseThreadNonGC, 1, false, true, true)
-ValueNumFuncDef(ReadyToRunGenericStaticBase, 2, false, true, true)
-ValueNumFuncDef(GetsharedGcstaticBaseDynamicclass, 2, false, true, true)
-ValueNumFuncDef(GetsharedNongcstaticBaseDynamicclass, 2, false, true, true)
-ValueNumFuncDef(GetgenericsGcthreadstaticBase, 1, false, true, true)
-ValueNumFuncDef(GetgenericsNongcthreadstaticBase, 1, false, true, true)
-ValueNumFuncDef(GetsharedGcthreadstaticBase, 2, false, true, true)
-ValueNumFuncDef(GetsharedNongcthreadstaticBase, 2, false, true, true)
-ValueNumFuncDef(GetsharedGcthreadstaticBaseNoctor, 2, false, true, true)
-ValueNumFuncDef(GetsharedGcthreadstaticBaseNoctorOptimized, 1, false, true, true)
-ValueNumFuncDef(GetsharedNongcthreadstaticBaseNoctor, 2, false, true, true)
-ValueNumFuncDef(GetsharedNongcthreadstaticBaseNoctorOptimized, 1, false, true, true)
-ValueNumFuncDef(GetsharedGcthreadstaticBaseDynamicclass, 2, false, true, true)
-ValueNumFuncDef(GetsharedNongcthreadstaticBaseDynamicclass, 2, false, true, true)
-
-ValueNumFuncDef(ClassinitSharedDynamicclass, 2, false, false, false)
-ValueNumFuncDef(RuntimeHandleMethod, 2, false, true, false)
-ValueNumFuncDef(RuntimeHandleClass, 2, false, true, false)
-ValueNumFuncDef(ReadyToRunGenericHandle, 2, false, true, false)
-
-ValueNumFuncDef(GetStaticAddrTLS, 1, false, true, false)
-
-ValueNumFuncDef(JitNew, 2, false, true, false)
-ValueNumFuncDef(JitNewArr, 3, false, true, false)
-ValueNumFuncDef(JitNewMdArr, 4, false, true, false)
-ValueNumFuncDef(JitReadyToRunNew, 2, false, true, false)
-ValueNumFuncDef(JitReadyToRunNewArr, 3, false, true, false)
-ValueNumFuncDef(Box, 3, false, false, false)
-ValueNumFuncDef(BoxNullable, 3, false, false, false)
-
-ValueNumFuncDef(LazyStrCns, 2, false, true, false) // Lazy-initialized string literal (helper)
-ValueNumFuncDef(InvariantLoad, 1, false, false, false) // Args: 0: (VN of) the address.
-ValueNumFuncDef(InvariantNonNullLoad, 1, false, true, false) // Args: 0: (VN of) the address.
-ValueNumFuncDef(Unbox, 2, false, true, false)
-
-ValueNumFuncDef(LT_UN, 2, false, false, false) // unsigned or unordered comparisons
-ValueNumFuncDef(LE_UN, 2, false, false, false)
-ValueNumFuncDef(GE_UN, 2, false, false, false)
-ValueNumFuncDef(GT_UN, 2, false, false, false)
-
-ValueNumFuncDef(ADD_OVF, 2, true, false, false) // overflow checking operations
-ValueNumFuncDef(SUB_OVF, 2, false, false, false)
-ValueNumFuncDef(MUL_OVF, 2, true, false, false)
-
-ValueNumFuncDef(ADD_UN_OVF, 2, true, false, false) // unsigned overflow checking operations
-ValueNumFuncDef(SUB_UN_OVF, 2, false, false, false)
-ValueNumFuncDef(MUL_UN_OVF, 2, true, false, false)
+ValueNumFuncDef(DivideByZeroExc, 1, false, false, false, false) // Division by zero check. Args: 0: divisor value, throws when it is zero
+ValueNumFuncDef(IndexOutOfRangeExc, 2, false, false, false, false) // Array bounds check, Args: 0: array length; 1: index value, throws when the bounds check fails.
+ValueNumFuncDef(InvalidCastExc, 2, false, false, false, false) // CastClass check, Args: 0: ref value being cast; 1: handle of type being cast to, throws when the cast fails.
+ValueNumFuncDef(NewArrOverflowExc, 1, false, false, false, false) // Raises Integer overflow when Arg 0 is negative
+ValueNumFuncDef(HelperMultipleExc, 0, false, false, false, false) // Represents one or more different exceptions that could be thrown by a Jit Helper method
+
+ValueNumFuncDef(FltRound, 1, false, false, false, false)
+ValueNumFuncDef(DblRound, 1, false, false, false, false)
+
+ValueNumFuncDef(Abs, 1, false, false, false, false)
+ValueNumFuncDef(Acos, 1, false, false, false, false)
+ValueNumFuncDef(Acosh, 1, false, false, false, false)
+ValueNumFuncDef(Asin, 1, false, false, false, false)
+ValueNumFuncDef(Asinh, 1, false, false, false, false)
+ValueNumFuncDef(Atan, 1, false, false, false, false)
+ValueNumFuncDef(Atanh, 1, false, false, false, false)
+ValueNumFuncDef(Atan2, 2, false, false, false, false)
+ValueNumFuncDef(Cbrt, 1, false, false, false, false)
+ValueNumFuncDef(Ceiling, 1, false, false, false, false)
+ValueNumFuncDef(Cos, 1, false, false, false, false)
+ValueNumFuncDef(Cosh, 1, false, false, false, false)
+ValueNumFuncDef(Exp, 1, false, false, false, false)
+ValueNumFuncDef(Floor, 1, false, false, false, false)
+ValueNumFuncDef(FMod, 2, false, false, false, false)
+ValueNumFuncDef(ILogB, 1, false, false, false, false)
+ValueNumFuncDef(Log, 1, false, false, false, false)
+ValueNumFuncDef(Log2, 1, false, false, false, false)
+ValueNumFuncDef(Log10, 1, false, false, false, false)
+ValueNumFuncDef(Max, 2, false, false, false, false)
+ValueNumFuncDef(Min, 2, false, false, false, false)
+ValueNumFuncDef(Pow, 2, false, false, false, false)
+ValueNumFuncDef(RoundDouble, 1, false, false, false, false)
+ValueNumFuncDef(RoundInt32, 1, false, false, false, false)
+ValueNumFuncDef(RoundSingle, 1, false, false, false, false)
+ValueNumFuncDef(Sin, 1, false, false, false, false)
+ValueNumFuncDef(Sinh, 1, false, false, false, false)
+ValueNumFuncDef(Sqrt, 1, false, false, false, false)
+ValueNumFuncDef(Tan, 1, false, false, false, false)
+ValueNumFuncDef(Tanh, 1, false, false, false, false)
+ValueNumFuncDef(Truncate, 1, false, false, false, false)
+
+ValueNumFuncDef(ManagedThreadId, 0, false, false, false, false)
+
+ValueNumFuncDef(ObjGetType, 1, false, true, false, false)
+ValueNumFuncDef(GetgenericsGcstaticBase, 1, false, true, true, false)
+ValueNumFuncDef(GetgenericsNongcstaticBase, 1, false, true, true, false)
+ValueNumFuncDef(GetsharedGcstaticBase, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcstaticBase, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedGcstaticBaseNoctor, 1, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcstaticBaseNoctor, 1, false, true, true, false)
+ValueNumFuncDef(ReadyToRunStaticBaseGC, 1, false, true, true, false)
+ValueNumFuncDef(ReadyToRunStaticBaseNonGC, 1, false, true, true, false)
+ValueNumFuncDef(ReadyToRunStaticBaseThread, 1, false, true, true, false)
+ValueNumFuncDef(ReadyToRunStaticBaseThreadNonGC, 1, false, true, true, false)
+ValueNumFuncDef(ReadyToRunGenericStaticBase, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedGcstaticBaseDynamicclass, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcstaticBaseDynamicclass, 2, false, true, true, false)
+ValueNumFuncDef(GetgenericsGcthreadstaticBase, 1, false, true, true, false)
+ValueNumFuncDef(GetgenericsNongcthreadstaticBase, 1, false, true, true, false)
+ValueNumFuncDef(GetsharedGcthreadstaticBase, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcthreadstaticBase, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedGcthreadstaticBaseNoctor, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedGcthreadstaticBaseNoctorOptimized, 1, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcthreadstaticBaseNoctor, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcthreadstaticBaseNoctorOptimized, 1, false, true, true, false)
+ValueNumFuncDef(GetsharedGcthreadstaticBaseDynamicclass, 2, false, true, true, false)
+ValueNumFuncDef(GetsharedNongcthreadstaticBaseDynamicclass, 2, false, true, true, false)
+
+ValueNumFuncDef(ClassinitSharedDynamicclass, 2, false, false, false, false)
+ValueNumFuncDef(RuntimeHandleMethod, 2, false, true, false, false)
+ValueNumFuncDef(RuntimeHandleClass, 2, false, true, false, false)
+ValueNumFuncDef(ReadyToRunGenericHandle, 2, false, true, false, false)
+
+ValueNumFuncDef(GetStaticAddrTLS, 1, false, true, false, false)
+
+ValueNumFuncDef(JitNew, 2, false, true, false, false)
+ValueNumFuncDef(JitNewArr, 3, false, true, false, false)
+ValueNumFuncDef(JitNewMdArr, 4, false, true, false, false)
+ValueNumFuncDef(JitReadyToRunNew, 2, false, true, false, false)
+ValueNumFuncDef(JitReadyToRunNewArr, 3, false, true, false, false)
+ValueNumFuncDef(Box, 3, false, false, false, false)
+ValueNumFuncDef(BoxNullable, 3, false, false, false, false)
+
+ValueNumFuncDef(LazyStrCns, 2, false, true, false, false) // Lazy-initialized string literal (helper)
+ValueNumFuncDef(InvariantLoad, 1, false, false, false, false) // Args: 0: (VN of) the address.
+ValueNumFuncDef(InvariantNonNullLoad, 1, false, true, false, false) // Args: 0: (VN of) the address.
+ValueNumFuncDef(Unbox, 2, false, true, false, false)
+
+ValueNumFuncDef(LT_UN, 2, false, false, false, false) // unsigned or unordered comparisons
+ValueNumFuncDef(LE_UN, 2, false, false, false, false)
+ValueNumFuncDef(GE_UN, 2, false, false, false, false)
+ValueNumFuncDef(GT_UN, 2, false, false, false, false)
+
+ValueNumFuncDef(ADD_OVF, 2, true, false, false, false) // overflow checking operations
+ValueNumFuncDef(SUB_OVF, 2, false, false, false, false)
+ValueNumFuncDef(MUL_OVF, 2, true, false, false, false)
+
+ValueNumFuncDef(ADD_UN_OVF, 2, true, false, false, false) // unsigned overflow checking operations
+ValueNumFuncDef(SUB_UN_OVF, 2, false, false, false, false)
+ValueNumFuncDef(MUL_UN_OVF, 2, true, false, false, false)
#ifdef FEATURE_SIMD
-ValueNumFuncDef(SimdType, 2, false, false, false) // A value number function to compose a SIMD type
+ValueNumFuncDef(SimdType, 2, false, false, false, false) // A value number function to compose a SIMD type
#endif
#if defined(TARGET_XARCH)
-#define HARDWARE_INTRINSIC(isa, name, size, argCount, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
-ValueNumFuncDef(HWI_##isa##_##name, argCount, false, false, false) // All of the HARDWARE_INTRINSICS for x86/x64
+#define HARDWARE_INTRINSIC(isa, name, size, argCount, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+ValueNumFuncDef(HWI_##isa##_##name, argCount, ((flag) & HW_Flag_Commutative) >> 0, false, false, extra) // All of the HARDWARE_INTRINSICS for x86/x64
#include "hwintrinsiclistxarch.h"
#define VNF_HWI_FIRST VNF_HWI_Vector128_Abs
#elif defined (TARGET_ARM64)
-#define HARDWARE_INTRINSIC(isa, name, size, argCount, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
-ValueNumFuncDef(HWI_##isa##_##name, argCount, false, false, false) // All of the HARDWARE_INTRINSICS for arm64
+#define HARDWARE_INTRINSIC(isa, name, size, argCount, extra, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, category, flag) \
+ValueNumFuncDef(HWI_##isa##_##name, argCount, ((flag) & HW_Flag_Commutative) >> 0, false, false, extra) // All of the HARDWARE_INTRINSICS for arm64
#include "hwintrinsiclistarm64.h"
#define VNF_HWI_FIRST VNF_HWI_Vector64_Abs
diff --git a/src/coreclr/md/ceefilegen/cceegen.cpp b/src/coreclr/md/ceefilegen/cceegen.cpp
index 050d16288ef93b..391411b3984aec 100644
--- a/src/coreclr/md/ceefilegen/cceegen.cpp
+++ b/src/coreclr/md/ceefilegen/cceegen.cpp
@@ -178,20 +178,6 @@ STDMETHODIMP CCeeGen::ComputePointer(HCEESECTION section, ULONG RVA, UCHAR **lpB
return hr;
}
-STDMETHODIMP CCeeGen::GetIMapTokenIface (
- IUnknown **pIMapToken)
-{
- _ASSERTE(!"E_NOTIMPL");
- return E_NOTIMPL;
-}
-
-STDMETHODIMP CCeeGen::AddNotificationHandler (
- IUnknown *pHandler)
-{
- _ASSERTE(!"E_NOTIMPL");
- return E_NOTIMPL;
-}
-
STDMETHODIMP CCeeGen::GenerateCeeFile ()
{
_ASSERTE(!"E_NOTIMPL");
@@ -260,8 +246,6 @@ CCeeGen::CCeeGen() // protected ctor
m_cRefs = 0;
m_peSectionMan = NULL;
m_pTokenMap = NULL;
- m_pRemapHandler = NULL;
-
}
// Shared init code between derived classes, called by virtual Init()
@@ -288,7 +272,6 @@ HRESULT CCeeGen::Init() // not-virtual, protected
m_pTokenMap = NULL;
m_fTokenMapSupported = FALSE;
- m_pRemapHandler = NULL;
// These text section needs special support for handling string management now that we have
// merged the sections together, so create it with an underlying CeeSectionString rather than the
@@ -321,17 +304,6 @@ HRESULT CCeeGen::Init() // not-virtual, protected
return hr;
}
-HRESULT CCeeGen::cloneInstance(CCeeGen *destination) { //public, virtual
- _ASSERTE(destination);
-
- destination->m_pTokenMap = m_pTokenMap;
- destination->m_fTokenMapSupported = m_fTokenMapSupported;
- destination->m_pRemapHandler = m_pRemapHandler;
-
- //Create a deep copy of the section manager (and each of it's sections);
- return m_peSectionMan->cloneInstance(destination->m_peSectionMan);
-}
-
HRESULT CCeeGen::Cleanup() // virtual
{
HRESULT hr;
@@ -357,12 +329,6 @@ HRESULT CCeeGen::Cleanup() // virtual
m_pTokenMap = NULL;
}
- if (m_pRemapHandler)
- {
- m_pRemapHandler->Release();
- m_pRemapHandler = NULL;
- }
-
if (m_peSectionMan) {
m_peSectionMan->Cleanup();
delete m_peSectionMan;
@@ -546,48 +512,7 @@ HRESULT CCeeGen::getMapTokenIface(IUnknown **pIMapToken, IMetaDataEmit *emitter)
}
m_pTokenMap = pMapper;
m_fTokenMapSupported = (emitter == 0);
-
- // If we've been holding onto a token remap handler waiting
- // for the token mapper to get created, add it to the token
- // mapper now and release our hold on it.
- if (m_pRemapHandler && m_pTokenMap)
- {
- m_pTokenMap->AddTokenMapper(m_pRemapHandler);
- m_pRemapHandler->Release();
- m_pRemapHandler = NULL;
- }
}
*pIMapToken = getTokenMapper()->GetMapTokenIface();
return S_OK;
}
-
-HRESULT CCeeGen::addNotificationHandler(IUnknown *pHandler)
-{
- // Null is no good...
- if (!pHandler)
- return E_POINTER;
-
- HRESULT hr = S_OK;
- IMapToken *pIMapToken = NULL;
-
- // Is this an IMapToken? If so, we can put it to good use...
- if (SUCCEEDED(pHandler->QueryInterface(IID_IMapToken,
- (void**)&pIMapToken)))
- {
- // You gotta have a token mapper to use an IMapToken, though.
- if (m_pTokenMap)
- {
- hr = m_pTokenMap->AddTokenMapper(pIMapToken);
- pIMapToken->Release();
- }
- else
- {
- // Hold onto it for later, just in case a token mapper
- // gets created. We're holding a reference to it here,
- // too.
- m_pRemapHandler = pIMapToken;
- }
- }
-
- return hr;
-}
diff --git a/src/coreclr/md/ceefilegen/ceegentokenmapper.cpp b/src/coreclr/md/ceefilegen/ceegentokenmapper.cpp
index bf4118948c5084..fe9bc3f7c08b0b 100644
--- a/src/coreclr/md/ceefilegen/ceegentokenmapper.cpp
+++ b/src/coreclr/md/ceefilegen/ceegentokenmapper.cpp
@@ -58,11 +58,6 @@ HRESULT STDMETHODCALLTYPE CeeGenTokenMapper::Map(
_ASSERTE(IndexForType(tkFrom) < GetMaxMapSize());
_ASSERTE(IndexForType(tkTo) != -1 && IndexForType(tkTo) < GetMaxMapSize());
- // If there is another token mapper that the user wants called, go
- // ahead and call it now.
- if (m_pIMapToken)
- m_pIMapToken->Map(tkFrom, tkTo);
-
ridFrom = RidFromToken(tkFrom);
pMap = &m_rgMap[IndexForType(tkFrom)];
diff --git a/src/coreclr/md/ceefilegen/pesectionman.cpp b/src/coreclr/md/ceefilegen/pesectionman.cpp
index 42635acc6c7154..cee08e25a77313 100644
--- a/src/coreclr/md/ceefilegen/pesectionman.cpp
+++ b/src/coreclr/md/ceefilegen/pesectionman.cpp
@@ -148,44 +148,6 @@ HRESULT PESectionMan::newSection(const char* name, PESection **section,
return S_OK;
}
-//Clone each of our sections. This will cause a deep copy of the sections
-HRESULT PESectionMan::cloneInstance(PESectionMan *destination) {
- _ASSERTE(destination);
- PESection *pSection;
- PESection **destPtr;
- HRESULT hr = NOERROR;
-
- //Copy each of the sections
- for (PESection** ptr = sectStart; ptr < sectCur; ptr++) {
- destPtr = destination->sectStart;
- pSection = NULL;
-
- // try to find the matching section by name
- for (; destPtr < destination->sectCur; destPtr++)
- {
- if (strcmp((*destPtr)->m_name, (*ptr)->m_name) == 0)
- {
- pSection = *destPtr;
- break;
- }
- }
- if (destPtr >= destination->sectCur)
- {
- // cannot find a section in the destination with matching name
- // so create one!
- IfFailRet( destination->getSectionCreate((*ptr)->m_name,
- (*ptr)->flags(),
- &pSection) );
- }
- if (pSection)
- IfFailRet( (*ptr)->cloneInstance(pSection) );
- }
-
- //destination->sectEnd=destination->sectStart + (sectEnd-sectStart);
- return S_OK;
-}
-
-
//*****************************************************************************
// Implementation for PESection
//*****************************************************************************
@@ -380,47 +342,6 @@ HRESULT PESection::applyRelocs(CeeGenTokenMapper *pTokenMapper)
return S_OK;
}
-HRESULT PESection::cloneInstance(PESection *destination) {
- PESectionReloc *cur;
- INT32 newSize;
- HRESULT hr = NOERROR;
-
- _ASSERTE(destination);
-
- destination->dirEntry = dirEntry;
-
- //Merge the information currently in the BlobFetcher into
- //out current blob fetcher
- m_blobFetcher.Merge(&(destination->m_blobFetcher));
-
- //Copy the name.
- strncpy_s(destination->m_name, sizeof(destination->m_name), m_name, sizeof(m_name) - 1);
-
- //Clone the relocs
- //If the arrays aren't the same size, reallocate as necessary.
- //@FUTURE: Make this a ref-counted structure and don't copy it.
-
- newSize = (INT32)(m_relocCur-m_relocStart);
-
- if (newSize>(destination->m_relocEnd - destination->m_relocStart)) {
- delete[] destination->m_relocStart;
-
- destination->m_relocStart = new (nothrow) PESectionReloc[newSize];
- if (destination->m_relocStart == NULL)
- IfFailGo( E_OUTOFMEMORY );
- destination->m_relocEnd = destination->m_relocStart+(newSize);
- }
-
- //copy the correct data over into our new array.
- memcpy(destination->m_relocStart, m_relocStart, sizeof(PESectionReloc)*(newSize));
- destination->m_relocCur = destination->m_relocStart + (newSize);
- for (cur=destination->m_relocStart; curm_relocCur; cur++) {
- cur->section=destination;
- }
-ErrExit:
- return hr;
-}
-
void PESection::SetInitialGrowth(unsigned growth)
{
m_blobFetcher.SetInitialGrowth(growth);
diff --git a/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets b/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
index ad2c022f4be7fc..2412bd4b7947a4 100644
--- a/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
+++ b/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
@@ -48,7 +48,7 @@ The .NET Foundation licenses this file to you under the MIT license.
@executable_pathlibeventpipe-disabled
- libeventpipe-enabled
+ libeventpipe-enabled
@@ -96,7 +96,7 @@ The .NET Foundation licenses this file to you under the MIT license.
-
+
@@ -119,12 +119,12 @@ The .NET Foundation licenses this file to you under the MIT license.
-
-
+
+
-
+
diff --git a/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets b/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets
index 9ce8ef00756dd6..e11fac5806808d 100644
--- a/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets
+++ b/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets
@@ -214,12 +214,11 @@ The .NET Foundation licenses this file to you under the MIT license.
-
-
+
@@ -230,6 +229,7 @@ The .NET Foundation licenses this file to you under the MIT license.
+
diff --git a/src/coreclr/nativeaot/Directory.Build.props b/src/coreclr/nativeaot/Directory.Build.props
index 76c5cb9eecbb72..dce8e37c32fe38 100644
--- a/src/coreclr/nativeaot/Directory.Build.props
+++ b/src/coreclr/nativeaot/Directory.Build.props
@@ -65,8 +65,7 @@
FEATURE_OBJCMARSHAL;$(DefineConstants)
- false
- true
+ trueFEATURE_PERFTRACING;$(DefineConstants)
diff --git a/src/coreclr/nativeaot/Runtime.Base/src/System/RuntimeTypeHandle.cs b/src/coreclr/nativeaot/Runtime.Base/src/System/RuntimeTypeHandle.cs
index 85f880035e904b..8ecb99bc49e584 100644
--- a/src/coreclr/nativeaot/Runtime.Base/src/System/RuntimeTypeHandle.cs
+++ b/src/coreclr/nativeaot/Runtime.Base/src/System/RuntimeTypeHandle.cs
@@ -18,6 +18,6 @@ public class Type
[StructLayout(LayoutKind.Sequential)]
public struct RuntimeTypeHandle
{
- private EETypePtr _pEEType;
+ private IntPtr _value;
}
}
diff --git a/src/coreclr/nativeaot/Runtime/AsmOffsets.h b/src/coreclr/nativeaot/Runtime/AsmOffsets.h
index 203fee38bf15ff..30540742974b45 100644
--- a/src/coreclr/nativeaot/Runtime/AsmOffsets.h
+++ b/src/coreclr/nativeaot/Runtime/AsmOffsets.h
@@ -72,16 +72,6 @@ ASM_OFFSET( 10, 20, InterfaceDispatchCache, m_rgEntries)
ASM_SIZEOF( 8, 10, InterfaceDispatchCacheEntry)
#endif
-#ifdef FEATURE_DYNAMIC_CODE
-ASM_OFFSET( 0, 0, CallDescrData, pSrc)
-ASM_OFFSET( 4, 8, CallDescrData, numStackSlots)
-ASM_OFFSET( 8, C, CallDescrData, fpReturnSize)
-ASM_OFFSET( C, 10, CallDescrData, pArgumentRegisters)
-ASM_OFFSET( 10, 18, CallDescrData, pFloatArgumentRegisters)
-ASM_OFFSET( 14, 20, CallDescrData, pTarget)
-ASM_OFFSET( 18, 28, CallDescrData, pReturnBuffer)
-#endif
-
// Undefine macros that are only used in this header for convenience.
#undef ASM_OFFSET
#undef ASM_SIZEOF
diff --git a/src/coreclr/nativeaot/Runtime/AsmOffsetsVerify.cpp b/src/coreclr/nativeaot/Runtime/AsmOffsetsVerify.cpp
index 489d10aa9b9b87..705e198aaaca32 100644
--- a/src/coreclr/nativeaot/Runtime/AsmOffsetsVerify.cpp
+++ b/src/coreclr/nativeaot/Runtime/AsmOffsetsVerify.cpp
@@ -16,7 +16,6 @@
#include "RuntimeInstance.h"
#include "CachedInterfaceDispatch.h"
#include "shash.h"
-#include "CallDescr.h"
class AsmOffsets
{
diff --git a/src/coreclr/nativeaot/Runtime/CMakeLists.txt b/src/coreclr/nativeaot/Runtime/CMakeLists.txt
index e836c3f2d391f1..864ad7972b48a8 100644
--- a/src/coreclr/nativeaot/Runtime/CMakeLists.txt
+++ b/src/coreclr/nativeaot/Runtime/CMakeLists.txt
@@ -196,7 +196,6 @@ endif (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
list(APPEND RUNTIME_SOURCES_ARCH_ASM
${ARCH_SOURCES_DIR}/AllocFast.${ASM_SUFFIX}
- ${ARCH_SOURCES_DIR}/CallDescrWorker.${ASM_SUFFIX}
${ARCH_SOURCES_DIR}/ExceptionHandling.${ASM_SUFFIX}
${ARCH_SOURCES_DIR}/GcProbe.${ASM_SUFFIX}
${ARCH_SOURCES_DIR}/Interlocked.${ASM_SUFFIX}
diff --git a/src/coreclr/nativeaot/Runtime/CallDescr.h b/src/coreclr/nativeaot/Runtime/CallDescr.h
deleted file mode 100644
index 946b96d2c8e7c9..00000000000000
--- a/src/coreclr/nativeaot/Runtime/CallDescr.h
+++ /dev/null
@@ -1,13 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-struct CallDescrData
-{
- uint8_t* pSrc;
- int numStackSlots;
- int fpReturnSize;
- uint8_t* pArgumentRegisters;
- uint8_t* pFloatArgumentRegisters;
- void* pTarget;
- void* pReturnBuffer;
-};
diff --git a/src/coreclr/nativeaot/Runtime/ICodeManager.h b/src/coreclr/nativeaot/Runtime/ICodeManager.h
index 4205c62c4de080..b21aa3fde6be71 100644
--- a/src/coreclr/nativeaot/Runtime/ICodeManager.h
+++ b/src/coreclr/nativeaot/Runtime/ICodeManager.h
@@ -172,6 +172,8 @@ enum UnwindStackFrameFlags
// If this is a reverse P/Invoke frame, do not continue the unwind
// after extracting the saved transition frame.
USFF_StopUnwindOnTransitionFrame = 1,
+ // Registers not containing GC roots can be omitted.
+ USFF_GcUnwind = 2,
};
class ICodeManager
diff --git a/src/coreclr/nativeaot/Runtime/IntrinsicConstants.h b/src/coreclr/nativeaot/Runtime/IntrinsicConstants.h
index 67ba547488e960..41ec8dec9c3d02 100644
--- a/src/coreclr/nativeaot/Runtime/IntrinsicConstants.h
+++ b/src/coreclr/nativeaot/Runtime/IntrinsicConstants.h
@@ -34,6 +34,10 @@ enum XArchIntrinsicConstants
XArchIntrinsicConstants_Avx512dq_vl = 0x400000,
XArchIntrinsicConstants_Avx512Vbmi = 0x800000,
XArchIntrinsicConstants_Avx512Vbmi_vl = 0x1000000,
+ XArchIntrinsicConstants_Serialize = 0x2000000,
+ XArchIntrinsicConstants_VectorT128 = 0x4000000,
+ XArchIntrinsicConstants_VectorT256 = 0x8000000,
+ XArchIntrinsicConstants_VectorT512 = 0x10000000,
};
#endif //HOST_X86 || HOST_AMD64
@@ -49,6 +53,7 @@ enum ARM64IntrinsicConstants
ARM64IntrinsicConstants_Sha256 = 0x0040,
ARM64IntrinsicConstants_Atomics = 0x0080,
ARM64IntrinsicConstants_Rcpc = 0x0100,
+ ARM64IntrinsicConstants_VectorT128 = 0x0200,
};
// Bit position for the ARM64IntrinsicConstants_Atomics flags, to be used with tbz / tbnz instructions
diff --git a/src/coreclr/nativeaot/Runtime/MiscHelpers.cpp b/src/coreclr/nativeaot/Runtime/MiscHelpers.cpp
index e073885514555f..da9a5c4b97618f 100644
--- a/src/coreclr/nativeaot/Runtime/MiscHelpers.cpp
+++ b/src/coreclr/nativeaot/Runtime/MiscHelpers.cpp
@@ -38,6 +38,7 @@
#include "GCMemoryHelpers.h"
#include "GCMemoryHelpers.inl"
#include "yieldprocessornormalized.h"
+#include "RhConfig.h"
COOP_PINVOKE_HELPER(void, RhDebugBreak, ())
{
@@ -120,16 +121,6 @@ COOP_PINVOKE_HELPER(HANDLE, RhGetOSModuleFromPointer, (PTR_VOID pPointerVal))
return NULL;
}
-COOP_PINVOKE_HELPER(HANDLE, RhGetOSModuleFromEEType, (MethodTable * pEEType))
-{
- return pEEType->GetTypeManagerPtr()->AsTypeManager()->GetOsModuleHandle();
-}
-
-COOP_PINVOKE_HELPER(TypeManagerHandle, RhGetModuleFromEEType, (MethodTable * pEEType))
-{
- return *pEEType->GetTypeManagerPtr();
-}
-
COOP_PINVOKE_HELPER(FC_BOOL_RET, RhFindBlob, (TypeManagerHandle *pTypeManagerHandle, uint32_t blobId, uint8_t ** ppbBlob, uint32_t * pcbBlob))
{
TypeManagerHandle typeManagerHandle = *pTypeManagerHandle;
@@ -349,18 +340,6 @@ COOP_PINVOKE_HELPER(uint8_t *, RhGetCodeTarget, (uint8_t * pCodeOrg))
return pCodeOrg;
}
-// Get the universal transition thunk. If the universal transition stub is called through
-// the normal PE static linkage model, a jump stub would be used which may interfere with
-// the custom calling convention of the universal transition thunk. So instead, a special
-// api just for getting the thunk address is needed.
-// TODO: On ARM this may still result in a jump stub that trashes R12. Determine if anything
-// needs to be done about that when we implement the stub for ARM.
-extern "C" void RhpUniversalTransition();
-COOP_PINVOKE_HELPER(void*, RhGetUniversalTransitionThunk, ())
-{
- return (void*)RhpUniversalTransition;
-}
-
extern CrstStatic g_ThunkPoolLock;
EXTERN_C NATIVEAOT_API void __cdecl RhpAcquireThunkPoolLock()
@@ -422,6 +401,13 @@ COOP_PINVOKE_HELPER(int32_t, RhGetProcessCpuCount, ())
return PalGetProcessCpuCount();
}
+COOP_PINVOKE_HELPER(uint32_t, RhGetKnobValues, (char *** pResultKeys, char *** pResultValues))
+{
+ *pResultKeys = g_pRhConfig->GetKnobNames();
+ *pResultValues = g_pRhConfig->GetKnobValues();
+ return g_pRhConfig->GetKnobCount();
+}
+
#if defined(TARGET_X86) || defined(TARGET_AMD64)
EXTERN_C NATIVEAOT_API void __cdecl RhCpuIdEx(int* cpuInfo, int functionId, int subFunctionId)
{
diff --git a/src/coreclr/nativeaot/Runtime/PalRedhawk.h b/src/coreclr/nativeaot/Runtime/PalRedhawk.h
index a718812f23ddce..c30d7d05ec3dbb 100644
--- a/src/coreclr/nativeaot/Runtime/PalRedhawk.h
+++ b/src/coreclr/nativeaot/Runtime/PalRedhawk.h
@@ -752,8 +752,6 @@ REDHAWK_PALIMPORT uint32_t REDHAWK_PALAPI PalEventUnregister(REGHANDLE arg1);
REDHAWK_PALIMPORT uint32_t REDHAWK_PALAPI PalEventWrite(REGHANDLE arg1, const EVENT_DESCRIPTOR * arg2, uint32_t arg3, EVENT_DATA_DESCRIPTOR * arg4);
#endif
-REDHAWK_PALIMPORT _Ret_maybenull_ void* REDHAWK_PALAPI PalSetWerDataBuffer(_In_ void* pNewBuffer);
-
REDHAWK_PALIMPORT UInt32_BOOL REDHAWK_PALAPI PalAllocateThunksFromTemplate(_In_ HANDLE hTemplateModule, uint32_t templateRva, size_t templateSize, _Outptr_result_bytebuffer_(templateSize) void** newThunksOut);
REDHAWK_PALIMPORT UInt32_BOOL REDHAWK_PALAPI PalFreeThunksFromTemplate(_In_ void *pBaseAddress);
diff --git a/src/coreclr/nativeaot/Runtime/RhConfig.cpp b/src/coreclr/nativeaot/Runtime/RhConfig.cpp
index 0b965c18994f86..506cf750672bfa 100644
--- a/src/coreclr/nativeaot/Runtime/RhConfig.cpp
+++ b/src/coreclr/nativeaot/Runtime/RhConfig.cpp
@@ -120,14 +120,8 @@ bool RhConfig::Environment::TryGetStringValue(const char* name, char** value)
return true;
}
-struct CompilerEmbeddedSettingsBlob
-{
- uint32_t Size;
- char Data[1];
-};
-
-extern "C" CompilerEmbeddedSettingsBlob g_compilerEmbeddedSettingsBlob;
-extern "C" CompilerEmbeddedSettingsBlob g_compilerEmbeddedKnobsBlob;
+extern "C" RhConfig::Config g_compilerEmbeddedSettingsBlob;
+extern "C" RhConfig::Config g_compilerEmbeddedKnobsBlob;
bool RhConfig::ReadConfigValue(_In_z_ const char *name, uint64_t* pValue, bool decimal)
{
@@ -136,7 +130,7 @@ bool RhConfig::ReadConfigValue(_In_z_ const char *name, uint64_t* pValue, bool d
// Check the embedded configuration
const char *embeddedValue = nullptr;
- if (GetEmbeddedVariable(&g_embeddedSettings, &g_compilerEmbeddedSettingsBlob, name, true, &embeddedValue))
+ if (GetEmbeddedVariable(&g_compilerEmbeddedSettingsBlob, name, true, &embeddedValue))
{
*pValue = strtoull(embeddedValue, NULL, decimal ? 10 : 16);
return true;
@@ -148,7 +142,7 @@ bool RhConfig::ReadConfigValue(_In_z_ const char *name, uint64_t* pValue, bool d
bool RhConfig::ReadKnobUInt64Value(_In_z_ const char *name, uint64_t* pValue)
{
const char *embeddedValue = nullptr;
- if (GetEmbeddedVariable(&g_embeddedKnobs, &g_compilerEmbeddedKnobsBlob, name, false, &embeddedValue))
+ if (GetEmbeddedVariable(&g_compilerEmbeddedKnobsBlob, name, false, &embeddedValue))
{
*pValue = strtoull(embeddedValue, NULL, 10);
return true;
@@ -160,7 +154,7 @@ bool RhConfig::ReadKnobUInt64Value(_In_z_ const char *name, uint64_t* pValue)
bool RhConfig::ReadKnobBooleanValue(_In_z_ const char *name, bool* pValue)
{
const char *embeddedValue = nullptr;
- if (GetEmbeddedVariable(&g_embeddedKnobs, &g_compilerEmbeddedKnobsBlob, name, false, &embeddedValue))
+ if (GetEmbeddedVariable(&g_compilerEmbeddedKnobsBlob, name, false, &embeddedValue))
{
*pValue = strcmp(embeddedValue, "true") == 0;
return true;
@@ -169,29 +163,30 @@ bool RhConfig::ReadKnobBooleanValue(_In_z_ const char *name, bool* pValue)
return false;
}
-bool RhConfig::GetEmbeddedVariable(void *volatile * embeddedSettings, void* compilerEmbeddedSettingsBlob, _In_z_ const char* configName, bool caseSensitive, _Out_ const char** configValue)
+char** RhConfig::GetKnobNames()
{
- // Read the config if we haven't yet
- if (*embeddedSettings == NULL)
- {
- ReadEmbeddedSettings(embeddedSettings, compilerEmbeddedSettingsBlob);
- }
+ return g_compilerEmbeddedKnobsBlob.GetKeys();
+}
- // Config wasn't read or reading failed
- if (*embeddedSettings == CONFIG_INI_NOT_AVAIL)
- {
- return false;
- }
+char** RhConfig::GetKnobValues()
+{
+ return g_compilerEmbeddedKnobsBlob.GetValues();
+}
- const ConfigPair* configPairs = (const ConfigPair*)*embeddedSettings;
+uint32_t RhConfig::GetKnobCount()
+{
+ return g_compilerEmbeddedKnobsBlob.GetCount();
+}
+bool RhConfig::GetEmbeddedVariable(Config* config, _In_z_ const char* configName, bool caseSensitive, _Out_ const char** configValue)
+{
// Find the first name which matches
- for (uint32_t iSettings = 0; iSettings < ((CompilerEmbeddedSettingsBlob*)compilerEmbeddedSettingsBlob)->Size; iSettings++)
+ for (uint32_t iSettings = 0; iSettings < config->GetCount(); iSettings++)
{
- if ((caseSensitive && strcmp(configName, configPairs[iSettings].Key) == 0)
- || (!caseSensitive && _stricmp(configName, configPairs[iSettings].Key) == 0))
+ if ((caseSensitive && strcmp(configName, config->GetKeyAt(iSettings)) == 0)
+ || (!caseSensitive && _stricmp(configName, config->GetKeyAt(iSettings)) == 0))
{
- *configValue = configPairs[iSettings].Value;
+ *configValue = config->GetValueAt(iSettings);
return true;
}
}
@@ -200,107 +195,4 @@ bool RhConfig::GetEmbeddedVariable(void *volatile * embeddedSettings, void* comp
return false;
}
-void RhConfig::ReadEmbeddedSettings(void *volatile * embeddedSettings, void* compilerEmbeddedSettingsBlob)
-{
- if (*embeddedSettings == NULL)
- {
- uint32_t size = ((CompilerEmbeddedSettingsBlob*)compilerEmbeddedSettingsBlob)->Size;
- char* data = ((CompilerEmbeddedSettingsBlob*)compilerEmbeddedSettingsBlob)->Data;
-
- //if reading the file contents failed set embeddedSettings to CONFIG_INI_NOT_AVAIL
- if (size == 0)
- {
- //only set if another thread hasn't initialized the buffer yet, otherwise ignore and let the first setter win
- PalInterlockedCompareExchangePointer(embeddedSettings, CONFIG_INI_NOT_AVAIL, NULL);
-
- return;
- }
-
- ConfigPair* iniBuff = new (nothrow) ConfigPair[size];
- if (iniBuff == NULL)
- {
- //only set if another thread hasn't initialized the buffer yet, otherwise ignore and let the first setter win
- PalInterlockedCompareExchangePointer(embeddedSettings, CONFIG_INI_NOT_AVAIL, NULL);
-
- return;
- }
-
- uint32_t iBuff = 0;
- uint32_t iIniBuff = 0;
- char* currLine;
-
- //while we haven't reached the max number of config pairs, or the end of the file, read the next line
- while (iBuff < size)
- {
- currLine = &data[iBuff];
-
- //find the end of the line
- while ((data[iBuff] != '\0') && (iBuff < size))
- iBuff++;
-
- //parse the line
- //only increment iIniBuff if the parsing succeeded otherwise reuse the config struct
- if (ParseConfigLine(&iniBuff[iIniBuff], currLine))
- {
- iIniBuff++;
- }
-
- //advance to the next line;
- iBuff++;
- }
-
- //if another thread initialized first let the first setter win
- //delete the iniBuff to avoid leaking memory
- if (PalInterlockedCompareExchangePointer(embeddedSettings, iniBuff, NULL) != NULL)
- {
- delete[] iniBuff;
- }
- }
-
- return;
-}
-
-//Parses one line of config and populates values in the passed in configPair
-//returns: true if the parsing was successful, false if the parsing failed.
-//NOTE: if the method fails configPair is left in an uninitialized state
-bool RhConfig::ParseConfigLine(_Out_ ConfigPair* configPair, _In_z_ const char * line)
-{
- uint32_t iLine = 0;
- uint32_t iKey = 0;
- uint32_t iVal = 0;
-
- //while we haven't reached the end of the key signalled by '=', or the end of the line, or the key maxlen
- while (line[iLine] != '=' && line[iLine] != '\0' && iKey < CONFIG_KEY_MAXLEN)
- {
- configPair->Key[iKey++] = line[iLine++];
- }
-
- //if the current char is not '=' we reached the key maxlen, or the line ended return false
- if (line[iLine] != '=')
- {
- return FALSE;
- }
-
- configPair->Key[iKey] = '\0';
-
- //increment to start of the value
- iLine++;
-
- //while we haven't reached the end of the line, or val maxlen
- while (line[iLine] != '\0' && iVal < CONFIG_VAL_MAXLEN)
- {
- configPair->Value[iVal++] = line[iLine++];
- }
-
- //if the current char is not '\0' we didn't reach the end of the line return false
- if (line[iLine] != '\0')
- {
- return FALSE;
- }
-
- configPair->Value[iVal] = '\0';
-
- return TRUE;
-}
-
#endif
diff --git a/src/coreclr/nativeaot/Runtime/RhConfig.h b/src/coreclr/nativeaot/Runtime/RhConfig.h
index a3cd83493cab96..59d1d37fc15546 100644
--- a/src/coreclr/nativeaot/Runtime/RhConfig.h
+++ b/src/coreclr/nativeaot/Runtime/RhConfig.h
@@ -11,6 +11,8 @@
// Values can also be embedded in the compiled binary.
//
+#ifndef RHCONFIG_H
+#define RHCONFIG_H
#ifndef DACCESS_COMPILE
@@ -18,26 +20,20 @@
class RhConfig
{
-
-#define CONFIG_INI_NOT_AVAIL (void*)0x1 //signal for ini file failed to load
-#define CONFIG_KEY_MAXLEN 50 //arbitrary max length of config keys increase if needed
#define CONFIG_VAL_MAXLEN 16 //64 bit uint in hex
-private:
- struct ConfigPair
+public:
+ struct Config
{
+ uint32_t m_count;
+ char* m_first[];
public:
- char Key[CONFIG_KEY_MAXLEN + 1]; //maxlen + null terminator
- char Value[CONFIG_VAL_MAXLEN + 1]; //maxlen + null terminator
+ uint32_t GetCount() { return m_count; }
+ char* GetKeyAt(int32_t index) { return m_first[index]; }
+ char* GetValueAt(int32_t index) { return m_first[m_count + index]; }
+ char** GetKeys() { return m_first; }
+ char** GetValues() { return &m_first[m_count]; }
};
- // g_embeddedSettings is a buffer of ConfigPair structs embedded in the compiled binary.
- //
- //NOTE: g_embeddedSettings is only set in ReadEmbeddedSettings and must be set atomically only once
- // using PalInterlockedCompareExchangePointer to avoid races when initializing
- void* volatile g_embeddedSettings = NULL;
- void* volatile g_embeddedKnobs = NULL;
-
-public:
class Environment
{
public: // static
@@ -52,6 +48,10 @@ class RhConfig
bool ReadKnobUInt64Value(_In_z_ const char* wszName, uint64_t* pValue);
bool ReadKnobBooleanValue(_In_z_ const char* wszName, bool* pValue);
+ char** GetKnobNames();
+ char** GetKnobValues();
+ uint32_t GetKnobCount();
+
#define DEFINE_VALUE_ACCESSOR(_name, defaultVal) \
uint64_t Get##_name() \
{ \
@@ -99,16 +99,9 @@ class RhConfig
#define CONFIG_FILE_MAXLEN RCV_Count * sizeof(ConfigPair) + 2000
private:
- //Parses one line of config and populates values in the passed in configPair
- //returns: true if the parsing was successful, false if the parsing failed.
- //NOTE: if the method fails configPair is left in an uninitialized state
- bool ParseConfigLine(_Out_ ConfigPair* configPair, _In_z_ const char * line);
-
- void ReadEmbeddedSettings(void *volatile * embeddedSettings, void* compilerEmbeddedSettingsBlob);
-
// Gets a pointer to the embedded configuration value. Memory is held by the callee.
// Returns true if the variable was found, false otherwise
- bool GetEmbeddedVariable(void *volatile * embeddedSettings, void* compilerEmbeddedSettingsBlob, _In_z_ const char* configName, bool caseSensitive, _Out_ const char** configValue);
+ bool GetEmbeddedVariable(Config* config, _In_z_ const char* configName, bool caseSensitive, _Out_ const char** configValue);
uint32_t m_uiConfigValuesRead;
uint64_t m_uiConfigValues[RCV_Count];
@@ -117,3 +110,5 @@ class RhConfig
extern RhConfig * g_pRhConfig;
#endif //!DACCESS_COMPILE
+
+#endif // RHCONFIG_H
diff --git a/src/coreclr/nativeaot/Runtime/RuntimeInstance.cpp b/src/coreclr/nativeaot/Runtime/RuntimeInstance.cpp
index 146c15aeff192c..2d1a6971b10fb8 100644
--- a/src/coreclr/nativeaot/Runtime/RuntimeInstance.cpp
+++ b/src/coreclr/nativeaot/Runtime/RuntimeInstance.cpp
@@ -35,14 +35,6 @@ bool ShouldHijackForGcStress(uintptr_t CallsiteIP, HijackType ht);
#include "shash.inl"
-#ifndef DACCESS_COMPILE
-COOP_PINVOKE_HELPER(uint8_t *, RhSetErrorInfoBuffer, (uint8_t * pNewBuffer))
-{
- return (uint8_t *) PalSetWerDataBuffer(pNewBuffer);
-}
-#endif // DACCESS_COMPILE
-
-
ThreadStore * RuntimeInstance::GetThreadStore()
{
return m_pThreadStore;
diff --git a/src/coreclr/nativeaot/Runtime/StackFrameIterator.cpp b/src/coreclr/nativeaot/Runtime/StackFrameIterator.cpp
index 3dc6e98dfeb76d..f4a19246e0137d 100644
--- a/src/coreclr/nativeaot/Runtime/StackFrameIterator.cpp
+++ b/src/coreclr/nativeaot/Runtime/StackFrameIterator.cpp
@@ -44,9 +44,6 @@ GVAL_IMPL_INIT(PTR_VOID, g_ReturnFromUniversalTransitionAddr, PointerToReturnFro
EXTERN_C PTR_VOID PointerToReturnFromUniversalTransition_DebugStepTailCall;
GVAL_IMPL_INIT(PTR_VOID, g_ReturnFromUniversalTransition_DebugStepTailCallAddr, PointerToReturnFromUniversalTransition_DebugStepTailCall);
-
-EXTERN_C PTR_VOID PointerToReturnFromCallDescrThunk;
-GVAL_IMPL_INIT(PTR_VOID, g_ReturnFromCallDescrThunkAddr, PointerToReturnFromCallDescrThunk);
#endif
#ifdef TARGET_X86
@@ -1200,130 +1197,6 @@ void StackFrameIterator::UnwindUniversalTransitionThunk()
#define STACK_ALIGN_SIZE 4
#endif
-#ifdef TARGET_AMD64
-struct CALL_DESCR_CONTEXT
-{
- uintptr_t Rbp;
- uintptr_t Rsi;
- uintptr_t Rbx;
- uintptr_t IP;
-};
-#elif defined(TARGET_ARM)
-struct CALL_DESCR_CONTEXT
-{
- uintptr_t R4;
- uintptr_t R5;
- uintptr_t R7;
- uintptr_t IP;
-};
-#elif defined(TARGET_ARM64)
-struct CALL_DESCR_CONTEXT
-{
- uintptr_t FP;
- uintptr_t IP;
- uintptr_t X19;
- uintptr_t X20;
-};
-#elif defined(TARGET_X86)
-struct CALL_DESCR_CONTEXT
-{
- uintptr_t Rbx;
- uintptr_t Rbp;
- uintptr_t IP;
-};
-#elif defined (TARGET_WASM)
-struct CALL_DESCR_CONTEXT
-{
- uintptr_t IP;
-};
-#else
-#error NYI - For this arch
-#endif
-
-typedef DPTR(CALL_DESCR_CONTEXT) PTR_CALL_DESCR_CONTEXT;
-
-void StackFrameIterator::UnwindCallDescrThunk()
-{
- ASSERT((m_dwFlags & MethodStateCalculated) == 0);
-
-#if defined(USE_PORTABLE_HELPERS) // @TODO: Corresponding helper code is only defined in assembly code
- return;
-#else // defined(USE_PORTABLE_HELPERS)
- ASSERT(CategorizeUnadjustedReturnAddress(m_ControlPC) == InCallDescrThunk);
-
- uintptr_t newSP;
-#ifdef TARGET_AMD64
- // RBP points to the SP that we want to capture. (This arrangement allows for
- // the arguments from this function to be loaded into memory with an adjustment
- // to SP, like an alloca
- newSP = *(PTR_UIntNative)m_RegDisplay.pRbp;
-
- PTR_CALL_DESCR_CONTEXT pContext = (PTR_CALL_DESCR_CONTEXT)newSP;
-
- m_RegDisplay.pRbp = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, Rbp);
- m_RegDisplay.pRsi = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, Rsi);
- m_RegDisplay.pRbx = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, Rbx);
-
- // And adjust SP to be the state that it should be in just after returning from
- // the CallDescrFunction
- newSP += sizeof(CALL_DESCR_CONTEXT);
-#elif defined(TARGET_ARM)
- // R7 points to the SP that we want to capture. (This arrangement allows for
- // the arguments from this function to be loaded into memory with an adjustment
- // to SP, like an alloca
- newSP = *(PTR_UIntNative)m_RegDisplay.pR7;
- PTR_CALL_DESCR_CONTEXT pContext = (PTR_CALL_DESCR_CONTEXT)newSP;
-
- m_RegDisplay.pR4 = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, R4);
- m_RegDisplay.pR5 = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, R5);
- m_RegDisplay.pR7 = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, R7);
-
- // And adjust SP to be the state that it should be in just after returning from
- // the CallDescrFunction
- newSP += sizeof(CALL_DESCR_CONTEXT);
-
-#elif defined(TARGET_ARM64)
- // pFP points to the SP that we want to capture. (This arrangement allows for
- // the arguments from this function to be loaded into memory with an adjustment
- // to SP, like an alloca
- newSP = *(PTR_UIntNative)m_RegDisplay.pFP;
- PTR_CALL_DESCR_CONTEXT pContext = (PTR_CALL_DESCR_CONTEXT)newSP;
-
- m_RegDisplay.pX19 = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, X19);
- m_RegDisplay.pX20 = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, X20);
-
- // And adjust SP to be the state that it should be in just after returning from
- // the CallDescrFunction
- newSP += sizeof(CALL_DESCR_CONTEXT);
-
-#elif defined(TARGET_X86)
- // RBP points to the SP that we want to capture. (This arrangement allows for
- // the arguments from this function to be loaded into memory with an adjustment
- // to SP, like an alloca
- newSP = *(PTR_UIntNative)m_RegDisplay.pRbp;
-
- PTR_CALL_DESCR_CONTEXT pContext = (PTR_CALL_DESCR_CONTEXT)(newSP - offsetof(CALL_DESCR_CONTEXT, Rbp));
-
- m_RegDisplay.pRbp = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, Rbp);
- m_RegDisplay.pRbx = PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, Rbx);
-
- // And adjust SP to be the state that it should be in just after returning from
- // the CallDescrFunction
- newSP += sizeof(CALL_DESCR_CONTEXT) - offsetof(CALL_DESCR_CONTEXT, Rbp);
-
-#else
- PORTABILITY_ASSERT("UnwindCallDescrThunk");
- PTR_CALL_DESCR_CONTEXT pContext = NULL;
-#endif
-
- m_RegDisplay.SetAddrOfIP(PTR_TO_MEMBER(CALL_DESCR_CONTEXT, pContext, IP));
- m_RegDisplay.SetIP(pContext->IP);
- m_RegDisplay.SetSP(newSP);
- SetControlPC(dac_cast(pContext->IP));
-
-#endif // defined(USE_PORTABLE_HELPERS)
-}
-
void StackFrameIterator::UnwindThrowSiteThunk()
{
ASSERT((m_dwFlags & MethodStateCalculated) == 0);
@@ -1442,6 +1315,11 @@ void StackFrameIterator::NextInternal()
unwindFlags |= USFF_StopUnwindOnTransitionFrame;
}
+ if ((m_dwFlags & GcStackWalkFlags) == GcStackWalkFlags)
+ {
+ unwindFlags |= USFF_GcUnwind;
+ }
+
FAILFAST_OR_DAC_FAIL(GetCodeManager()->UnwindStackFrame(&m_methodInfo, unwindFlags, &m_RegDisplay,
&m_pPreviousTransitionFrame));
@@ -1654,11 +1532,7 @@ void StackFrameIterator::UnwindNonEHThunkSequence()
{
ASSERT(m_pConservativeStackRangeLowerBound == NULL);
- if (category == InCallDescrThunk)
- {
- UnwindCallDescrThunk();
- }
- else if (category == InUniversalTransitionThunk)
+ if (category == InUniversalTransitionThunk)
{
UnwindUniversalTransitionThunk();
ASSERT(m_pConservativeStackRangeLowerBound != NULL);
@@ -1851,7 +1725,6 @@ bool StackFrameIterator::IsNonEHThunk(ReturnAddressCategory category)
default:
return false;
case InUniversalTransitionThunk:
- case InCallDescrThunk:
return true;
}
}
@@ -1921,11 +1794,7 @@ StackFrameIterator::ReturnAddressCategory StackFrameIterator::CategorizeUnadjust
#else // defined(USE_PORTABLE_HELPERS)
#if defined(FEATURE_DYNAMIC_CODE)
- if (EQUALS_RETURN_ADDRESS(returnAddress, ReturnFromCallDescrThunk))
- {
- return InCallDescrThunk;
- }
- else if (EQUALS_RETURN_ADDRESS(returnAddress, ReturnFromUniversalTransition) ||
+ if (EQUALS_RETURN_ADDRESS(returnAddress, ReturnFromUniversalTransition) ||
EQUALS_RETURN_ADDRESS(returnAddress, ReturnFromUniversalTransition_DebugStepTailCall))
{
return InUniversalTransitionThunk;
diff --git a/src/coreclr/nativeaot/Runtime/StackFrameIterator.h b/src/coreclr/nativeaot/Runtime/StackFrameIterator.h
index 9551e4caeed497..5ea7676cf51e8f 100644
--- a/src/coreclr/nativeaot/Runtime/StackFrameIterator.h
+++ b/src/coreclr/nativeaot/Runtime/StackFrameIterator.h
@@ -79,10 +79,6 @@ class StackFrameIterator
// NOTE: This function always publishes a non-NULL conservative stack range lower bound.
void UnwindUniversalTransitionThunk();
- // If our control PC indicates that we're in the call descr thunk that we use to call an arbitrary managed
- // function with an arbitrary signature from a normal managed function handle the stack walk specially.
- void UnwindCallDescrThunk();
-
void EnterInitialInvalidState(Thread * pThreadToWalk);
void InternalInit(Thread * pThreadToWalk, PTR_PInvokeTransitionFrame pFrame, uint32_t dwFlags); // GC stackwalk
@@ -115,7 +111,6 @@ class StackFrameIterator
InThrowSiteThunk,
InFuncletInvokeThunk,
InFilterFuncletInvokeThunk,
- InCallDescrThunk,
InUniversalTransitionThunk,
};
diff --git a/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.S b/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.S
deleted file mode 100644
index 483d4b5f9ac8cc..00000000000000
--- a/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.S
+++ /dev/null
@@ -1,13 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-.intel_syntax noprefix
-#include
-
-NESTED_ENTRY RhCallDescrWorker, _TEXT, NoHandler
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- // UNIXTODO: Implement this function
- int 3
-NESTED_END RhCallDescrWorker, _TEXT
diff --git a/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.asm b/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.asm
deleted file mode 100644
index 3f4ded05d9918e..00000000000000
--- a/src/coreclr/nativeaot/Runtime/amd64/CallDescrWorker.asm
+++ /dev/null
@@ -1,105 +0,0 @@
-;; Licensed to the .NET Foundation under one or more agreements.
-;; The .NET Foundation licenses this file to you under the MIT license.
-
-include AsmMacros.inc
-
-
-;;;;;;;;;;;;;;;;;;;;;;; CallingConventionConverter Thunks Helpers ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-;;extern "C" void RhCallDescrWorker(CallDescrData * pCallDescrData);
-
- NESTED_ENTRY RhCallDescrWorker, _TEXT
-
- push_nonvol_reg rbx ; save nonvolatile registers
- push_nonvol_reg rsi ;
- push_nonvol_reg rbp ;
- set_frame rbp, 0 ; set frame pointer
-
- END_PROLOGUE
-
- mov rbx, rcx ; save pCallDescrData in rbx
-
- mov ecx, dword ptr [rbx + OFFSETOF__CallDescrData__numStackSlots]
-
- test ecx, 1
- jz StackAligned
- push rax
-StackAligned:
-
- mov rsi, [rbx + OFFSETOF__CallDescrData__pSrc] ; set source argument list address
- lea rsi, [rsi + 8 * rcx]
-
-StackCopyLoop: ; copy the arguments to stack top-down to carefully probe for sufficient
- ; stack space
- sub rsi, 8
- push qword ptr [rsi]
- dec ecx
- jnz StackCopyLoop
-
- ;
- ; N.B. All four argument registers are loaded regardless of the actual number
- ; of arguments.
- ;
-
- mov rax, [rbx + OFFSETOF__CallDescrData__pFloatArgumentRegisters] ; get floating pointer arg registers pointer
-
- mov rcx, 0[rsp] ; load first four argument registers
- mov rdx, 8[rsp] ;
- mov r8, 10h[rsp] ;
- mov r9, 18h[rsp] ;
- test rax, rax ;
- jz DoCall ;
- movdqa xmm0, [rax + 00h] ; load floating point registers if they are used
- movdqa xmm1, [rax + 10h] ;
- movdqa xmm2, [rax + 20h] ;
- movdqa xmm3, [rax + 30h] ;
-DoCall:
- call qword ptr [rbx + OFFSETOF__CallDescrData__pTarget] ; call target function
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- ; Symbol used to identify thunk call to managed function so the special
- ; case unwinder can unwind through this function. Sadly we cannot directly
- ; export this symbol right now because it confuses DIA unwinder to believe
- ; it's the beginning of a new method, therefore we export the address
- ; of an auxiliary variable holding the address instead.
-
- ; Save FP return value
-
- mov ecx, dword ptr [rbx + OFFSETOF__CallDescrData__fpReturnSize]
- test ecx, ecx
- jz ReturnsInt
-
- cmp ecx, 4
- je ReturnsFloat
- cmp ecx, 8
- je ReturnsDouble
- ; unexpected
- jmp Epilog
-
-ReturnsInt:
- mov rbx, [rbx + OFFSETOF__CallDescrData__pReturnBuffer]
- mov [rbx], rax
-
-Epilog:
- lea rsp, 0[rbp] ; deallocate argument list
- pop rbp ; restore nonvolatile register
- pop rsi ;
- pop rbx ;
- ret
-
-ReturnsFloat:
-; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- mov rbx, [rbx + OFFSETOF__CallDescrData__pReturnBuffer]
- movss real4 ptr [rbx], xmm0
- jmp Epilog
-
-ReturnsDouble:
-; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- mov rbx, [rbx + OFFSETOF__CallDescrData__pReturnBuffer]
- movsd real8 ptr [rbx], xmm0
- jmp Epilog
-
- NESTED_END RhCallDescrWorker, _TEXT
-
-end
diff --git a/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.S b/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.S
deleted file mode 100644
index 53184d9b28b2a3..00000000000000
--- a/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.S
+++ /dev/null
@@ -1,20 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-.syntax unified
-.thumb
-
-#include // generated by the build from AsmOffsets.cpp
-#include
-
-// TODO: Implement Arm support
-
-NESTED_ENTRY RhCallDescrWorker, _TEXT, NoHandler
-LOCAL_LABEL(ReturnFromCallDescrThunk):
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- // UNIXTODO: Implement this function
- EMIT_BREAKPOINT
-NESTED_END RhCallDescrWorker, _TEXT
-
diff --git a/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.asm b/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.asm
deleted file mode 100644
index 7ee66320cb3325..00000000000000
--- a/src/coreclr/nativeaot/Runtime/arm/CallDescrWorker.asm
+++ /dev/null
@@ -1,128 +0,0 @@
-;; Licensed to the .NET Foundation under one or more agreements.
-;; The .NET Foundation licenses this file to you under the MIT license.
-
-#include "AsmMacros.h"
-
- TEXTAREA
-
-;;-----------------------------------------------------------------------------
-;; This helper routine enregisters the appropriate arguments and makes the
-;; actual call.
-;;-----------------------------------------------------------------------------
-;;void RhCallDescrWorker(CallDescrData * pCallDescrData);
- NESTED_ENTRY RhCallDescrWorker
- PROLOG_PUSH {r4,r5,r7,lr}
- PROLOG_STACK_SAVE r7
-
- mov r5,r0 ; save pCallDescrData in r5
-
- ldr r1, [r5,#OFFSETOF__CallDescrData__numStackSlots]
- cbz r1, Ldonestack
-
- ;; Add frame padding to ensure frame size is a multiple of 8 (a requirement of the OS ABI).
- ;; We push four registers (above) and numStackSlots arguments (below). If this comes to an odd number
- ;; of slots we must pad with another. This simplifies to "if the low bit of numStackSlots is set,
- ;; extend the stack another four bytes".
- lsls r2, r1, #2
- and r3, r2, #4
- sub sp, sp, r3
-
- ;; This loop copies numStackSlots words
- ;; from [pSrcEnd-4,pSrcEnd-8,...] to [sp-4,sp-8,...]
- ldr r0, [r5,#OFFSETOF__CallDescrData__pSrc]
- add r0,r0,r2
-Lstackloop
- ldr r2, [r0,#-4]!
- str r2, [sp,#-4]!
- subs r1, r1, #1
- bne Lstackloop
-Ldonestack
-
- ;; If FP arguments are supplied in registers (r3 != NULL) then initialize all of them from the pointer
- ;; given in r3. Do not use "it" since it faults in floating point even when the instruction is not executed.
- ldr r3, [r5,#OFFSETOF__CallDescrData__pFloatArgumentRegisters]
- cbz r3, LNoFloatingPoint
- vldm r3, {s0-s15}
-LNoFloatingPoint
-
- ;; Copy [pArgumentRegisters, ..., pArgumentRegisters + 12]
- ;; into r0, ..., r3
-
- ldr r4, [r5,#OFFSETOF__CallDescrData__pArgumentRegisters]
- ldm r4, {r0-r3}
-
- CHECK_STACK_ALIGNMENT
-
- ;; call pTarget
- ;; Note that remoting expect target in r4.
- ldr r4, [r5,#OFFSETOF__CallDescrData__pTarget]
- blx r4
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- ;; Symbol used to identify thunk call to managed function so the special
- ;; case unwinder can unwind through this function. Sadly we cannot directly
- ;; export this symbol right now because it confuses DIA unwinder to believe
- ;; it's the beginning of a new method, therefore we export the address
- ;; of an auxiliary variable holding the address instead.
-
- ldr r3, [r5,#OFFSETOF__CallDescrData__fpReturnSize]
-
- ;; Save FP return value if appropriate
- cbz r3, LFloatingPointReturnDone
-
- ;; Float return case
- ;; Do not use "it" since it faults in floating point even when the instruction is not executed.
- cmp r3, #4
- bne LNoFloatReturn
- vmov r0, s0
- b LFloatingPointReturnDone
-LNoFloatReturn
-
- ;; Double return case
- ;; Do not use "it" since it faults in floating point even when the instruction is not executed.
- cmp r3, #8
- bne LNoDoubleReturn
- vmov r0, r1, s0, s1
- b LFloatingPointReturnDone
-LNoDoubleReturn
-; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- ldr r2, [r5, #OFFSETOF__CallDescrData__pReturnBuffer]
-
- cmp r3, #16
- bne LNoFloatHFAReturn
- vstm r2, {s0-s3}
- b LReturnDone
-LNoFloatHFAReturn
-
- cmp r3, #32
- bne LNoDoubleHFAReturn
- vstm r2, {d0-d3}
- b LReturnDone
-LNoDoubleHFAReturn
-
- EMIT_BREAKPOINT ; Unreachable
-
-LFloatingPointReturnDone
-
-; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- ldr r5, [r5, #OFFSETOF__CallDescrData__pReturnBuffer]
-
- ;; Save return value into retbuf
- str r0, [r5, #(0)]
- str r1, [r5, #(4)]
-
-LReturnDone
-
-#ifdef _DEBUG
- ;; trash the floating point registers to ensure that the HFA return values
- ;; won't survive by accident
- vldm sp, {d0-d3}
-#endif
-
- EPILOG_STACK_RESTORE r7
- EPILOG_POP {r4,r5,r7,pc}
-
- NESTED_END RhCallDescrWorker
-
- END
diff --git a/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.S b/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.S
deleted file mode 100644
index fa24f81393ac81..00000000000000
--- a/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.S
+++ /dev/null
@@ -1,140 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-#include
-#include "AsmOffsets.inc"
-
-//-----------------------------------------------------------------------------
-// This helper routine enregisters the appropriate arguments and makes the
-// actual call.
-//
-// INPUT: x0: pointer to CallDescrData struct
-//
-//-----------------------------------------------------------------------------
-//void RhCallDescrWorker(CallDescrData * pCallDescrData);
- NESTED_ENTRY RhCallDescrWorker, _TEXT, NoHandler
-
- PROLOG_SAVE_REG_PAIR_INDEXED fp, lr, -32
- PROLOG_SAVE_REG_PAIR x19, x20, 16
-
- // Save the value of SP before we start pushing any arguments
- mov x20, sp
-
- mov x19, x0 // save pCallDescrData in x19
-
- ldr w1, [x19, #OFFSETOF__CallDescrData__numStackSlots]
- cbz w1, Ldonestack
-
- // Add frame padding to ensure frame size is a multiple of 16 (a requirement of the OS ABI).
- // We push two registers (above) and numStackSlots arguments (below). If this comes to an odd number
- // of slots we must pad with another. This simplifies to "if the low bit of numStackSlots is set,
- // extend the stack another eight bytes".
- ldr x0, [x19, #OFFSETOF__CallDescrData__pSrc]
- add x0, x0, x1, lsl #3 // pSrcEnd=pSrc+8*numStackSlots
- ands x2, x1, #1
- beq Lstackloop
-
- // This loop copies numStackSlots words
- // from [pSrcEnd-8,pSrcEnd-16,...] to [sp-8,sp-16,...]
-
- // Pad and store one stack slot as number of slots are odd
- ldr x4, [x0,#-8]!
- str x4, [sp,#-16]!
- subs x1, x1, #1
- beq Ldonestack
-Lstackloop:
- ldp x2, x4, [x0,#-16]!
- stp x2, x4, [sp,#-16]!
- subs x1, x1, #2
- bne Lstackloop
-Ldonestack:
-
- // If FP arguments are supplied in registers (x9 != NULL) then initialize all of them from the pointer
- // given in x9.
- ldr x9, [x19, #OFFSETOF__CallDescrData__pFloatArgumentRegisters]
- cbz x9, LNoFloatingPoint
- ldp d0, d1, [x9]
- ldp d2, d3, [x9, #16]
- ldp d4, d5, [x9, #32]
- ldp d6, d7, [x9, #48]
-LNoFloatingPoint:
-
- // Copy [pArgumentRegisters, ..., pArgumentRegisters + 64]
- // into x0, ..., x7, x8
-
- ldr x9, [x19, #OFFSETOF__CallDescrData__pArgumentRegisters]
- ldp x0, x1, [x9]
- ldp x2, x3, [x9, #16]
- ldp x4, x5, [x9, #32]
- ldp x6, x7, [x9, #48]
- ldr x8, [x9, #64]
-
- // call pTarget
- ldr x9, [x19, #OFFSETOF__CallDescrData__pTarget]
- blr x9
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- // Symbol used to identify thunk call to managed function so the special
- // case unwinder can unwind through this function. Sadly we cannot directly
- // export this symbol right now because it confuses DIA unwinder to believe
- // it's the beginning of a new method, therefore we export the address
- // of an auxiliary variable holding the address instead.
-
- ldr w3, [x19, #OFFSETOF__CallDescrData__fpReturnSize]
-
- // Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- ldr x19, [x19, #OFFSETOF__CallDescrData__pReturnBuffer]
-
- // Int return case
- cbz w3, LIntReturn
-
- // Float return case
- cmp w3, #4
- beq LFloatOrDoubleReturn
-
- // Double return case
- cmp w3, #8
- bne LCheckHFAReturn
-
-LFloatOrDoubleReturn:
- str d0, [x19]
- b LReturnDone
-
-LCheckHFAReturn:
- cmp w3, #16
- beq LFloatOrDoubleHFAReturn
- cmp w3, #32
- beq LFloatOrDoubleHFAReturn
- b LNoHFAReturn
-
-LFloatOrDoubleHFAReturn:
- //Single/Double HFAReturn return case
- stp d0, d1, [x19, #00]
- stp d2, d3, [x19, #16]
- b LReturnDone
-
-LNoHFAReturn:
-
- EMIT_BREAKPOINT // Unreachable
-
-LIntReturn:
- // Save return value(s) into retbuf for int
- stp x0, x1, [x19]
-
-LReturnDone:
-
-#ifdef _DEBUG
- // Trash the floating point registers to ensure that the HFA return values
- // won't survive by accident
- ldp d0, d1, [sp]
- ldp d2, d3, [sp, #16]
-#endif
- // Restore the value of SP
- mov sp, x20
-
- EPILOG_RESTORE_REG_PAIR x19, x20, 16
- EPILOG_RESTORE_REG_PAIR_INDEXED fp, lr, 32
- EPILOG_RETURN
-
- NESTED_END RhCallDescrWorker
diff --git a/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.asm b/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.asm
deleted file mode 100644
index 09eef6d781ebf4..00000000000000
--- a/src/coreclr/nativeaot/Runtime/arm64/CallDescrWorker.asm
+++ /dev/null
@@ -1,143 +0,0 @@
-;; Licensed to the .NET Foundation under one or more agreements.
-;; The .NET Foundation licenses this file to you under the MIT license.
-
-#include "AsmMacros.h"
-
- TEXTAREA
-
-;;-----------------------------------------------------------------------------
-;; This helper routine enregisters the appropriate arguments and makes the
-;; actual call.
-;;
-;; INPUT: x0: pointer to CallDescrData struct
-;;
-;;-----------------------------------------------------------------------------
-;;void RhCallDescrWorker(CallDescrData * pCallDescrData);
- NESTED_ENTRY RhCallDescrWorker
-
- PROLOG_SAVE_REG_PAIR fp, lr, #-32!
- PROLOG_SAVE_REG_PAIR x19, x20, #16
-
- ;; Save the value of SP before we start pushing any arguments
- mov x20, sp
-
- mov x19, x0 ; save pCallDescrData in x19
-
- ldr w1, [x19, #OFFSETOF__CallDescrData__numStackSlots]
- cbz w1, Ldonestack
-
- ;; Add frame padding to ensure frame size is a multiple of 16 (a requirement of the OS ABI).
- ;; We push two registers (above) and numStackSlots arguments (below). If this comes to an odd number
- ;; of slots we must pad with another. This simplifies to "if the low bit of numStackSlots is set,
- ;; extend the stack another eight bytes".
- ldr x0, [x19, #OFFSETOF__CallDescrData__pSrc]
- add x0, x0, x1 lsl #3 ; pSrcEnd=pSrc+8*numStackSlots
- ands x2, x1, #1
- beq Lstackloop
-
- ;; This loop copies numStackSlots words
- ;; from [pSrcEnd-8,pSrcEnd-16,...] to [sp-8,sp-16,...]
-
- ;; Pad and store one stack slot as number of slots are odd
- ldr x4, [x0,#-8]!
- str x4, [sp,#-16]!
- subs x1, x1, #1
- beq Ldonestack
-Lstackloop
- ldp x2, x4, [x0,#-16]!
- stp x2, x4, [sp,#-16]!
- subs x1, x1, #2
- bne Lstackloop
-Ldonestack
-
- ;; If FP arguments are supplied in registers (x9 != NULL) then initialize all of them from the pointer
- ;; given in x9.
- ldr x9, [x19, #OFFSETOF__CallDescrData__pFloatArgumentRegisters]
- cbz x9, LNoFloatingPoint
- ldp d0, d1, [x9]
- ldp d2, d3, [x9, #16]
- ldp d4, d5, [x9, #32]
- ldp d6, d7, [x9, #48]
-LNoFloatingPoint
-
- ;; Copy [pArgumentRegisters, ..., pArgumentRegisters + 64]
- ;; into x0, ..., x7, x8
-
- ldr x9, [x19, #OFFSETOF__CallDescrData__pArgumentRegisters]
- ldp x0, x1, [x9]
- ldp x2, x3, [x9, #16]
- ldp x4, x5, [x9, #32]
- ldp x6, x7, [x9, #48]
- ldr x8, [x9, #64]
-
- ;; call pTarget
- ldr x9, [x19, #OFFSETOF__CallDescrData__pTarget]
- blr x9
-
- EXPORT_POINTER_TO_ADDRESS PointerToReturnFromCallDescrThunk
-
- ;; Symbol used to identify thunk call to managed function so the special
- ;; case unwinder can unwind through this function. Sadly we cannot directly
- ;; export this symbol right now because it confuses DIA unwinder to believe
- ;; it's the beginning of a new method, therefore we export the address
- ;; of an auxiliary variable holding the address instead.
-
- ldr w3, [x19, #OFFSETOF__CallDescrData__fpReturnSize]
-
- ;; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- ldr x19, [x19, #OFFSETOF__CallDescrData__pReturnBuffer]
-
- ;; Int return case
- cbz w3, LIntReturn
-
- ;; Float return case
- cmp w3, #4
- beq LFloatOrDoubleReturn
-
- ;; Double return case
- cmp w3, #8
- bne LCheckHFAReturn
-
-LFloatOrDoubleReturn
- str d0, [x19]
- b LReturnDone
-
-LCheckHFAReturn
- cmp w3, #16
- beq LFloatOrDoubleHFAReturn
- cmp w3, #32
- beq LFloatOrDoubleHFAReturn
- b LNoHFAReturn
-
-LFloatOrDoubleHFAReturn
- ;;Single/Double HFAReturn return case
- stp d0, d1, [x19, #00]
- stp d2, d3, [x19, #16]
- b LReturnDone
-
-LNoHFAReturn
-
- EMIT_BREAKPOINT ; Unreachable
-
-LIntReturn
- ;; Save return value(s) into retbuf for int
- stp x0, x1, [x19]
-
-LReturnDone
-
-#ifdef _DEBUG
- ;; Trash the floating point registers to ensure that the HFA return values
- ;; won't survive by accident
- ldp d0, d1, [sp]
- ldp d2, d3, [sp, #16]
-#endif
- ;; Restore the value of SP
- mov sp, x20
-
- EPILOG_RESTORE_REG_PAIR x19, x20, #16
- EPILOG_RESTORE_REG_PAIR fp, lr, #32!
- EPILOG_RETURN
-
- NESTED_END RhCallDescrWorker
-
- END
diff --git a/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.cpp b/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.cpp
index 9bb8d088a8d99b..25272643d18a30 100644
--- a/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.cpp
+++ b/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.cpp
@@ -1,6 +1,24 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+#include
+
+#ifdef __APPLE__
+#include
+#endif
+
+#ifdef __NetBSD__
+#include
+#include
+#include
+#include
+#endif
+
+#ifdef __FreeBSD__
+#include
+#include
+#endif
+
#include
#ifdef ENABLE_PERFTRACING
@@ -13,6 +31,7 @@
bool aot_ipc_get_process_id_disambiguation_key(uint32_t process_id, uint64_t *key);
+// Consider moving this code to shared library code. See https://github.com/dotnet/runtime/issues/87069
bool
aot_ipc_get_process_id_disambiguation_key(
uint32_t process_id,
@@ -26,7 +45,54 @@ aot_ipc_get_process_id_disambiguation_key(
*key = 0;
// Mono implementation, restricted just to Unix
-#ifdef TARGET_UNIX
+#if defined (__APPLE__) || defined (__FreeBSD__)
+ // On OS X, we return the process start time expressed in Unix time (the number of seconds
+ // since the start of the Unix epoch).
+ struct kinfo_proc info = {};
+ size_t size = sizeof (info);
+ int mib [4] = { CTL_KERN, KERN_PROC, KERN_PROC_PID, process_id };
+
+ const int result_sysctl = sysctl (mib, sizeof(mib)/sizeof(*mib), &info, &size, NULL, 0);
+ if (result_sysctl == 0) {
+#if defined (__APPLE__)
+ struct timeval proc_starttime = info.kp_proc.p_starttime;
+#else // __FreeBSD__
+ struct timeval proc_starttime = info.ki_start;
+#endif
+ long seconds_since_epoch = proc_starttime.tv_sec;
+ *key = seconds_since_epoch;
+ return true;
+ } else {
+ EP_ASSERT (!"Failed to get start time of a process.");
+ return false;
+ }
+#elif defined (__NetBSD__)
+ // On NetBSD, we return the process start time expressed in Unix time (the number of seconds
+ // since the start of the Unix epoch).
+ kvm_t *kd;
+ int cnt;
+ struct kinfo_proc2 *info;
+
+ kd = kvm_open (NULL, NULL, NULL, KVM_NO_FILES, "kvm_open");
+ if (!kd) {
+ EP_ASSERT (!"Failed to get start time of a process.");
+ return false;
+ }
+
+ info = kvm_getproc2 (kd, KERN_PROC_PID, process_id, sizeof (struct kinfo_proc2), &cnt);
+ if (!info || cnt < 1) {
+ kvm_close (kd);
+ EP_ASSERT (!"Failed to get start time of a process.");
+ return false;
+ }
+
+ kvm_close (kd);
+
+ long seconds_since_epoch = info->p_ustart_sec;
+ *key = seconds_since_epoch;
+
+ return true;
+#elif defined (__linux__)
// Here we read /proc//stat file to get the start time for the process.
// We return this value (which is expressed in jiffies since boot time).
diff --git a/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.h b/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.h
index aaea67334eb0f4..9a56a52060042c 100644
--- a/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.h
+++ b/src/coreclr/nativeaot/Runtime/eventpipe/ds-rt-aot.h
@@ -13,6 +13,8 @@
#include
#include
+#include
+
#undef DS_LOG_ALWAYS_0
#define DS_LOG_ALWAYS_0(msg) do {} while (0)
@@ -127,8 +129,10 @@ ds_rt_config_value_get_enable (void)
{
STATIC_CONTRACT_NOTHROW;
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: EventPipe Configuration values - RhConfig?
+ bool value;
+ if (RhConfig::Environment::TryGetBooleanValue("EnableDiagnostics", &value))
+ return value;
+
return true;
}
@@ -137,8 +141,12 @@ inline
ep_char8_t *
ds_rt_config_value_get_ports (void)
{
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: EventPipe Configuration values - RhConfig?
+ STATIC_CONTRACT_NOTHROW;
+
+ char* value;
+ if (RhConfig::Environment::TryGetStringValue("DiagnosticPorts", &value))
+ return (ep_char8_t*)value;
+
return nullptr;
}
@@ -148,8 +156,14 @@ uint32_t
ds_rt_config_value_get_default_port_suspend (void)
{
STATIC_CONTRACT_NOTHROW;
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: EventPipe Configuration values - RhConfig?
+
+ uint64_t value;
+ if (RhConfig::Environment::TryGetIntegerValue("DefaultDiagnosticPortSuspend", &value))
+ {
+ EP_ASSERT(value <= UINT32_MAX);
+ return static_cast(value);
+ }
+
return 0;
}
@@ -281,10 +295,15 @@ ds_rt_server_log_pause_message (void)
{
STATIC_CONTRACT_NOTHROW;
- const char diagPortsName[] = "DOTNET_DiagnosticPorts";
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: Cannot find nocache versions of RhConfig
- // PalDebugBreak();
+ ep_char8_t * ports = ds_rt_config_value_get_ports ();
+ uint32_t port_suspended = ds_rt_config_value_get_default_port_suspend ();
+
+ printf ("The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.\n");
+ printf ("DOTNET_DiagnosticPorts=\"%s\"\n", ports == nullptr ? "" : ports);
+ printf ("DOTNET_DefaultDiagnosticPortSuspend=%d\n", port_suspended);
+ fflush (stdout);
+
+ ep_rt_utf8_string_free (ports);
}
#endif /* ENABLE_PERFTRACING */
diff --git a/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.cpp b/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.cpp
index c629f2b86c5eb6..f7793b4065f535 100644
--- a/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.cpp
+++ b/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.cpp
@@ -90,13 +90,22 @@ ep_rt_aot_entrypoint_assembly_name_get_utf8 (void)
{
// shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
// TODO: Implement EventPipe assembly name - return filename in nativeaot?
- PalDebugBreak();
-
- // fallback to the empty string if we can't get assembly info, e.g., if the runtime is
- // suspended before an assembly is loaded.
return reinterpret_cast("");
}
+const ep_char8_t *
+ep_rt_aot_diagnostics_command_line_get (void)
+{
+ // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
+ // TODO: revisit commandline for AOT
+#ifdef TARGET_WINDOWS
+ const ep_char16_t* command_line = reinterpret_cast(::GetCommandLineW());
+ return ep_rt_utf16_to_utf8_string(command_line, -1);
+#else
+ return "";
+#endif
+}
+
uint32_t
ep_rt_aot_atomic_inc_uint32_t (volatile uint32_t *value)
{
diff --git a/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h b/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h
index d17566514a379f..2687fc8f35ed50 100644
--- a/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h
+++ b/src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h
@@ -20,6 +20,7 @@
#include "rhassert.h"
#include
+#include
#ifdef TARGET_UNIX
#define sprintf_s snprintf
@@ -47,6 +48,9 @@
#undef EP_ALIGN_UP
#define EP_ALIGN_UP(val,align) _rt_aot_align_up(val,align)
+#define _TEXT(s) #s
+#define STRINGIFY(s) _TEXT(s)
+
#ifdef TARGET_UNIX
extern pthread_key_t eventpipe_tls_key;
extern __thread EventPipeThreadHolder* eventpipe_tls_instance;
@@ -88,12 +92,11 @@ ep_rt_entrypoint_assembly_name_get_utf8 (void)
static
const ep_char8_t *
-ep_rt_runtime_version_get_utf8 (void) {
+ep_rt_runtime_version_get_utf8 (void)
+{
STATIC_CONTRACT_NOTHROW;
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: Find a way to use CoreCLR runtime_version.h here if a more exact version is needed
- return reinterpret_cast("8.0.0");
+ return reinterpret_cast(STRINGIFY(RuntimeProductVersion));
}
/*
@@ -1388,10 +1391,7 @@ ep_rt_utf8_to_utf16le_string (
// Shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
// Implementation would just use strlen and malloc to make a new buffer, and would then copy the string chars one by one.
// Assumes that only ASCII is used for ep_char8_t
- size_t len_utf8 = strlen(str);
- if (len_utf8 == 0)
- return NULL;
-
+ size_t len_utf8 = strlen(str);
ep_char16_t *str_utf16 = reinterpret_cast(malloc ((len_utf8 + 1) * sizeof (ep_char16_t)));
if (!str_utf16)
return NULL;
@@ -1511,16 +1511,9 @@ static
const ep_char8_t *
ep_rt_diagnostics_command_line_get (void)
{
-
STATIC_CONTRACT_NOTHROW;
-
- // shipping criteria: no EVENTPIPE-NATIVEAOT-TODO left in the codebase
- // TODO: revisit commandline for AOT
- // return reinterpret_cast(::GetCommandLineA());
-
- extern ep_char8_t *volatile _ep_rt_aot_diagnostics_cmd_line;
- ep_char8_t *old_cmd_line = _ep_rt_aot_diagnostics_cmd_line;
- return _ep_rt_aot_diagnostics_cmd_line;
+ extern const ep_char8_t * ep_rt_aot_diagnostics_command_line_get (void);
+ return ep_rt_aot_diagnostics_command_line_get();
}
/*
diff --git a/src/coreclr/nativeaot/Runtime/i386/CallDescrWorker.asm b/src/coreclr/nativeaot/Runtime/i386/CallDescrWorker.asm
deleted file mode 100644
index 481cba1d898863..00000000000000
--- a/src/coreclr/nativeaot/Runtime/i386/CallDescrWorker.asm
+++ /dev/null
@@ -1,96 +0,0 @@
-;; Licensed to the .NET Foundation under one or more agreements.
-;; The .NET Foundation licenses this file to you under the MIT license.
-
- .586
- .model flat
- option casemap:none
- .code
-
-include AsmMacros.inc
-
-ifdef FEATURE_DYNAMIC_CODE
-;;;;;;;;;;;;;;;;;;;;;;; CallingConventionConverter Thunks Helpers ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-;------------------------------------------------------------------------------
-; This helper routine enregisters the appropriate arguments and makes the
-; actual call.
-;------------------------------------------------------------------------------
-; void __fastcall CallDescrWorker(CallDescrWorkerParams * pParams)
-FASTCALL_FUNC RhCallDescrWorker, 4
- push ebp
- mov ebp, esp
- push ebx
- mov ebx, ecx
-
- mov ecx, [ebx + OFFSETOF__CallDescrData__numStackSlots]
- mov eax, [ebx + OFFSETOF__CallDescrData__pSrc] ; copy the stack
- test ecx, ecx
- jz donestack
- lea eax, [eax + 4 * ecx - 4] ; last argument
- push dword ptr [eax]
- dec ecx
- jz donestack
- sub eax, 4
- push dword ptr [eax]
- dec ecx
- jz donestack
-stackloop:
- sub eax, 4
- push dword ptr [eax]
- dec ecx
- jnz stackloop
-donestack:
-
- ; now we must push each field of the ArgumentRegister structure
- mov eax, [ebx + OFFSETOF__CallDescrData__pArgumentRegisters]
- mov edx, dword ptr [eax]
- mov ecx, dword ptr [eax + 4]
- mov eax,[ebx + OFFSETOF__CallDescrData__pTarget]
- call eax
-
- EXPORT_POINTER_TO_ADDRESS _PointerToReturnFromCallDescrThunk
-
- ; Symbol used to identify thunk call to managed function so the special
- ; case unwinder can unwind through this function. Sadly we cannot directly
- ; export this symbol right now because it confuses DIA unwinder to believe
- ; it's the beginning of a new method, therefore we export the address
- ; by means of an auxiliary variable.
-
- ; Save FP return value if necessary
- mov ecx, [ebx + OFFSETOF__CallDescrData__fpReturnSize]
- cmp ecx, 0
- je ReturnsInt
-
- cmp ecx, 4
- je ReturnsFloat
- cmp ecx, 8
- je ReturnsDouble
- ; unexpected
- jmp Epilog
-
-ReturnsInt:
-; Unlike desktop returnValue is a pointer to a return buffer, not the buffer itself
- mov ebx, [ebx + OFFSETOF__CallDescrData__pReturnBuffer]
- mov [ebx], eax
- mov [ebx + 4], edx
-
-Epilog:
- pop ebx
- pop ebp
- retn
-
-ReturnsFloat:
- mov ebx, [ebx + OFFSETOF__CallDescrData__pReturnBuffer]
- fstp dword ptr [ebx] ; Spill the Float return value
- jmp Epilog
-
-ReturnsDouble:
- mov ebx, [ebx + OFFSETOF__CallDescrData__pReturnBuffer]
- fstp qword ptr [ebx] ; Spill the Double return value
- jmp Epilog
-
-FASTCALL_ENDFUNC
-
-endif
-
-end
diff --git a/src/coreclr/nativeaot/Runtime/portable.cpp b/src/coreclr/nativeaot/Runtime/portable.cpp
index c8f43790085b9e..25eb9485fde57c 100644
--- a/src/coreclr/nativeaot/Runtime/portable.cpp
+++ b/src/coreclr/nativeaot/Runtime/portable.cpp
@@ -394,12 +394,6 @@ void * ReturnFromUniversalTransition_DebugStepTailCall;
#endif // USE_PORTABLE_HELPERS
-// @TODO Implement CallDescrThunk
-EXTERN_C void * ReturnFromCallDescrThunk;
-#ifdef USE_PORTABLE_HELPERS
-void * ReturnFromCallDescrThunk;
-#endif
-
#if defined(USE_PORTABLE_HELPERS)
//
// Return address hijacking
@@ -462,14 +456,6 @@ COOP_PINVOKE_HELPER(int64_t, RhpLockCmpXchg64, (int64_t * location, int64_t valu
return PalInterlockedCompareExchange64(location, value, comparand);
}
-#endif // USE_PORTABLE_HELPERS
-
-COOP_PINVOKE_HELPER(void, RhpMemoryBarrier, ())
-{
- PalMemoryBarrier();
-}
-
-#if defined(USE_PORTABLE_HELPERS)
EXTERN_C NATIVEAOT_API void* __cdecl RhAllocateThunksMapping()
{
return NULL;
@@ -516,11 +502,6 @@ COOP_PINVOKE_HELPER(int, RhpGetThunkBlockSize, ())
return 0;
}
-COOP_PINVOKE_HELPER(void, RhCallDescrWorker, (void * callDescr))
-{
- ASSERT_UNCONDITIONALLY("NYI");
-}
-
COOP_PINVOKE_HELPER(void *, RhGetCommonStubAddress, ())
{
ASSERT_UNCONDITIONALLY("NYI");
diff --git a/src/coreclr/nativeaot/Runtime/startup.cpp b/src/coreclr/nativeaot/Runtime/startup.cpp
index a178a11d102fae..58740157c49609 100644
--- a/src/coreclr/nativeaot/Runtime/startup.cpp
+++ b/src/coreclr/nativeaot/Runtime/startup.cpp
@@ -197,51 +197,59 @@ bool DetectCPUFeatures()
{
__cpuid(cpuidInfo, 0x00000001);
- if (((cpuidInfo[CPUID_EDX] & (1 << 25)) != 0) && ((cpuidInfo[CPUID_EDX] & (1 << 26)) != 0)) // SSE & SSE2
+ const int requiredBaselineEdxFlags = (1 << 25) // SSE
+ | (1 << 26); // SSE2
+
+ if ((cpuidInfo[CPUID_EDX] & requiredBaselineEdxFlags) == requiredBaselineEdxFlags)
{
- if ((cpuidInfo[CPUID_ECX] & (1 << 25)) != 0) // AESNI
+ g_cpuFeatures |= XArchIntrinsicConstants_VectorT128;
+
+ if ((cpuidInfo[CPUID_ECX] & (1 << 25)) != 0) // AESNI
{
g_cpuFeatures |= XArchIntrinsicConstants_Aes;
}
- if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // PCLMULQDQ
+ if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // PCLMULQDQ
{
g_cpuFeatures |= XArchIntrinsicConstants_Pclmulqdq;
}
- if ((cpuidInfo[CPUID_ECX] & (1 << 0)) != 0) // SSE3
+ if ((cpuidInfo[CPUID_ECX] & (1 << 0)) != 0) // SSE3
{
g_cpuFeatures |= XArchIntrinsicConstants_Sse3;
- if ((cpuidInfo[CPUID_ECX] & (1 << 9)) != 0) // SSSE3
+ if ((cpuidInfo[CPUID_ECX] & (1 << 9)) != 0) // SSSE3
{
g_cpuFeatures |= XArchIntrinsicConstants_Ssse3;
- if ((cpuidInfo[CPUID_ECX] & (1 << 19)) != 0) // SSE4.1
+ if ((cpuidInfo[CPUID_ECX] & (1 << 19)) != 0) // SSE4.1
{
g_cpuFeatures |= XArchIntrinsicConstants_Sse41;
- if ((cpuidInfo[CPUID_ECX] & (1 << 20)) != 0) // SSE4.2
+ if ((cpuidInfo[CPUID_ECX] & (1 << 20)) != 0) // SSE4.2
{
g_cpuFeatures |= XArchIntrinsicConstants_Sse42;
- if ((cpuidInfo[CPUID_ECX] & (1 << 22)) != 0) // MOVBE
+ if ((cpuidInfo[CPUID_ECX] & (1 << 22)) != 0) // MOVBE
{
g_cpuFeatures |= XArchIntrinsicConstants_Movbe;
}
- if ((cpuidInfo[CPUID_ECX] & (1 << 23)) != 0) // POPCNT
+ if ((cpuidInfo[CPUID_ECX] & (1 << 23)) != 0) // POPCNT
{
g_cpuFeatures |= XArchIntrinsicConstants_Popcnt;
}
- if (((cpuidInfo[CPUID_ECX] & (1 << 27)) != 0) && ((cpuidInfo[CPUID_ECX] & (1 << 28)) != 0)) // OSXSAVE & AVX
+ const int requiredAvxEcxFlags = (1 << 27) // OSXSAVE
+ | (1 << 28); // AVX
+
+ if ((cpuidInfo[CPUID_ECX] & requiredAvxEcxFlags) == requiredAvxEcxFlags)
{
- if (PalIsAvxEnabled() && (xmmYmmStateSupport() == 1))
+ if (PalIsAvxEnabled() && (xmmYmmStateSupport() == 1)) // XGETBV == 11
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx;
- if ((cpuidInfo[CPUID_ECX] & (1 << 12)) != 0) // FMA
+ if ((cpuidInfo[CPUID_ECX] & (1 << 12)) != 0) // FMA
{
g_cpuFeatures |= XArchIntrinsicConstants_Fma;
}
@@ -250,66 +258,69 @@ bool DetectCPUFeatures()
{
__cpuidex(cpuidInfo, 0x00000007, 0x00000000);
- if ((cpuidInfo[CPUID_EBX] & (1 << 5)) != 0) // AVX2
+ if ((cpuidInfo[CPUID_EBX] & (1 << 5)) != 0) // AVX2
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx2;
+ g_cpuFeatures |= XArchIntrinsicConstants_VectorT256;
- __cpuidex(cpuidInfo, 0x00000007, 0x00000001);
- if ((cpuidInfo[CPUID_EAX] & (1 << 4)) != 0) // AVX-VNNI
+ if (PalIsAvx512Enabled() && (avx512StateSupport() == 1)) // XGETBV XRC0[7:5] == 111
{
- g_cpuFeatures |= XArchIntrinsicConstants_AvxVnni;
- }
-
- if (PalIsAvx512Enabled() && (avx512StateSupport() == 1)) // XGETBV XRC0[7:5] == 111
- {
- if ((cpuidInfo[CPUID_EBX] & (1 << 16)) != 0) // AVX512F
+ if ((cpuidInfo[CPUID_EBX] & (1 << 16)) != 0) // AVX512F
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512f;
+ g_cpuFeatures |= XArchIntrinsicConstants_VectorT512;
bool isAVX512_VLSupported = false;
- if ((cpuidInfo[CPUID_EBX] & (1 << 31)) != 0) // AVX512VL
+ if ((cpuidInfo[CPUID_EBX] & (1 << 31)) != 0) // AVX512VL
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512f_vl;
isAVX512_VLSupported = true;
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 30)) != 0) // AVX512BW
+ if ((cpuidInfo[CPUID_EBX] & (1 << 30)) != 0) // AVX512BW
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512bw;
- if (isAVX512_VLSupported)
+ if (isAVX512_VLSupported) // AVX512BW_VL
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512bw_vl;
}
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 28)) != 0) // AVX512CD
+ if ((cpuidInfo[CPUID_EBX] & (1 << 28)) != 0) // AVX512CD
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512cd;
- if (isAVX512_VLSupported)
+ if (isAVX512_VLSupported) // AVX512CD_VL
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512cd_vl;
}
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 17)) != 0) // AVX512DQ
+ if ((cpuidInfo[CPUID_EBX] & (1 << 17)) != 0) // AVX512DQ
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512dq;
- if (isAVX512_VLSupported)
+ if (isAVX512_VLSupported) // AVX512DQ_VL
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512dq_vl;
}
}
- if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // AVX512VBMI
+ if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // AVX512VBMI
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512Vbmi;
- if (isAVX512_VLSupported)
+ if (isAVX512_VLSupported) // AVX512VBMI_VL
{
g_cpuFeatures |= XArchIntrinsicConstants_Avx512Vbmi_vl;
}
}
}
}
+
+ __cpuidex(cpuidInfo, 0x00000007, 0x00000001);
+
+ if ((cpuidInfo[CPUID_EAX] & (1 << 4)) != 0) // AVX-VNNI
+ {
+ g_cpuFeatures |= XArchIntrinsicConstants_AvxVnni;
+ }
}
}
}
@@ -333,6 +344,11 @@ bool DetectCPUFeatures()
{
g_cpuFeatures |= XArchIntrinsicConstants_Bmi2;
}
+
+ if ((cpuidInfo[CPUID_EDX] & (1 << 14)) != 0)
+ {
+ g_cpuFeatures |= XArchIntrinsicConstants_Serialize; // SERIALIZE
+ }
}
}
diff --git a/src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp b/src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp
index a856be48f4ab8c..bbe70ae48cd6c0 100644
--- a/src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp
+++ b/src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp
@@ -892,12 +892,6 @@ REDHAWK_PALEXPORT void PalFlushInstructionCache(_In_ void* pAddress, size_t size
#endif
}
-REDHAWK_PALEXPORT _Ret_maybenull_ void* REDHAWK_PALAPI PalSetWerDataBuffer(_In_ void* pNewBuffer)
-{
- static void* pBuffer;
- return PalInterlockedExchangePointer(&pBuffer, pNewBuffer);
-}
-
extern "C" HANDLE GetCurrentProcess()
{
return (HANDLE)-1;
@@ -1446,7 +1440,7 @@ REDHAWK_PALEXPORT void REDHAWK_PALAPI PAL_GetCpuCapabilityFlags(int* flags)
#endif
#ifdef HWCAP_ASIMD
if (hwCap & HWCAP_ASIMD)
- *flags |= ARM64IntrinsicConstants_AdvSimd;
+ *flags |= ARM64IntrinsicConstants_AdvSimd | ARM64IntrinsicConstants_VectorT128;
#endif
#ifdef HWCAP_ASIMDRDM
if (hwCap & HWCAP_ASIMDRDM)
@@ -1545,7 +1539,7 @@ REDHAWK_PALEXPORT void REDHAWK_PALAPI PAL_GetCpuCapabilityFlags(int* flags)
// Every ARM64 CPU should support SIMD and FP
// If the OS have no function to query for CPU capabilities we set just these
- *flags |= ARM64IntrinsicConstants_AdvSimd;
+ *flags |= ARM64IntrinsicConstants_AdvSimd | ARM64IntrinsicConstants_VectorT128;
#endif // HAVE_AUXV_HWCAP_H
}
#endif
diff --git a/src/coreclr/nativeaot/Runtime/windows/CoffNativeCodeManager.cpp b/src/coreclr/nativeaot/Runtime/windows/CoffNativeCodeManager.cpp
index 96cc1e3934ef45..cc409cf2167a16 100644
--- a/src/coreclr/nativeaot/Runtime/windows/CoffNativeCodeManager.cpp
+++ b/src/coreclr/nativeaot/Runtime/windows/CoffNativeCodeManager.cpp
@@ -590,9 +590,9 @@ bool CoffNativeCodeManager::UnwindStackFrame(MethodInfo * pMethodInfo,
#define WORDPTR PDWORD
#elif defined(TARGET_AMD64)
#define FOR_EACH_NONVOLATILE_REGISTER(F) \
- F(Rax, pRax) F(Rcx, pRcx) F(Rdx, pRdx) F(Rbx, pRbx) F(Rbp, pRbp) F(Rsi, pRsi) F(Rdi, pRdi) \
- F(R8, pR8) F(R9, pR9) F(R10, pR10) F(R11, pR11) F(R12, pR12) F(R13, pR13) F(R14, pR14) F(R15, pR15)
- #define WORDPTR PDWORD64
+ F(Rbx, pRbx) F(Rbp, pRbp) F(Rsi, pRsi) F(Rdi, pRdi) \
+ F(R12, pR12) F(R13, pR13) F(R14, pR14) F(R15, pR15)
+#define WORDPTR PDWORD64
#elif defined(TARGET_ARM64)
#define FOR_EACH_NONVOLATILE_REGISTER(F) \
F(X19, pX19) F(X20, pX20) F(X21, pX21) F(X22, pX22) F(X23, pX23) F(X24, pX24) \
@@ -612,7 +612,11 @@ bool CoffNativeCodeManager::UnwindStackFrame(MethodInfo * pMethodInfo,
#if defined(TARGET_X86)
PORTABILITY_ASSERT("CoffNativeCodeManager::UnwindStackFrame");
#elif defined(TARGET_AMD64)
- memcpy(&context.Xmm6, pRegisterSet->Xmm, sizeof(pRegisterSet->Xmm));
+
+ if (!(flags & USFF_GcUnwind))
+ {
+ memcpy(&context.Xmm6, pRegisterSet->Xmm, sizeof(pRegisterSet->Xmm));
+ }
context.Rsp = pRegisterSet->SP;
context.Rip = pRegisterSet->IP;
@@ -634,10 +638,16 @@ bool CoffNativeCodeManager::UnwindStackFrame(MethodInfo * pMethodInfo,
pRegisterSet->pIP = PTR_PCODE(pRegisterSet->SP - sizeof(TADDR));
- memcpy(pRegisterSet->Xmm, &context.Xmm6, sizeof(pRegisterSet->Xmm));
+ if (!(flags & USFF_GcUnwind))
+ {
+ memcpy(pRegisterSet->Xmm, &context.Xmm6, sizeof(pRegisterSet->Xmm));
+ }
#elif defined(TARGET_ARM64)
- for (int i = 8; i < 16; i++)
- context.V[i].Low = pRegisterSet->D[i - 8];
+ if (!(flags & USFF_GcUnwind))
+ {
+ for (int i = 8; i < 16; i++)
+ context.V[i].Low = pRegisterSet->D[i - 8];
+ }
context.Sp = pRegisterSet->SP;
context.Pc = pRegisterSet->IP;
@@ -659,8 +669,11 @@ bool CoffNativeCodeManager::UnwindStackFrame(MethodInfo * pMethodInfo,
pRegisterSet->pIP = contextPointers.Lr;
- for (int i = 8; i < 16; i++)
- pRegisterSet->D[i - 8] = context.V[i].Low;
+ if (!(flags & USFF_GcUnwind))
+ {
+ for (int i = 8; i < 16; i++)
+ pRegisterSet->D[i - 8] = context.V[i].Low;
+ }
#endif // defined(TARGET_X86)
FOR_EACH_NONVOLATILE_REGISTER(CONTEXT_TO_REGDISPLAY);
diff --git a/src/coreclr/nativeaot/Runtime/windows/PalRedhawkInline.h b/src/coreclr/nativeaot/Runtime/windows/PalRedhawkInline.h
index c6ed7d641a20bf..1d9f7bd1711d62 100644
--- a/src/coreclr/nativeaot/Runtime/windows/PalRedhawkInline.h
+++ b/src/coreclr/nativeaot/Runtime/windows/PalRedhawkInline.h
@@ -1,6 +1,10 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+#if defined(HOST_ARM64)
+#include
+#endif
+
// Implementation of Redhawk PAL inline functions
EXTERN_C long __cdecl _InterlockedIncrement(long volatile *);
@@ -121,7 +125,6 @@ EXTERN_C void __faststorefence();
#pragma intrinsic(__faststorefence)
#define PalMemoryBarrier() __faststorefence()
-
#elif defined(HOST_ARM)
EXTERN_C void __yield(void);
@@ -130,11 +133,11 @@ EXTERN_C void __dmb(unsigned int _Type);
#pragma intrinsic(__dmb)
FORCEINLINE void PalYieldProcessor()
{
- __dmb(0xA /* _ARM_BARRIER_ISHST */);
+ __dmb(_ARM_BARRIER_ISHST);
__yield();
}
-#define PalMemoryBarrier() __dmb(0xF /* _ARM_BARRIER_SY */)
+#define PalMemoryBarrier() __dmb(_ARM_BARRIER_ISH)
#elif defined(HOST_ARM64)
@@ -144,11 +147,11 @@ EXTERN_C void __dmb(unsigned int _Type);
#pragma intrinsic(__dmb)
FORCEINLINE void PalYieldProcessor()
{
- __dmb(0xA /* _ARM64_BARRIER_ISHST */);
+ __dmb(_ARM64_BARRIER_ISHST);
__yield();
}
-#define PalMemoryBarrier() __dmb(0xF /* _ARM64_BARRIER_SY */)
+#define PalMemoryBarrier() __dmb(_ARM64_BARRIER_ISH)
#else
#error Unsupported architecture
diff --git a/src/coreclr/nativeaot/Runtime/windows/PalRedhawkMinWin.cpp b/src/coreclr/nativeaot/Runtime/windows/PalRedhawkMinWin.cpp
index c7b1f3e313fa39..6691df67ada460 100644
--- a/src/coreclr/nativeaot/Runtime/windows/PalRedhawkMinWin.cpp
+++ b/src/coreclr/nativeaot/Runtime/windows/PalRedhawkMinWin.cpp
@@ -769,12 +769,6 @@ REDHAWK_PALEXPORT void PalFlushInstructionCache(_In_ void* pAddress, size_t size
FlushInstructionCache(GetCurrentProcess(), pAddress, size);
}
-REDHAWK_PALEXPORT _Ret_maybenull_ void* REDHAWK_PALAPI PalSetWerDataBuffer(_In_ void* pNewBuffer)
-{
- static void* pBuffer;
- return InterlockedExchangePointer(&pBuffer, pNewBuffer);
-}
-
#if defined(HOST_ARM64)
#include "IntrinsicConstants.h"
@@ -796,7 +790,7 @@ REDHAWK_PALIMPORT void REDHAWK_PALAPI PAL_GetCpuCapabilityFlags(int* flags)
#endif
// FP and SIMD support are enabled by default
- *flags |= ARM64IntrinsicConstants_AdvSimd;
+ *flags |= ARM64IntrinsicConstants_AdvSimd | ARM64IntrinsicConstants_VectorT128;
if (IsProcessorFeaturePresent(PF_ARM_V8_CRYPTO_INSTRUCTIONS_AVAILABLE))
{
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/Internal/Runtime/Augments/RuntimeAugments.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/Internal/Runtime/Augments/RuntimeAugments.cs
index 9d46a80d656aac..39fda81c8a8388 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/Internal/Runtime/Augments/RuntimeAugments.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/Internal/Runtime/Augments/RuntimeAugments.cs
@@ -195,9 +195,9 @@ public static IntPtr GetPointerFromTypeHandle(RuntimeTypeHandle typeHandle)
return typeHandle.ToEETypePtr().RawValue;
}
- public static TypeManagerHandle GetModuleFromTypeHandle(RuntimeTypeHandle typeHandle)
+ public static unsafe TypeManagerHandle GetModuleFromTypeHandle(RuntimeTypeHandle typeHandle)
{
- return RuntimeImports.RhGetModuleFromEEType(GetPointerFromTypeHandle(typeHandle));
+ return typeHandle.ToMethodTable()->TypeManager;
}
public static RuntimeTypeHandle CreateRuntimeTypeHandle(IntPtr ldTokenResult)
@@ -817,21 +817,6 @@ public static string TryGetMethodDisplayStringFromIp(IntPtr ip)
private static volatile ReflectionExecutionDomainCallbacks s_reflectionExecutionDomainCallbacks;
private static TypeLoaderCallbacks s_typeLoaderCallbacks;
- public static void ReportUnhandledException(Exception exception)
- {
- RuntimeExceptionHelpers.ReportUnhandledException(exception);
- }
-
- public static unsafe RuntimeTypeHandle GetRuntimeTypeHandleFromObjectReference(object obj)
- {
- return new RuntimeTypeHandle(obj.GetEETypePtr());
- }
-
- public static IntPtr GetUniversalTransitionThunk()
- {
- return RuntimeImports.RhGetUniversalTransitionThunk();
- }
-
public static object CreateThunksHeap(IntPtr commonStubAddress)
{
object newHeap = RuntimeImports.RhCreateThunksHeap(commonStubAddress);
@@ -868,22 +853,6 @@ public static int GetThunkSize()
return RuntimeImports.RhGetThunkSize();
}
- [DebuggerStepThrough]
- /* TEMP workaround due to bug 149078 */
- [MethodImpl(MethodImplOptions.NoInlining)]
- public static void CallDescrWorker(IntPtr callDescr)
- {
- RuntimeImports.RhCallDescrWorker(callDescr);
- }
-
- [DebuggerStepThrough]
- /* TEMP workaround due to bug 149078 */
- [MethodImpl(MethodImplOptions.NoInlining)]
- public static void CallDescrWorkerNative(IntPtr callDescr)
- {
- RuntimeImports.RhCallDescrWorkerNative(callDescr);
- }
-
public static Delegate CreateObjectArrayDelegate(Type delegateType, Func invoker)
{
return Delegate.CreateObjectArrayDelegate(delegateType, invoker);
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/AppContext.NativeAot.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/AppContext.NativeAot.cs
index 830ae290d5a2c9..535eb123ddf0f3 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/AppContext.NativeAot.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/AppContext.NativeAot.cs
@@ -1,13 +1,30 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+using System.Collections.Generic;
using System.Runtime;
using System.Runtime.ExceptionServices;
+using System.Text;
namespace System
{
public static partial class AppContext
{
+ private static unsafe Dictionary InitializeDataStore()
+ {
+ uint count = RuntimeImports.RhGetKnobValues(out byte** keys, out byte** values);
+
+ var dataStore = new Dictionary((int)count);
+ for (int i = 0; i < count; i++)
+ {
+ dataStore.Add(
+ Encoding.UTF8.GetString(keys[i], string.strlen(keys[i])),
+ Encoding.UTF8.GetString(values[i], string.strlen(values[i])));
+ }
+
+ return dataStore;
+ }
+
[RuntimeExport("OnFirstChanceException")]
internal static void OnFirstChanceException(object e)
{
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Attribute.NativeAot.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Attribute.NativeAot.cs
index f7604612b5ad5e..fe6505f24c9c90 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Attribute.NativeAot.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Attribute.NativeAot.cs
@@ -121,7 +121,7 @@ private static Attribute OneOrNull(IEnumerable results)
return null;
CustomAttributeData result = enumerator.Current;
if (enumerator.MoveNext())
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(result);
return result.Instantiate();
}
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/BindingFlagSupport/QueryResult.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/BindingFlagSupport/QueryResult.cs
index 301fe25f6fa389..6288b58c405217 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/BindingFlagSupport/QueryResult.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/BindingFlagSupport/QueryResult.cs
@@ -112,11 +112,10 @@ public void CopyTo(MemberInfo[] array, int startIndex)
// Assuming the policy says it's ok to ignore the ambiguity, we're to resolve in favor of the member
// declared by the most derived type. Since QueriedMemberLists are sorted in order of decreasing derivation,
// that means we let the first match win - unless, of course, they're both the "most derived member".
- if (match.DeclaringType.Equals(challenger.DeclaringType))
- throw new AmbiguousMatchException();
-
- if (!_policies.OkToIgnoreAmbiguity(match, challenger))
- throw new AmbiguousMatchException();
+ // If they're not from same type, we throw if the policy doesn't allow ambiguity.
+ if (match.DeclaringType.Equals(challenger.DeclaringType) ||
+ !_policies.OkToIgnoreAmbiguity(match, challenger))
+ throw ThrowHelper.GetAmbiguousMatchException(match);
}
else
{
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs
index 3e0765205c6373..99858e0bbaa784 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs
@@ -199,7 +199,7 @@ public sealed override Type[] GetGenericArguments()
if (ns != null && !ns.Equals(ifc.Namespace))
continue;
if (match != null)
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(match);
match = ifc;
}
return match;
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/ParameterInfos/RuntimeThinMethodParameterInfo.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/ParameterInfos/RuntimeThinMethodParameterInfo.cs
index b1c543238de156..a93368402a4bc2 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/ParameterInfos/RuntimeThinMethodParameterInfo.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/ParameterInfos/RuntimeThinMethodParameterInfo.cs
@@ -62,8 +62,7 @@ public sealed override bool HasDefaultValue
{
get
{
- // Compat: returning "true" makes no sense but this is how it's always been.
- return true;
+ return false;
}
}
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/RuntimeTypeInfo.BindingFlags.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/RuntimeTypeInfo.BindingFlags.cs
index dcb0ed3e093320..35ae4a96ea7c28 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/RuntimeTypeInfo.BindingFlags.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/TypeInfos/RuntimeTypeInfo.BindingFlags.cs
@@ -155,9 +155,10 @@ protected sealed override PropertyInfo GetPropertyImpl(string name, BindingFlags
if (types == null || types.Length == 0)
{
// no arguments
+ PropertyInfo firstCandidate = candidates[0];
+
if (candidates.Count == 1)
{
- PropertyInfo firstCandidate = candidates[0];
if (returnType is not null && !returnType.IsEquivalentTo(firstCandidate.PropertyType))
return null;
return firstCandidate;
@@ -165,8 +166,10 @@ protected sealed override PropertyInfo GetPropertyImpl(string name, BindingFlags
else
{
if (returnType is null)
+ {
// if we are here we have no args or property type to select over and we have more than one property with that name
- throw new AmbiguousMatchException();
+ throw ThrowHelper.GetAmbiguousMatchException(firstCandidate);
+ }
}
}
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.Com.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.Com.cs
index 17e1bdb85ddfcb..1ac3410c8afa04 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.Com.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.Com.cs
@@ -50,6 +50,7 @@ public static IntPtr CreateAggregatedObject(IntPtr pOuter, T o) where T : not
[SupportedOSPlatform("windows")]
[EditorBrowsable(EditorBrowsableState.Never)]
+ [return: NotNullIfNotNull(nameof(o))]
public static object? CreateWrapperOfType(object? o, Type t)
{
throw new NotSupportedException(SR.PlatformNotSupported_ComInterop);
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs
index 61f8c38b6cb93b..68b50d5bb966ee 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs
@@ -351,9 +351,6 @@ internal static unsafe object IsInstanceOf(EETypePtr pTargetType, object obj)
[RuntimeImport(RuntimeLibrary, "RhBoxAny")]
internal static extern unsafe object RhBoxAny(ref byte pData, MethodTable* pEEType);
- internal static unsafe object RhBoxAny(ref byte pData, EETypePtr pEEType)
- => RhBoxAny(ref pData, pEEType.ToPointer());
-
[MethodImpl(MethodImplOptions.InternalCall)]
[RuntimeImport(RuntimeLibrary, "RhNewObject")]
internal static extern unsafe object RhNewObject(MethodTable* pEEType);
@@ -473,8 +470,6 @@ internal static unsafe int RhCompatibleReentrantWaitAny(bool alertable, int time
[MethodImplAttribute(MethodImplOptions.InternalCall)]
[RuntimeImport(RuntimeLibrary, "RhResolveDispatchOnType")]
- // For my life cannot figure out the ordering of modifiers this is expecting.
-#pragma warning disable IDE0036
internal static extern unsafe IntPtr RhResolveDispatchOnType(EETypePtr instanceType, EETypePtr interfaceType, ushort slot, EETypePtr* pGenericContext);
internal static unsafe IntPtr RhResolveDispatchOnType(EETypePtr instanceType, EETypePtr interfaceType, ushort slot)
@@ -554,16 +549,12 @@ internal static IntPtr RhGetModuleSection(TypeManagerHandle module, ReadyToRunSe
internal static extern uint RhGetLoadedOSModules(IntPtr[] resultArray);
[MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhGetOSModuleFromPointer")]
- internal static extern IntPtr RhGetOSModuleFromPointer(IntPtr pointerVal);
-
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhGetModuleFromEEType")]
- internal static extern TypeManagerHandle RhGetModuleFromEEType(IntPtr pEEType);
+ [RuntimeImport(RuntimeLibrary, "RhGetKnobValues")]
+ internal static extern unsafe uint RhGetKnobValues(out byte** keyArray, out byte** valueArray);
[MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhGetOSModuleFromEEType")]
- internal static extern IntPtr RhGetOSModuleFromEEType(IntPtr pEEType);
+ [RuntimeImport(RuntimeLibrary, "RhGetOSModuleFromPointer")]
+ internal static extern IntPtr RhGetOSModuleFromPointer(IntPtr pointerVal);
[MethodImplAttribute(MethodImplOptions.InternalCall)]
[RuntimeImport(RuntimeLibrary, "RhGetThreadStaticStorage")]
@@ -616,11 +607,6 @@ internal static IntPtr RhGetModuleSection(TypeManagerHandle module, ReadyToRunSe
internal static extern unsafe int RhGetModuleFileName(IntPtr moduleHandle, out char* moduleName);
#endif
- // returns the previous value.
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhSetErrorInfoBuffer")]
- internal static extern unsafe void* RhSetErrorInfoBuffer(void* pNewBuffer);
-
//
// StackTrace helper
//
@@ -645,21 +631,6 @@ internal static IntPtr RhGetModuleSection(TypeManagerHandle module, ReadyToRunSe
[RuntimeImport(RuntimeLibrary, "RhSetThreadExitCallback")]
internal static extern unsafe void RhSetThreadExitCallback(delegate* unmanaged pCallback);
- // Functions involved in thunks from managed to managed functions (Universal transition transitions
- // from an arbitrary method call into a defined function, and CallDescrWorker goes the other way.
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhGetUniversalTransitionThunk")]
- internal static extern IntPtr RhGetUniversalTransitionThunk();
-
- // For Managed to Managed calls
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhCallDescrWorker")]
- internal static extern void RhCallDescrWorker(IntPtr callDescr);
-
- // For Managed to Native calls
- [LibraryImport(RuntimeLibrary, EntryPoint = "RhCallDescrWorker")]
- internal static partial void RhCallDescrWorkerNative(IntPtr callDescr);
-
// Moves memory from smem to dmem. Size must be a positive value.
// This copy uses an intrinsic to be safe for copying arbitrary bits of
// heap memory
@@ -791,10 +762,6 @@ internal static unsafe partial void RhEventPipeInternal_WriteEventData(
[RuntimeImport(RuntimeLibrary, "RhpCheckedXchg")]
internal static extern object InterlockedExchange([NotNullIfNotNull(nameof(value))] ref object? location1, object? value);
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhpMemoryBarrier")]
- internal static extern void MemoryBarrier();
-
[Intrinsic]
[MethodImplAttribute(MethodImplOptions.InternalCall)]
[RuntimeImport(RuntimeLibrary, "acos")]
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeTypeHandle.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeTypeHandle.cs
index 3a9e422c9558e9..e9c15296ccd88f 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeTypeHandle.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeTypeHandle.cs
@@ -16,10 +16,7 @@ namespace System
[StructLayout(LayoutKind.Sequential)]
public unsafe struct RuntimeTypeHandle : IEquatable, ISerializable
{
- //
- // Caution: There can be and are multiple MethodTable for the "same" type (e.g. int[]). That means
- // you can't use the raw IntPtr value for comparisons.
- //
+ private IntPtr _value;
internal RuntimeTypeHandle(EETypePtr pEEType)
: this(pEEType.RawValue)
@@ -123,7 +120,5 @@ internal bool IsNull
return _value == new IntPtr(0);
}
}
-
- private IntPtr _value;
}
}
diff --git a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Interlocked.cs b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Interlocked.cs
index 56f16cbcfbbe4c..f8a5e61ecbb65c 100644
--- a/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Interlocked.cs
+++ b/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Interlocked.cs
@@ -4,9 +4,6 @@
using System.Runtime;
using System.Runtime.CompilerServices;
using System.Diagnostics.CodeAnalysis;
-using System.Runtime.Versioning;
-
-using Internal.Runtime.CompilerServices;
namespace System.Threading
{
@@ -26,22 +23,6 @@ public static long CompareExchange(ref long location1, long value, long comparan
return RuntimeImports.InterlockedCompareExchange(ref location1, value, comparand);
}
- [Intrinsic]
- public static unsafe float CompareExchange(ref float location1, float value, float comparand)
- {
- float ret;
- *(int*)&ret = CompareExchange(ref Unsafe.As(ref location1), *(int*)&value, *(int*)&comparand);
- return ret;
- }
-
- [Intrinsic]
- public static unsafe double CompareExchange(ref double location1, double value, double comparand)
- {
- double ret;
- *(long*)&ret = CompareExchange(ref Unsafe.As(ref location1), *(long*)&value, *(long*)&comparand);
- return ret;
- }
-
[Intrinsic]
[MethodImpl(MethodImplOptions.AggressiveInlining)]
[return: NotNullIfNotNull(nameof(location1))]
@@ -87,22 +68,6 @@ public static long Exchange(ref long location1, long value)
return oldValue;
}
- [Intrinsic]
- public static unsafe float Exchange(ref float location1, float value)
- {
- float ret;
- *(int*)&ret = Exchange(ref Unsafe.As(ref location1), *(int*)&value);
- return ret;
- }
-
- [Intrinsic]
- public static unsafe double Exchange(ref double location1, double value)
- {
- double ret;
- *(long*)&ret = Exchange(ref Unsafe.As(ref location1), *(long*)&value);
- return ret;
- }
-
[Intrinsic]
[MethodImpl(MethodImplOptions.AggressiveInlining)]
[return: NotNullIfNotNull(nameof(location1))]
@@ -196,14 +161,6 @@ private static long ExchangeAdd(ref long location1, long value)
#endregion
- #region MemoryBarrier
- [Intrinsic]
- public static void MemoryBarrier()
- {
- RuntimeImports.MemoryBarrier();
- }
- #endregion
-
#region Read
public static long Read(ref long location)
{
diff --git a/src/coreclr/nativeaot/System.Private.Reflection.Execution/src/Internal/Reflection/Execution/ExecutionEnvironmentImplementation.MappingTables.cs b/src/coreclr/nativeaot/System.Private.Reflection.Execution/src/Internal/Reflection/Execution/ExecutionEnvironmentImplementation.MappingTables.cs
index 1da045804d9251..8f1a187c6fc0a0 100644
--- a/src/coreclr/nativeaot/System.Private.Reflection.Execution/src/Internal/Reflection/Execution/ExecutionEnvironmentImplementation.MappingTables.cs
+++ b/src/coreclr/nativeaot/System.Private.Reflection.Execution/src/Internal/Reflection/Execution/ExecutionEnvironmentImplementation.MappingTables.cs
@@ -743,7 +743,7 @@ private static FunctionPointersToOffsets ComputeLdftnReverseLookup_InvokeMap(Nat
continue;
entryParser.SkipInteger(); // entryMethodHandleOrNameAndSigRaw
- entryParser.SkipInteger(); // entryDeclaringTypeRaw
+ RuntimeTypeHandle declaringTypeHandle = externalReferences.GetRuntimeTypeHandleFromIndex(entryParser.GetUnsigned());
IntPtr entryMethodEntrypoint = externalReferences.GetFunctionPointerFromIndex(entryParser.GetUnsigned());
functionPointers.Add(new FunctionPointerOffsetPair(entryMethodEntrypoint, parserOffset));
@@ -752,17 +752,21 @@ private static FunctionPointersToOffsets ComputeLdftnReverseLookup_InvokeMap(Nat
// stack trace resolution - the reverse LdFtn lookup internally used by the reflection
// method resolution will work off an IP address on the stack which is an address
// within the actual method, not the stub.
- IntPtr targetAddress = RuntimeAugments.GetCodeTarget(entryMethodEntrypoint);
- if (targetAddress != IntPtr.Zero && targetAddress != entryMethodEntrypoint)
+ if (RuntimeAugments.IsValueType(declaringTypeHandle))
{
- functionPointers.Add(new FunctionPointerOffsetPair(targetAddress, parserOffset));
- }
- IntPtr targetAddress2;
- if (TypeLoaderEnvironment.TryGetTargetOfUnboxingAndInstantiatingStub(entryMethodEntrypoint, out targetAddress2) &&
- targetAddress2 != entryMethodEntrypoint &&
- targetAddress2 != targetAddress)
- {
- functionPointers.Add(new FunctionPointerOffsetPair(targetAddress2, parserOffset));
+ IntPtr targetAddress = RuntimeAugments.GetCodeTarget(entryMethodEntrypoint);
+ if (targetAddress != IntPtr.Zero && targetAddress != entryMethodEntrypoint)
+ {
+ functionPointers.Add(new FunctionPointerOffsetPair(targetAddress, parserOffset));
+ }
+
+ IntPtr targetAddress2;
+ if (TypeLoaderEnvironment.TryGetTargetOfUnboxingAndInstantiatingStub(entryMethodEntrypoint, out targetAddress2) &&
+ targetAddress2 != entryMethodEntrypoint &&
+ targetAddress2 != targetAddress)
+ {
+ functionPointers.Add(new FunctionPointerOffsetPair(targetAddress2, parserOffset));
+ }
}
}
diff --git a/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Reflection/Execution/AssemblyBinderImplementation.cs b/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Reflection/Execution/AssemblyBinderImplementation.cs
index dd3f3dac45b191..bd6c07a99e2651 100644
--- a/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Reflection/Execution/AssemblyBinderImplementation.cs
+++ b/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Reflection/Execution/AssemblyBinderImplementation.cs
@@ -84,7 +84,7 @@ public sealed override bool Bind(RuntimeAssemblyName refName, bool cacheMissedLo
{
if (foundMatch)
{
- exception = new AmbiguousMatchException();
+ exception = new AmbiguousMatchException(SR.Format(SR.AmbiguousMatchException_Assembly, refName.FullName));
return false;
}
diff --git a/src/coreclr/nativeaot/System.Private.TypeLoader/src/Resources/Strings.resx b/src/coreclr/nativeaot/System.Private.TypeLoader/src/Resources/Strings.resx
index 949e8adc2fbadc..cddf33a5bca989 100644
--- a/src/coreclr/nativeaot/System.Private.TypeLoader/src/Resources/Strings.resx
+++ b/src/coreclr/nativeaot/System.Private.TypeLoader/src/Resources/Strings.resx
@@ -1,17 +1,17 @@
-
@@ -129,4 +129,7 @@
Cannot load assembly '{0}'. The assembly exists but its version {1} is lower than the requested version {2}.
+
+ Ambiguous match found for assembly '{0}'.
+
diff --git a/src/coreclr/nativeaot/Test.CoreLib/src/System/Runtime/RuntimeImports.cs b/src/coreclr/nativeaot/Test.CoreLib/src/System/Runtime/RuntimeImports.cs
index bd1681f250dfa6..5232ab073a6ca8 100644
--- a/src/coreclr/nativeaot/Test.CoreLib/src/System/Runtime/RuntimeImports.cs
+++ b/src/coreclr/nativeaot/Test.CoreLib/src/System/Runtime/RuntimeImports.cs
@@ -103,10 +103,6 @@ internal static unsafe Array RhNewArray(EETypePtr pEEType, int length)
[RuntimeImport(RuntimeLibrary, "RhpLockCmpXchg64")]
internal static extern long InterlockedCompareExchange(ref long location1, long value, long comparand);
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- [RuntimeImport(RuntimeLibrary, "RhpMemoryBarrier")]
- internal static extern void MemoryBarrier();
-
// Moves memory from smem to dmem. Size must be a positive value.
// This copy uses an intrinsic to be safe for copying arbitrary bits of
// heap memory
diff --git a/src/coreclr/nativeaot/Test.CoreLib/src/System/RuntimeTypeHandle.cs b/src/coreclr/nativeaot/Test.CoreLib/src/System/RuntimeTypeHandle.cs
index d921e27015b73c..f0401dc09e07a9 100644
--- a/src/coreclr/nativeaot/Test.CoreLib/src/System/RuntimeTypeHandle.cs
+++ b/src/coreclr/nativeaot/Test.CoreLib/src/System/RuntimeTypeHandle.cs
@@ -10,17 +10,17 @@ namespace System
[StructLayout(LayoutKind.Sequential)]
public struct RuntimeTypeHandle
{
- private EETypePtr _pEEType;
+ private IntPtr _value;
- internal RuntimeTypeHandle(EETypePtr pEEType)
+ internal RuntimeTypeHandle(IntPtr value)
{
- _pEEType = pEEType;
+ _value = value;
}
[Intrinsic]
internal static unsafe IntPtr ToIntPtr(RuntimeTypeHandle handle)
{
- return (IntPtr)handle._pEEType.ToPointer();
+ return handle._value;
}
}
}
@@ -32,7 +32,7 @@ internal static class LdTokenHelpers
{
private static RuntimeTypeHandle GetRuntimeTypeHandle(IntPtr pEEType)
{
- return new RuntimeTypeHandle(new EETypePtr(pEEType));
+ return new RuntimeTypeHandle(pEEType);
}
}
}
diff --git a/src/coreclr/nativeaot/Test.CoreLib/src/System/Threading/Interlocked.cs b/src/coreclr/nativeaot/Test.CoreLib/src/System/Threading/Interlocked.cs
index 34d02ffca1d6dc..58decc90318c1e 100644
--- a/src/coreclr/nativeaot/Test.CoreLib/src/System/Threading/Interlocked.cs
+++ b/src/coreclr/nativeaot/Test.CoreLib/src/System/Threading/Interlocked.cs
@@ -31,9 +31,6 @@ public static long CompareExchange(ref long location1, long value, long comparan
}
[Intrinsic]
- public static void MemoryBarrier()
- {
- RuntimeImports.MemoryBarrier();
- }
+ public static void MemoryBarrier() => MemoryBarrier();
}
}
diff --git a/src/coreclr/nativeaot/docs/optimizing.md b/src/coreclr/nativeaot/docs/optimizing.md
index 9f2c9ae70d12a8..9a90eb97c21286 100644
--- a/src/coreclr/nativeaot/docs/optimizing.md
+++ b/src/coreclr/nativeaot/docs/optimizing.md
@@ -40,4 +40,5 @@ Since `PublishTrimmed` is implied to be true with Native AOT, some framework fea
* `Speed`: when generating optimized code, favor code execution speed.
* `Size`: when generating optimized code, favor smaller code size.
* ``: By default, the compiler targets the minimum instruction set supported by the target OS and architecture. This option allows targeting newer instruction sets for better performance. The native binary will require the instruction sets to be supported by the hardware in order to run. For example, `avx2,bmi2,fma,pclmul,popcnt,aes` will produce binary that takes advantage of instruction sets that are typically present on current Intel and AMD processors. Run `ilc --help` for the full list of available instruction sets. `ilc` can be executed from the NativeAOT package in your local nuget cache e.g. `%USERPROFILE%\.nuget\packages\runtime.win-x64.microsoft.dotnet.ilcompiler\8.0.0-...\tools\ilc.exe` on Windows or `~/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-.../tools/ilc` on Linux.
+* ``: By default, the compiler targets the a `Vector` size of `16` or `32` bytes, depending on the underlying instruction sets supported. This option allows specifying a different maximum bit width. For example, if by default on x64 hardware `Vector` will be 16-bytes. However, if `AVX2` is targeted then `Vector` will automatically grow to be 32-bytes instead, setting `128` would keep the size as 16-bytes. Alternatively, even if `AVX512F` is targeted then by default `Vector` will not grow larger than 32-bytes, setting `512` would allow it to grow to 64-bytes.
diff --git a/src/coreclr/pal/src/misc/jitsupport.cpp b/src/coreclr/pal/src/misc/jitsupport.cpp
index f7ca5c36e71217..30426290043514 100644
--- a/src/coreclr/pal/src/misc/jitsupport.cpp
+++ b/src/coreclr/pal/src/misc/jitsupport.cpp
@@ -235,7 +235,10 @@ PAL_GetJitCpuCapabilityFlags(CORJIT_FLAGS *flags)
#endif
#ifdef HWCAP_ASIMD
if (hwCap & HWCAP_ASIMD)
+ {
flags->Set(InstructionSet_AdvSimd);
+ flags->Set(InstructionSet_VectorT128);
+ }
#endif
#ifdef HWCAP_ASIMDRDM
if (hwCap & HWCAP_ASIMDRDM)
@@ -292,6 +295,7 @@ PAL_GetJitCpuCapabilityFlags(CORJIT_FLAGS *flags)
// Set baseline flags if OS has not exposed mechanism for us to determine CPU capabilities
flags->Set(InstructionSet_ArmBase);
flags->Set(InstructionSet_AdvSimd);
+ flags->Set(InstructionSet_VectorT128);
// flags->Set(CORJIT_FLAGS::CORJIT_FLAG_HAS_ARM64_FP);
#endif // HAVE_AUXV_HWCAP_H
}
diff --git a/src/coreclr/scripts/superpmi-collect.proj b/src/coreclr/scripts/superpmi-collect.proj
index a7a465b4639bb2..490836a21cc014 100644
--- a/src/coreclr/scripts/superpmi-collect.proj
+++ b/src/coreclr/scripts/superpmi-collect.proj
@@ -72,19 +72,19 @@
-
+ %HELIX_WORKITEM_PAYLOAD%\binaries
-
+ $HELIX_WORKITEM_PAYLOAD/binaries
-
+ %HELIX_WORKITEM_PAYLOAD%\performance
-
+ $HELIX_WORKITEM_PAYLOAD/performance
@@ -130,12 +130,12 @@
-
+ $(Python) $(SuperPMIDirectory)$(FileSeparatorChar)superpmi.py collect --clean -log_level DEBUG --$(CollectionType) $(PmiArguments) -assemblies $(AssembliesDirectoryOnHelix) -arch $(Architecture) -build_type $(BuildConfig) -core_root $(SuperPMIDirectory)2:00
-
+ $(Python) $(SuperPMIDirectory)$(FileSeparatorChar)superpmi_benchmarks.py -performance_directory $(PerformanceDirectory) -superpmi_directory $(SuperPMIDirectory) -core_root $(SuperPMIDirectory) -arch $(Architecture) $(RunArguments)3:00
@@ -174,7 +174,7 @@
-
+
@@ -182,8 +182,15 @@
-
+ 30
+ src\benchmarks\micro\MicroBenchmarks.csproj
+ MicroBenchmarks.dll
+
+
+ 30
+ src/benchmarks/micro/MicroBenchmarks.csproj
+ MicroBenchmarks.dll
@@ -217,8 +224,24 @@
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ $(CollectionName).$(CollectionType).%(HelixWorkItem.PartitionId).$(MchFileTag)$(AssembliesPayload)$(FileSeparatorChar)%(HelixWorkItem.CollectAssemblies)
@@ -232,9 +255,19 @@
$(CollectionName).$(CollectionType).%(HelixWorkItem.Index).$(MchFileTag)$(WorkItemDirectory)
- $(WorkItemCommand) -partition_count $(PartitionCount) -partition_index %(HelixWorkItem.Index) -output_mch_path $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).mch -log_file $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).log
+ $(WorkItemCommand) -partition_count $(PartitionCount) -partition_index %(HelixWorkItem.Index) -benchmark_path $(BenchmarkPath) -benchmark_binary $(BenchmarkBinary) -output_mch_path $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).mch -log_file $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).log$(WorkItemTimeout)%(OutputFileName).mch;%(OutputFileName).mch.mct;%(OutputFileName).log
+
+
+
+ $(CollectionName).$(CollectionType).%(HelixWorkItem.Index).$(MchFileTag)
+ $(WorkItemDirectory)
+ $(WorkItemCommand) -partition_count 1 -partition_index 0 -benchmark_path %(HelixWorkItem.BenchmarkPath) -benchmark_binary %(HelixWorkItem.BenchmarkBinary) -output_mch_path $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).mch -log_file $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).log
+ $(WorkItemTimeout)
+ %(OutputFileName).mch;%(OutputFileName).mch.mct;%(OutputFileName).log
+
+
diff --git a/src/coreclr/scripts/superpmi_benchmarks.py b/src/coreclr/scripts/superpmi_benchmarks.py
index 7a09f47c8fd198..8b78a0099c4ddc 100644
--- a/src/coreclr/scripts/superpmi_benchmarks.py
+++ b/src/coreclr/scripts/superpmi_benchmarks.py
@@ -8,8 +8,9 @@
#
# Notes:
#
-# Script to perform the superpmi collection while executing the Microbenchmarks present
-# in https://github.com/dotnet/performance/tree/master/src/benchmarks/micro.
+# Script to perform the superpmi collection while executing the Microbenchmarks in
+# https://github.com/dotnet/performance/tree/master/src/benchmarks/micro and real-world in
+# in https://github.com/dotnet/performance/tree/master/src/benchmarks/real-world
import argparse
import re
@@ -34,6 +35,8 @@
parser.add_argument("-partition_count", help="Total number of partitions")
parser.add_argument("-partition_index", help="Partition index to do the collection for")
parser.add_argument("-arch", help="Architecture")
+parser.add_argument("-benchmark_path", help="Benchmark's csproj path in dotnet/performance repo")
+parser.add_argument("-benchmark_binary", help="Benchmark binary to execute")
parser.add_argument("--tiered_compilation", action="store_true", help="Sets DOTNET_TieredCompilation=1 when doing collections.")
parser.add_argument("--tiered_pgo", action="store_true", help="Sets DOTNET_TieredCompilation=1 and DOTNET_TieredPGO=1 when doing collections.")
@@ -90,6 +93,16 @@ def setup_args(args):
lambda arch: arch.lower() in ["x86", "x64", "arm", "arm64"],
"Unable to set arch")
+ coreclr_args.verify(args,
+ "benchmark_path",
+ lambda unused: True,
+ "Unable to set benchmark_path")
+
+ coreclr_args.verify(args,
+ "benchmark_binary",
+ lambda unused: True,
+ "Unable to set benchmark_binary")
+
coreclr_args.verify(args,
"tiered_compilation",
lambda unused: True,
@@ -123,9 +136,8 @@ def make_executable(file_name):
(stat.S_IROTH | stat.S_IXOTH))
run_command(["ls", "-l", file_name])
-
def build_and_run(coreclr_args, output_mch_name):
- """Build the microbenchmarks and run them under "superpmi collect"
+ """Build the microbenchmarks/real-world and run them under "superpmi collect"
Args:
coreclr_args (CoreClrArguments): Arguments use to drive
@@ -139,13 +151,15 @@ def build_and_run(coreclr_args, output_mch_name):
log_file = coreclr_args.log_file
partition_count = coreclr_args.partition_count
partition_index = coreclr_args.partition_index
+ benchmark_path = coreclr_args.benchmark_path
+ benchmark_binary = coreclr_args.benchmark_binary
dotnet_directory = os.path.join(performance_directory, "tools", "dotnet", arch)
dotnet_exe = os.path.join(dotnet_directory, "dotnet")
artifacts_directory = os.path.join(performance_directory, "artifacts")
artifacts_packages_directory = os.path.join(artifacts_directory, "packages")
- project_file = os.path.join(performance_directory, "src", "benchmarks", "micro", "MicroBenchmarks.csproj")
- benchmarks_dll = os.path.join(artifacts_directory, "MicroBenchmarks.dll")
+ project_file = os.path.join(performance_directory, benchmark_path)
+ benchmarks_dll = os.path.join(artifacts_directory, benchmark_binary)
# Workaround https://github.com/dotnet/sdk/issues/23430
project_file = os.path.realpath(project_file)
@@ -153,11 +167,11 @@ def build_and_run(coreclr_args, output_mch_name):
if is_windows:
shim_name = "%JitName%"
corerun_exe = "CoreRun.exe"
- script_name = "run_microbenchmarks.bat"
+ script_name = "run_benchmarks.bat"
else:
shim_name = "$JitName"
corerun_exe = "corerun"
- script_name = "run_microbenchmarks.sh"
+ script_name = "run_benchmarks.sh"
make_executable(dotnet_exe)
@@ -192,11 +206,25 @@ def build_and_run(coreclr_args, output_mch_name):
"--framework", "net8.0", "--no-restore", "/p:NuGetPackageRoot=" + artifacts_packages_directory,
"-o", artifacts_directory], _exit_on_fail=True)
- # Disable ReadyToRun so we always JIT R2R methods and collect them
- collection_command = f"{dotnet_exe} {benchmarks_dll} --filter \"*\" --corerun {os.path.join(core_root, corerun_exe)} --partition-count {partition_count} " \
- f"--partition-index {partition_index} --envVars DOTNET_JitName:{shim_name} " \
- " DOTNET_ZapDisable:1 DOTNET_ReadyToRun:0 " \
- "--iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --logBuildOutput"
+ if benchmark_binary.lower().startswith("microbenchmarks"):
+ # Disable ReadyToRun so we always JIT R2R methods and collect them
+ collection_command = f"{dotnet_exe} {benchmarks_dll} --filter \"*\" --corerun {os.path.join(core_root, corerun_exe)} --partition-count {partition_count} " \
+ f"--partition-index {partition_index} --envVars DOTNET_JitName:{shim_name} " \
+ " DOTNET_ZapDisable:1 DOTNET_ReadyToRun:0 " \
+ "--iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --logBuildOutput"
+ elif benchmark_binary.lower().startswith("demobenchmarks"):
+ # Disable ReadyToRun so we always JIT R2R methods and collect them
+ collection_command = f"{dotnet_exe} {benchmarks_dll} -f *CollisionBatcherTaskBenchmarks.* *GroupedCollisionTesterBenchmarks.* *GatherScatterBenchmarks.* " \
+ " *OneBodyConstraintBenchmarks.* *TwoBodyConstraintBenchmarks.* *ThreeBodyConstraintBenchmarks.* *FourBodyConstraintBenchmarks.* " \
+ " *SweepBenchmarks.* *ShapeRayBenchmarks.* *ShapePileBenchmark.* *RagdollTubeBenchmark.* " \
+ f" --corerun {os.path.join(core_root, corerun_exe)} --envVars DOTNET_JitName:{shim_name} " \
+ " DOTNET_ZapDisable:1 DOTNET_ReadyToRun:0 " \
+ "--iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --logBuildOutput"
+ else:
+ # Disable ReadyToRun so we always JIT R2R methods and collect them
+ collection_command = f"{dotnet_exe} {benchmarks_dll} --filter \"*\" --corerun {os.path.join(core_root, corerun_exe)} --envVars DOTNET_JitName:{shim_name} " \
+ " DOTNET_ZapDisable:1 DOTNET_ReadyToRun:0 " \
+ "--iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --logBuildOutput"
# Generate the execution script in Temp location
with TempDir() as temp_location:
diff --git a/src/coreclr/scripts/superpmi_collect_setup.py b/src/coreclr/scripts/superpmi_collect_setup.py
index 84240c90418f73..f93cd287e15161 100644
--- a/src/coreclr/scripts/superpmi_collect_setup.py
+++ b/src/coreclr/scripts/superpmi_collect_setup.py
@@ -376,8 +376,8 @@ def partition_files(src_directory, dst_directory, max_size, exclude_directories=
index += 1
-def setup_microbenchmark(workitem_directory, arch):
- """ Perform setup of microbenchmarks
+def setup_benchmark(workitem_directory, arch):
+ """ Perform setup of microbenchmarks/realworld
Args:
workitem_directory (string): Path to work
@@ -485,9 +485,9 @@ def main(main_args):
print('Copying {} -> {}'.format(coreclr_args.core_root_directory, core_root_dst_directory))
copy_directory(coreclr_args.core_root_directory, core_root_dst_directory, verbose_output=True, match_func=acceptable_copy)
- if coreclr_args.collection_name == "benchmarks":
- # Setup microbenchmarks
- setup_microbenchmark(workitem_payload_directory, arch)
+ if coreclr_args.collection_name == "benchmarks" or coreclr_args.collection_name == "realworld":
+ # Setup benchmarks
+ setup_benchmark(workitem_payload_directory, arch)
else:
# Setup for pmi/crossgen2 runs
diff --git a/src/coreclr/scripts/superpmi_diffs_setup.py b/src/coreclr/scripts/superpmi_diffs_setup.py
index db12452de8d86e..c54de696ccd972 100644
--- a/src/coreclr/scripts/superpmi_diffs_setup.py
+++ b/src/coreclr/scripts/superpmi_diffs_setup.py
@@ -171,7 +171,6 @@ def build_jit_analyze(coreclr_args, source_directory, jit_analyze_build_director
# The RID catalog is here: https://docs.microsoft.com/en-us/dotnet/core/rid-catalog.
# Windows x64 => win-x64
# Windows x86 => win-x86
- # Windows arm32 => win-arm
# Windows arm64 => win-arm64
# Linux x64 => linux-x64
# Linux arm32 => linux-arm
diff --git a/src/coreclr/tools/Common/Compiler/DependencyAnalysis/EmbeddedDataContainerNode.cs b/src/coreclr/tools/Common/Compiler/DependencyAnalysis/EmbeddedDataContainerNode.cs
index 3aa40631fb3fae..a43b776bddc929 100644
--- a/src/coreclr/tools/Common/Compiler/DependencyAnalysis/EmbeddedDataContainerNode.cs
+++ b/src/coreclr/tools/Common/Compiler/DependencyAnalysis/EmbeddedDataContainerNode.cs
@@ -1,33 +1,33 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
+using Internal.Text;
+
namespace ILCompiler.DependencyAnalysis
{
///
/// Represents a node that contains a set of embedded objects. The main function is
/// to serve as a base class, providing symbol name boundaries and node ordering.
///
- public abstract class EmbeddedDataContainerNode : ObjectNode
+ public abstract class EmbeddedDataContainerNode : ObjectNode, ISymbolDefinitionNode
{
- private ObjectAndOffsetSymbolNode _startSymbol;
- private ObjectAndOffsetSymbolNode _endSymbol;
- private string _startSymbolMangledName;
-
- public ObjectAndOffsetSymbolNode StartSymbol => _startSymbol;
- public ObjectAndOffsetSymbolNode EndSymbol => _endSymbol;
+ private string _mangledName;
- protected EmbeddedDataContainerNode(string startSymbolMangledName, string endSymbolMangledName)
+ protected EmbeddedDataContainerNode(string mangledName)
{
- _startSymbolMangledName = startSymbolMangledName;
- _startSymbol = new ObjectAndOffsetSymbolNode(this, 0, startSymbolMangledName, true);
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, endSymbolMangledName, true);
+ _mangledName = mangledName;
}
+ public int Offset => 0;
+
public override int ClassCode => -1410622237;
+ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
+ => sb.Append(nameMangler.CompilationUnitPrefix).Append(_mangledName);
+
public override int CompareToImpl(ISortableNode other, CompilerComparer comparer)
{
- return _startSymbolMangledName.CompareTo(((EmbeddedDataContainerNode)other)._startSymbolMangledName);
+ return _mangledName.CompareTo(((EmbeddedDataContainerNode)other)._mangledName);
}
}
}
diff --git a/src/coreclr/tools/Common/Compiler/DependencyAnalysis/ObjectAndOffsetSymbolNode.cs b/src/coreclr/tools/Common/Compiler/DependencyAnalysis/ObjectAndOffsetSymbolNode.cs
deleted file mode 100644
index 0f15706ea685c8..00000000000000
--- a/src/coreclr/tools/Common/Compiler/DependencyAnalysis/ObjectAndOffsetSymbolNode.cs
+++ /dev/null
@@ -1,60 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-using System;
-using System.Collections.Generic;
-
-using ILCompiler.DependencyAnalysisFramework;
-
-using Internal.Text;
-
-namespace ILCompiler.DependencyAnalysis
-{
- public class ObjectAndOffsetSymbolNode : DependencyNodeCore, ISymbolDefinitionNode
- {
- private ObjectNode _object;
- private int _offset;
- private Utf8String _name;
- private bool _includeCompilationUnitPrefix;
-
- public ObjectAndOffsetSymbolNode(ObjectNode obj, int offset, Utf8String name, bool includeCompilationUnitPrefix)
- {
- _object = obj;
- _offset = offset;
- _name = name;
- _includeCompilationUnitPrefix = includeCompilationUnitPrefix;
- }
-
- protected override string GetName(NodeFactory factory) => $"Symbol {_name} at offset {_offset.ToStringInvariant()}";
-
- public override bool HasConditionalStaticDependencies => false;
- public override bool HasDynamicDependencies => false;
- public override bool InterestingForDynamicDependencyAnalysis => false;
- public override bool StaticDependenciesAreComputed => true;
-
- public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
- {
- if (_includeCompilationUnitPrefix)
- sb.Append(nameMangler.CompilationUnitPrefix);
- sb.Append(_name);
- }
-
- int ISymbolNode.Offset => 0;
- int ISymbolDefinitionNode.Offset => _offset;
- public bool RepresentsIndirectionCell => false;
-
- public void SetSymbolOffset(int offset)
- {
- _offset = offset;
- }
-
- public ObjectNode Target => _object;
-
- public override IEnumerable GetStaticDependencies(NodeFactory factory)
- {
- return new DependencyListEntry[] { new DependencyListEntry(_object, "ObjectAndOffsetDependency") };
- }
-
- public override IEnumerable GetConditionalStaticDependencies(NodeFactory factory) => null;
- public override IEnumerable SearchDynamicDependencies(List> markedNodes, int firstNode, NodeFactory factory) => null;
- }
-}
diff --git a/src/coreclr/tools/Common/Compiler/InstructionSetSupport.cs b/src/coreclr/tools/Common/Compiler/InstructionSetSupport.cs
index 6d124bda4673c6..46528387fec8ca 100644
--- a/src/coreclr/tools/Common/Compiler/InstructionSetSupport.cs
+++ b/src/coreclr/tools/Common/Compiler/InstructionSetSupport.cs
@@ -96,18 +96,37 @@ public SimdVectorLength GetVectorTSimdVector()
{
if ((_targetArchitecture == TargetArchitecture.X64) || (_targetArchitecture == TargetArchitecture.X86))
{
- Debug.Assert(InstructionSet.X64_AVX2 == InstructionSet.X86_AVX2);
- Debug.Assert(InstructionSet.X64_SSE2 == InstructionSet.X86_SSE2);
- if (IsInstructionSetSupported(InstructionSet.X86_AVX2))
+ Debug.Assert(InstructionSet.X64_VectorT128 == InstructionSet.X86_VectorT128);
+ Debug.Assert(InstructionSet.X64_VectorT256 == InstructionSet.X86_VectorT256);
+ Debug.Assert(InstructionSet.X64_VectorT512 == InstructionSet.X86_VectorT512);
+
+ // TODO-XArch: Add support for 512-bit Vector
+ Debug.Assert(!IsInstructionSetSupported(InstructionSet.X64_VectorT512));
+
+ if (IsInstructionSetSupported(InstructionSet.X64_VectorT256))
+ {
+ Debug.Assert(!IsInstructionSetSupported(InstructionSet.X64_VectorT128));
return SimdVectorLength.Vector256Bit;
- else if (IsInstructionSetExplicitlyUnsupported(InstructionSet.X86_AVX2) && IsInstructionSetSupported(InstructionSet.X64_SSE2))
+ }
+ else if (IsInstructionSetSupported(InstructionSet.X64_VectorT128))
+ {
return SimdVectorLength.Vector128Bit;
+ }
else
+ {
return SimdVectorLength.None;
+ }
}
else if (_targetArchitecture == TargetArchitecture.ARM64)
{
- return SimdVectorLength.Vector128Bit;
+ if (IsInstructionSetSupported(InstructionSet.ARM64_VectorT128))
+ {
+ return SimdVectorLength.Vector128Bit;
+ }
+ else
+ {
+ return SimdVectorLength.None;
+ }
}
else if (_targetArchitecture == TargetArchitecture.ARM)
{
@@ -183,15 +202,24 @@ public static InstructionSetFlags GetNonSpecifiableInstructionSetsForArch(Target
return s_nonSpecifiableInstructionSets[architecture];
}
- private readonly SortedSet _supportedInstructionSets = new SortedSet();
- private readonly SortedSet _unsupportedInstructionSets = new SortedSet();
+ private readonly SortedSet _supportedInstructionSets;
+ private readonly SortedSet _unsupportedInstructionSets;
private readonly TargetArchitecture _architecture;
public InstructionSetSupportBuilder(TargetArchitecture architecture)
{
+ _supportedInstructionSets = new SortedSet();
+ _unsupportedInstructionSets = new SortedSet();
_architecture = architecture;
}
+ public InstructionSetSupportBuilder(InstructionSetSupportBuilder other)
+ {
+ _supportedInstructionSets = new SortedSet(other._supportedInstructionSets);
+ _unsupportedInstructionSets = new SortedSet(other._unsupportedInstructionSets);
+ _architecture = other._architecture;
+ }
+
///
/// Add a supported instruction set to the specified list.
///
@@ -245,9 +273,10 @@ public bool RemoveInstructionSetSupport(string instructionSet)
/// Seal modifications to instruction set support
///
/// returns "false" if instruction set isn't valid on this architecture
- public bool ComputeInstructionSetFlags(out InstructionSetFlags supportedInstructionSets,
- out InstructionSetFlags unsupportedInstructionSets,
- Action invalidInstructionSetImplication)
+ public bool ComputeInstructionSetFlags(int maxVectorTBitWidth,
+ out InstructionSetFlags supportedInstructionSets,
+ out InstructionSetFlags unsupportedInstructionSets,
+ Action invalidInstructionSetImplication)
{
supportedInstructionSets = new InstructionSetFlags();
unsupportedInstructionSets = new InstructionSetFlags();
@@ -288,6 +317,51 @@ public bool ComputeInstructionSetFlags(out InstructionSetFlags supportedInstruct
}
}
+ switch (_architecture)
+ {
+ case TargetArchitecture.X64:
+ case TargetArchitecture.X86:
+ {
+ Debug.Assert(InstructionSet.X86_SSE2 == InstructionSet.X64_SSE2);
+ Debug.Assert(InstructionSet.X86_AVX2 == InstructionSet.X64_AVX2);
+ Debug.Assert(InstructionSet.X86_AVX512F == InstructionSet.X64_AVX512F);
+
+ Debug.Assert(InstructionSet.X86_VectorT128 == InstructionSet.X64_VectorT128);
+ Debug.Assert(InstructionSet.X86_VectorT256 == InstructionSet.X64_VectorT256);
+ Debug.Assert(InstructionSet.X86_VectorT512 == InstructionSet.X64_VectorT512);
+
+ // We only want one size supported for Vector and we want the other sizes explicitly
+ // unsupported to ensure we throw away the given methods if runtime picks a larger size
+
+ Debug.Assert(supportedInstructionSets.HasInstructionSet(InstructionSet.X86_SSE2));
+ Debug.Assert((maxVectorTBitWidth == 0) || (maxVectorTBitWidth >= 128));
+ supportedInstructionSets.AddInstructionSet(InstructionSet.X86_VectorT128);
+
+ if (supportedInstructionSets.HasInstructionSet(InstructionSet.X86_AVX2))
+ {
+ if ((maxVectorTBitWidth == 0) || (maxVectorTBitWidth >= 256))
+ {
+ supportedInstructionSets.RemoveInstructionSet(InstructionSet.X86_VectorT128);
+ supportedInstructionSets.AddInstructionSet(InstructionSet.X86_VectorT256);
+
+ unsupportedInstructionSets.AddInstructionSet(InstructionSet.X86_VectorT128);
+ unsupportedInstructionSets.AddInstructionSet(InstructionSet.X86_VectorT512);
+ }
+
+ // TODO-XArch: Add support for 512-bit Vector
+ }
+ break;
+ }
+
+ case TargetArchitecture.ARM64:
+ {
+ Debug.Assert(supportedInstructionSets.HasInstructionSet(InstructionSet.ARM64_AdvSimd));
+ Debug.Assert((maxVectorTBitWidth == 0) || (maxVectorTBitWidth >= 128));
+ supportedInstructionSets.AddInstructionSet(InstructionSet.ARM64_VectorT128);
+ break;
+ }
+ }
+
return true;
}
}
diff --git a/src/coreclr/tools/Common/InstructionSetHelpers.cs b/src/coreclr/tools/Common/InstructionSetHelpers.cs
index 8aa5705916dede..c969927e1531f7 100644
--- a/src/coreclr/tools/Common/InstructionSetHelpers.cs
+++ b/src/coreclr/tools/Common/InstructionSetHelpers.cs
@@ -11,7 +11,7 @@ namespace System.CommandLine
{
internal static partial class Helpers
{
- public static InstructionSetSupport ConfigureInstructionSetSupport(string instructionSet, TargetArchitecture targetArchitecture, TargetOS targetOS,
+ public static InstructionSetSupport ConfigureInstructionSetSupport(string instructionSet, int maxVectorTBitWidth, TargetArchitecture targetArchitecture, TargetOS targetOS,
string mustNotBeMessage, string invalidImplicationMessage)
{
InstructionSetSupportBuilder instructionSetSupportBuilder = new(targetArchitecture);
@@ -74,11 +74,16 @@ public static InstructionSetSupport ConfigureInstructionSetSupport(string instru
}
}
- instructionSetSupportBuilder.ComputeInstructionSetFlags(out var supportedInstructionSet, out var unsupportedInstructionSet,
+ instructionSetSupportBuilder.ComputeInstructionSetFlags(maxVectorTBitWidth, out var supportedInstructionSet, out var unsupportedInstructionSet,
(string specifiedInstructionSet, string impliedInstructionSet) =>
throw new CommandLineException(string.Format(invalidImplicationMessage, specifiedInstructionSet, impliedInstructionSet)));
- InstructionSetSupportBuilder optimisticInstructionSetSupportBuilder = new InstructionSetSupportBuilder(targetArchitecture);
+ // Due to expansion by implication, the optimistic set is most often a pure superset of the supported set
+ //
+ // However, there are some gaps in cases like Arm64 neon where none of the optimistic sets imply it. Likewise,
+ // the optimistic set would be missing the explicitly unsupported sets. So we effectively clone the list and
+ // tack on the additional optimistic bits after. This ensures the optimistic set remains an accurate superset
+ InstructionSetSupportBuilder optimisticInstructionSetSupportBuilder = new InstructionSetSupportBuilder(instructionSetSupportBuilder);
// Optimistically assume some instruction sets are present.
if (targetArchitecture == TargetArchitecture.X86 || targetArchitecture == TargetArchitecture.X64)
@@ -93,15 +98,38 @@ public static InstructionSetSupport ConfigureInstructionSetSupport(string instru
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("movbe");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("popcnt");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("lzcnt");
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("serialize");
// If AVX was enabled, we can opportunistically enable instruction sets which use the VEX encodings
Debug.Assert(InstructionSet.X64_AVX == InstructionSet.X86_AVX);
if (supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX))
{
+ // TODO: Enable optimistic usage of AVX2 once we validate it doesn't break Vector usage
+ // optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("avx2");
+
+ if (supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX2))
+ {
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("avxvnni");
+ }
+
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("fma");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("bmi");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("bmi2");
- optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("avxvnni");
+ }
+
+ Debug.Assert(InstructionSet.X64_AVX512F == InstructionSet.X86_AVX512F);
+ if (supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512F))
+ {
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512F_VL));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512BW));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512BW_VL));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512CD));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512CD_VL));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512DQ));
+ Debug.Assert(supportedInstructionSet.HasInstructionSet(InstructionSet.X64_AVX512DQ_VL));
+
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("avx512vbmi");
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("avx512vbmi_vl");
}
}
else if (targetArchitecture == TargetArchitecture.ARM64)
@@ -111,9 +139,12 @@ public static InstructionSetSupport ConfigureInstructionSetSupport(string instru
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("sha1");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("sha2");
optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("lse");
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("dotprod");
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("rdma");
+ optimisticInstructionSetSupportBuilder.AddSupportedInstructionSet("rcpc");
}
- optimisticInstructionSetSupportBuilder.ComputeInstructionSetFlags(out var optimisticInstructionSet, out _,
+ optimisticInstructionSetSupportBuilder.ComputeInstructionSetFlags(maxVectorTBitWidth, out var optimisticInstructionSet, out _,
(string specifiedInstructionSet, string impliedInstructionSet) => throw new NotSupportedException());
optimisticInstructionSet.Remove(unsupportedInstructionSet);
optimisticInstructionSet.Add(supportedInstructionSet);
diff --git a/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSet.cs b/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSet.cs
index 0068f0be007764..32b60ecbcda7da 100644
--- a/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSet.cs
+++ b/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSet.cs
@@ -50,6 +50,9 @@ public enum ReadyToRunInstructionSet
Avx512DQ_VL=36,
Avx512Vbmi=37,
Avx512Vbmi_VL=38,
+ VectorT128=39,
+ VectorT256=40,
+ VectorT512=41,
}
}
diff --git a/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSetHelper.cs b/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSetHelper.cs
index b387eedd89f2b7..f593808be32989 100644
--- a/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSetHelper.cs
+++ b/src/coreclr/tools/Common/Internal/Runtime/ReadyToRunInstructionSetHelper.cs
@@ -44,6 +44,7 @@ public static class ReadyToRunInstructionSetHelper
case InstructionSet.ARM64_Vector128: return null;
case InstructionSet.ARM64_Dczva: return null;
case InstructionSet.ARM64_Rcpc: return ReadyToRunInstructionSet.Rcpc;
+ case InstructionSet.ARM64_VectorT128: return ReadyToRunInstructionSet.VectorT128;
default: throw new Exception("Unknown instruction set");
}
@@ -114,6 +115,9 @@ public static class ReadyToRunInstructionSetHelper
case InstructionSet.X64_AVX512VBMI_X64: return ReadyToRunInstructionSet.Avx512Vbmi;
case InstructionSet.X64_AVX512VBMI_VL: return ReadyToRunInstructionSet.Avx512Vbmi_VL;
case InstructionSet.X64_AVX512VBMI_VL_X64: return ReadyToRunInstructionSet.Avx512Vbmi_VL;
+ case InstructionSet.X64_VectorT128: return ReadyToRunInstructionSet.VectorT128;
+ case InstructionSet.X64_VectorT256: return ReadyToRunInstructionSet.VectorT256;
+ case InstructionSet.X64_VectorT512: return ReadyToRunInstructionSet.VectorT512;
default: throw new Exception("Unknown instruction set");
}
@@ -184,6 +188,9 @@ public static class ReadyToRunInstructionSetHelper
case InstructionSet.X86_AVX512VBMI_X64: return null;
case InstructionSet.X86_AVX512VBMI_VL: return ReadyToRunInstructionSet.Avx512Vbmi_VL;
case InstructionSet.X86_AVX512VBMI_VL_X64: return null;
+ case InstructionSet.X86_VectorT128: return ReadyToRunInstructionSet.VectorT128;
+ case InstructionSet.X86_VectorT256: return ReadyToRunInstructionSet.VectorT256;
+ case InstructionSet.X86_VectorT512: return ReadyToRunInstructionSet.VectorT512;
default: throw new Exception("Unknown instruction set");
}
diff --git a/src/coreclr/tools/Common/JitInterface/CorInfoInstructionSet.cs b/src/coreclr/tools/Common/JitInterface/CorInfoInstructionSet.cs
index df6c06206bf84d..f26abd68262391 100644
--- a/src/coreclr/tools/Common/JitInterface/CorInfoInstructionSet.cs
+++ b/src/coreclr/tools/Common/JitInterface/CorInfoInstructionSet.cs
@@ -15,7 +15,7 @@ namespace Internal.JitInterface
public enum InstructionSet
{
ILLEGAL = 0,
- NONE = 63,
+ NONE = 127,
ARM64_ArmBase = InstructionSet_ARM64.ArmBase,
ARM64_AdvSimd = InstructionSet_ARM64.AdvSimd,
ARM64_Aes = InstructionSet_ARM64.Aes,
@@ -29,6 +29,7 @@ public enum InstructionSet
ARM64_Vector128 = InstructionSet_ARM64.Vector128,
ARM64_Dczva = InstructionSet_ARM64.Dczva,
ARM64_Rcpc = InstructionSet_ARM64.Rcpc,
+ ARM64_VectorT128 = InstructionSet_ARM64.VectorT128,
ARM64_ArmBase_Arm64 = InstructionSet_ARM64.ArmBase_Arm64,
ARM64_AdvSimd_Arm64 = InstructionSet_ARM64.AdvSimd_Arm64,
ARM64_Aes_Arm64 = InstructionSet_ARM64.Aes_Arm64,
@@ -69,6 +70,9 @@ public enum InstructionSet
X64_AVX512DQ_VL = InstructionSet_X64.AVX512DQ_VL,
X64_AVX512VBMI = InstructionSet_X64.AVX512VBMI,
X64_AVX512VBMI_VL = InstructionSet_X64.AVX512VBMI_VL,
+ X64_VectorT128 = InstructionSet_X64.VectorT128,
+ X64_VectorT256 = InstructionSet_X64.VectorT256,
+ X64_VectorT512 = InstructionSet_X64.VectorT512,
X64_X86Base_X64 = InstructionSet_X64.X86Base_X64,
X64_SSE_X64 = InstructionSet_X64.SSE_X64,
X64_SSE2_X64 = InstructionSet_X64.SSE2_X64,
@@ -130,6 +134,9 @@ public enum InstructionSet
X86_AVX512DQ_VL = InstructionSet_X86.AVX512DQ_VL,
X86_AVX512VBMI = InstructionSet_X86.AVX512VBMI,
X86_AVX512VBMI_VL = InstructionSet_X86.AVX512VBMI_VL,
+ X86_VectorT128 = InstructionSet_X86.VectorT128,
+ X86_VectorT256 = InstructionSet_X86.VectorT256,
+ X86_VectorT512 = InstructionSet_X86.VectorT512,
X86_X86Base_X64 = InstructionSet_X86.X86Base_X64,
X86_SSE_X64 = InstructionSet_X86.SSE_X64,
X86_SSE2_X64 = InstructionSet_X86.SSE2_X64,
@@ -177,14 +184,15 @@ public enum InstructionSet_ARM64
Vector128 = 11,
Dczva = 12,
Rcpc = 13,
- ArmBase_Arm64 = 14,
- AdvSimd_Arm64 = 15,
- Aes_Arm64 = 16,
- Crc32_Arm64 = 17,
- Dp_Arm64 = 18,
- Rdm_Arm64 = 19,
- Sha1_Arm64 = 20,
- Sha256_Arm64 = 21,
+ VectorT128 = 14,
+ ArmBase_Arm64 = 15,
+ AdvSimd_Arm64 = 16,
+ Aes_Arm64 = 17,
+ Crc32_Arm64 = 18,
+ Dp_Arm64 = 19,
+ Rdm_Arm64 = 20,
+ Sha1_Arm64 = 21,
+ Sha256_Arm64 = 22,
}
public enum InstructionSet_X64
@@ -223,35 +231,38 @@ public enum InstructionSet_X64
AVX512DQ_VL = 30,
AVX512VBMI = 31,
AVX512VBMI_VL = 32,
- X86Base_X64 = 33,
- SSE_X64 = 34,
- SSE2_X64 = 35,
- SSE3_X64 = 36,
- SSSE3_X64 = 37,
- SSE41_X64 = 38,
- SSE42_X64 = 39,
- AVX_X64 = 40,
- AVX2_X64 = 41,
- AES_X64 = 42,
- BMI1_X64 = 43,
- BMI2_X64 = 44,
- FMA_X64 = 45,
- LZCNT_X64 = 46,
- PCLMULQDQ_X64 = 47,
- POPCNT_X64 = 48,
- AVXVNNI_X64 = 49,
- MOVBE_X64 = 50,
- X86Serialize_X64 = 51,
- AVX512F_X64 = 52,
- AVX512F_VL_X64 = 53,
- AVX512BW_X64 = 54,
- AVX512BW_VL_X64 = 55,
- AVX512CD_X64 = 56,
- AVX512CD_VL_X64 = 57,
- AVX512DQ_X64 = 58,
- AVX512DQ_VL_X64 = 59,
- AVX512VBMI_X64 = 60,
- AVX512VBMI_VL_X64 = 61,
+ VectorT128 = 33,
+ VectorT256 = 34,
+ VectorT512 = 35,
+ X86Base_X64 = 36,
+ SSE_X64 = 37,
+ SSE2_X64 = 38,
+ SSE3_X64 = 39,
+ SSSE3_X64 = 40,
+ SSE41_X64 = 41,
+ SSE42_X64 = 42,
+ AVX_X64 = 43,
+ AVX2_X64 = 44,
+ AES_X64 = 45,
+ BMI1_X64 = 46,
+ BMI2_X64 = 47,
+ FMA_X64 = 48,
+ LZCNT_X64 = 49,
+ PCLMULQDQ_X64 = 50,
+ POPCNT_X64 = 51,
+ AVXVNNI_X64 = 52,
+ MOVBE_X64 = 53,
+ X86Serialize_X64 = 54,
+ AVX512F_X64 = 55,
+ AVX512F_VL_X64 = 56,
+ AVX512BW_X64 = 57,
+ AVX512BW_VL_X64 = 58,
+ AVX512CD_X64 = 59,
+ AVX512CD_VL_X64 = 60,
+ AVX512DQ_X64 = 61,
+ AVX512DQ_VL_X64 = 62,
+ AVX512VBMI_X64 = 63,
+ AVX512VBMI_VL_X64 = 64,
}
public enum InstructionSet_X86
@@ -290,40 +301,43 @@ public enum InstructionSet_X86
AVX512DQ_VL = 30,
AVX512VBMI = 31,
AVX512VBMI_VL = 32,
- X86Base_X64 = 33,
- SSE_X64 = 34,
- SSE2_X64 = 35,
- SSE3_X64 = 36,
- SSSE3_X64 = 37,
- SSE41_X64 = 38,
- SSE42_X64 = 39,
- AVX_X64 = 40,
- AVX2_X64 = 41,
- AES_X64 = 42,
- BMI1_X64 = 43,
- BMI2_X64 = 44,
- FMA_X64 = 45,
- LZCNT_X64 = 46,
- PCLMULQDQ_X64 = 47,
- POPCNT_X64 = 48,
- AVXVNNI_X64 = 49,
- MOVBE_X64 = 50,
- X86Serialize_X64 = 51,
- AVX512F_X64 = 52,
- AVX512F_VL_X64 = 53,
- AVX512BW_X64 = 54,
- AVX512BW_VL_X64 = 55,
- AVX512CD_X64 = 56,
- AVX512CD_VL_X64 = 57,
- AVX512DQ_X64 = 58,
- AVX512DQ_VL_X64 = 59,
- AVX512VBMI_X64 = 60,
- AVX512VBMI_VL_X64 = 61,
+ VectorT128 = 33,
+ VectorT256 = 34,
+ VectorT512 = 35,
+ X86Base_X64 = 36,
+ SSE_X64 = 37,
+ SSE2_X64 = 38,
+ SSE3_X64 = 39,
+ SSSE3_X64 = 40,
+ SSE41_X64 = 41,
+ SSE42_X64 = 42,
+ AVX_X64 = 43,
+ AVX2_X64 = 44,
+ AES_X64 = 45,
+ BMI1_X64 = 46,
+ BMI2_X64 = 47,
+ FMA_X64 = 48,
+ LZCNT_X64 = 49,
+ PCLMULQDQ_X64 = 50,
+ POPCNT_X64 = 51,
+ AVXVNNI_X64 = 52,
+ MOVBE_X64 = 53,
+ X86Serialize_X64 = 54,
+ AVX512F_X64 = 55,
+ AVX512F_VL_X64 = 56,
+ AVX512BW_X64 = 57,
+ AVX512BW_VL_X64 = 58,
+ AVX512CD_X64 = 59,
+ AVX512CD_VL_X64 = 60,
+ AVX512DQ_X64 = 61,
+ AVX512DQ_VL_X64 = 62,
+ AVX512VBMI_X64 = 63,
+ AVX512VBMI_VL_X64 = 64,
}
public unsafe struct InstructionSetFlags : IEnumerable
{
- private const int FlagsFieldCount = 1;
+ private const int FlagsFieldCount = 2;
private const int BitsPerFlagsField = 64;
private fixed ulong _flags[FlagsFieldCount];
public IEnumerable ARM64Flags => this.Select((x) => (InstructionSet_ARM64)x);
@@ -527,6 +541,8 @@ public static InstructionSetFlags ExpandInstructionSetByImplicationHelper(Target
resultflags.AddInstructionSet(InstructionSet.ARM64_AdvSimd);
if (resultflags.HasInstructionSet(InstructionSet.ARM64_Vector128))
resultflags.AddInstructionSet(InstructionSet.ARM64_AdvSimd);
+ if (resultflags.HasInstructionSet(InstructionSet.ARM64_VectorT128))
+ resultflags.AddInstructionSet(InstructionSet.ARM64_AdvSimd);
break;
case TargetArchitecture.X64:
@@ -690,24 +706,46 @@ public static InstructionSetFlags ExpandInstructionSetByImplicationHelper(Target
resultflags.AddInstructionSet(InstructionSet.X64_X86Base);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X64_AVX2);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X64_FMA);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512CD))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512CD_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512CD);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512CD_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512DQ))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512DQ_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512DQ);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512DQ_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512VBMI))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512VBMI_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512VBMI);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512VBMI_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_VectorT128))
+ resultflags.AddInstructionSet(InstructionSet.X64_SSE2);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_VectorT256))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX2);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_VectorT512))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512CD_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512DQ_VL);
break;
case TargetArchitecture.X86:
@@ -755,24 +793,46 @@ public static InstructionSetFlags ExpandInstructionSetByImplicationHelper(Target
resultflags.AddInstructionSet(InstructionSet.X86_X86Base);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X86_AVX2);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X86_FMA);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512CD))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512CD_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512CD);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512CD_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512DQ))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512DQ_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512DQ);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512DQ_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512VBMI))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512VBMI_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512VBMI);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512VBMI_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_VectorT128))
+ resultflags.AddInstructionSet(InstructionSet.X86_SSE2);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_VectorT256))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX2);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_VectorT512))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512CD_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512DQ_VL);
break;
}
} while (!oldflags.Equals(resultflags));
@@ -830,6 +890,8 @@ private static InstructionSetFlags ExpandInstructionSetByReverseImplicationHelpe
resultflags.AddInstructionSet(InstructionSet.ARM64_Vector64);
if (resultflags.HasInstructionSet(InstructionSet.ARM64_AdvSimd))
resultflags.AddInstructionSet(InstructionSet.ARM64_Vector128);
+ if (resultflags.HasInstructionSet(InstructionSet.ARM64_AdvSimd))
+ resultflags.AddInstructionSet(InstructionSet.ARM64_VectorT128);
break;
case TargetArchitecture.X64:
@@ -935,24 +997,46 @@ private static InstructionSetFlags ExpandInstructionSetByReverseImplicationHelpe
resultflags.AddInstructionSet(InstructionSet.X64_X86Serialize);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX2))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_FMA))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512CD);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512CD))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512DQ);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512DQ))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512VBMI);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512VBMI))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512VBMI_VL);
if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW_VL))
resultflags.AddInstructionSet(InstructionSet.X64_AVX512VBMI_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_SSE2))
+ resultflags.AddInstructionSet(InstructionSet.X64_VectorT128);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX2))
+ resultflags.AddInstructionSet(InstructionSet.X64_VectorT256);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X64_VectorT512);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512BW_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512CD_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X64_AVX512DQ_VL))
+ resultflags.AddInstructionSet(InstructionSet.X64_AVX512F);
break;
case TargetArchitecture.X86:
@@ -1000,24 +1084,46 @@ private static InstructionSetFlags ExpandInstructionSetByReverseImplicationHelpe
resultflags.AddInstructionSet(InstructionSet.X86_X86Serialize);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX2))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_FMA))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512F_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512CD);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512CD))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512CD_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512BW_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512DQ);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512DQ))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512DQ_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512VBMI);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512VBMI))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512VBMI_VL);
if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW_VL))
resultflags.AddInstructionSet(InstructionSet.X86_AVX512VBMI_VL);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_SSE2))
+ resultflags.AddInstructionSet(InstructionSet.X86_VectorT128);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX2))
+ resultflags.AddInstructionSet(InstructionSet.X86_VectorT256);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512F))
+ resultflags.AddInstructionSet(InstructionSet.X86_VectorT512);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512BW_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512CD_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
+ if (resultflags.HasInstructionSet(InstructionSet.X86_AVX512DQ_VL))
+ resultflags.AddInstructionSet(InstructionSet.X86_AVX512F);
break;
}
} while (!oldflags.Equals(resultflags));
@@ -1035,8 +1141,8 @@ private static InstructionSetFlags ExpandInstructionSetByReverseImplicationHelpe
{ ("x86-x64-v3", TargetArchitecture.X86), "x86-x64-v2 avx2 bmi bmi2 lzcnt movbe fma" },
{ ("skylake", TargetArchitecture.X64), "x86-x64-v3" },
{ ("skylake", TargetArchitecture.X86), "x86-x64-v3" },
- { ("x86-x64-v4", TargetArchitecture.X64), "x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl" },
- { ("x86-x64-v4", TargetArchitecture.X86), "x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl" },
+ { ("x86-x64-v4", TargetArchitecture.X64), "x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl avx512dq avx512dq_vl" },
+ { ("x86-x64-v4", TargetArchitecture.X86), "x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl avx512dq avx512dq_vl" },
{ ("armv8-a", TargetArchitecture.ARM64), "neon" },
{ ("armv8.1-a", TargetArchitecture.ARM64), "armv8-a lse crc rdma" },
{ ("armv8.2-a", TargetArchitecture.ARM64), "armv8.1-a" },
@@ -1088,6 +1194,7 @@ public static IEnumerable ArchitectureToValidInstructionSets
yield return new InstructionSetInfo("Vector128", "", InstructionSet.ARM64_Vector128, false);
yield return new InstructionSetInfo("Dczva", "", InstructionSet.ARM64_Dczva, false);
yield return new InstructionSetInfo("rcpc", "", InstructionSet.ARM64_Rcpc, true);
+ yield return new InstructionSetInfo("vectort128", "VectorT128", InstructionSet.ARM64_VectorT128, true);
break;
case TargetArchitecture.X64:
@@ -1123,6 +1230,9 @@ public static IEnumerable ArchitectureToValidInstructionSets
yield return new InstructionSetInfo("avx512dq_vl", "Avx512DQ_VL", InstructionSet.X64_AVX512DQ_VL, true);
yield return new InstructionSetInfo("avx512vbmi", "Avx512Vbmi", InstructionSet.X64_AVX512VBMI, true);
yield return new InstructionSetInfo("avx512vbmi_vl", "Avx512Vbmi_VL", InstructionSet.X64_AVX512VBMI_VL, true);
+ yield return new InstructionSetInfo("vectort128", "VectorT128", InstructionSet.X64_VectorT128, true);
+ yield return new InstructionSetInfo("vectort256", "VectorT256", InstructionSet.X64_VectorT256, true);
+ yield return new InstructionSetInfo("vectort512", "VectorT512", InstructionSet.X64_VectorT512, true);
break;
case TargetArchitecture.X86:
@@ -1158,6 +1268,9 @@ public static IEnumerable ArchitectureToValidInstructionSets
yield return new InstructionSetInfo("avx512dq_vl", "Avx512DQ_VL", InstructionSet.X86_AVX512DQ_VL, true);
yield return new InstructionSetInfo("avx512vbmi", "Avx512Vbmi", InstructionSet.X86_AVX512VBMI, true);
yield return new InstructionSetInfo("avx512vbmi_vl", "Avx512Vbmi_VL", InstructionSet.X86_AVX512VBMI_VL, true);
+ yield return new InstructionSetInfo("vectort128", "VectorT128", InstructionSet.X86_VectorT128, true);
+ yield return new InstructionSetInfo("vectort256", "VectorT256", InstructionSet.X86_VectorT256, true);
+ yield return new InstructionSetInfo("vectort512", "VectorT512", InstructionSet.X86_VectorT512, true);
break;
}
}
@@ -1432,6 +1545,9 @@ public static InstructionSet LookupPlatformIntrinsicInstructionSet(TargetArchite
else
{ return InstructionSet.ARM64_Sha256; }
+ case "VectorT128":
+ { return InstructionSet.ARM64_VectorT128; }
+
}
break;
@@ -1598,6 +1714,15 @@ public static InstructionSet LookupPlatformIntrinsicInstructionSet(TargetArchite
else
{ return InstructionSet.X64_AVX512VBMI; }
+ case "VectorT128":
+ { return InstructionSet.X64_VectorT128; }
+
+ case "VectorT256":
+ { return InstructionSet.X64_VectorT256; }
+
+ case "VectorT512":
+ { return InstructionSet.X64_VectorT512; }
+
}
break;
@@ -1692,6 +1817,15 @@ public static InstructionSet LookupPlatformIntrinsicInstructionSet(TargetArchite
else
{ return InstructionSet.X86_AVX512VBMI; }
+ case "VectorT128":
+ { return InstructionSet.X86_VectorT128; }
+
+ case "VectorT256":
+ { return InstructionSet.X86_VectorT256; }
+
+ case "VectorT512":
+ { return InstructionSet.X86_VectorT512; }
+
}
break;
diff --git a/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs b/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs
index 71d8cd105dc60a..d2a792e25a0c18 100644
--- a/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs
+++ b/src/coreclr/tools/Common/JitInterface/CorInfoTypes.cs
@@ -1425,7 +1425,7 @@ public enum CorJitFlag : uint
CORJIT_FLAG_BBINSTR_IF_LOOPS = 32, // JIT must instrument current method if it has loops
CORJIT_FLAG_PUBLISH_SECRET_PARAM = 33, // JIT must place stub secret param into local 0. (used by IL stubs)
CORJIT_FLAG_UNUSED9 = 34,
- CORJIT_FLAG_SAMPLING_JIT_BACKGROUND = 35, // JIT is being invoked as a result of stack sampling for hot methods in the background
+ CORJIT_FLAG_UNUSED1 = 35,
CORJIT_FLAG_USE_PINVOKE_HELPERS = 36, // The JIT should use the PINVOKE_{BEGIN,END} helpers instead of emitting inline transitions
CORJIT_FLAG_REVERSE_PINVOKE = 37, // The JIT should insert REVERSE_PINVOKE_{ENTER,EXIT} helpers into method prolog/epilog
CORJIT_FLAG_TRACK_TRANSITIONS = 38, // The JIT should insert the helper variants that track transitions.
diff --git a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetDesc.txt b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetDesc.txt
index d5f74271ed13fb..3c669e1ea95fea 100644
--- a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetDesc.txt
+++ b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetDesc.txt
@@ -57,7 +57,9 @@ instructionset ,X86 ,Avx512DQ , ,35 ,AVX512DQ
instructionset ,X86 ,Avx512DQ_VL , ,36 ,AVX512DQ_VL ,avx512dq_vl
instructionset ,X86 ,Avx512Vbmi , ,37 ,AVX512VBMI ,avx512vbmi
instructionset ,X86 ,Avx512Vbmi_VL , ,38 ,AVX512VBMI_VL ,avx512vbmi_vl
-
+instructionset ,X86 ,VectorT128 , ,39 ,VectorT128 ,vectort128
+instructionset ,X86 ,VectorT256 , ,40 ,VectorT256 ,vectort256
+instructionset ,X86 ,VectorT512 , ,41 ,VectorT512 ,vectort512
instructionset64bit,X86 ,X86Base
instructionset64bit,X86 ,SSE
@@ -115,15 +117,33 @@ implication ,X86 ,AVXVNNI ,AVX2
implication ,X86 ,MOVBE ,SSE42
implication ,X86 ,X86Serialize ,X86Base
implication ,X86 ,AVX512F ,AVX2
+implication ,X86 ,AVX512F ,FMA
implication ,X86 ,AVX512F_VL ,AVX512F
implication ,X86 ,AVX512CD ,AVX512F
+implication ,X86 ,AVX512CD_VL ,AVX512CD
implication ,X86 ,AVX512CD_VL ,AVX512F_VL
implication ,X86 ,AVX512BW ,AVX512F
+implication ,X86 ,AVX512BW_VL ,AVX512BW
implication ,X86 ,AVX512BW_VL ,AVX512F_VL
implication ,X86 ,AVX512DQ ,AVX512F
+implication ,X86 ,AVX512DQ_VL ,AVX512DQ
implication ,X86 ,AVX512DQ_VL ,AVX512F_VL
implication ,X86 ,AVX512VBMI ,AVX512BW
+implication ,X86 ,AVX512VBMI_VL ,AVX512VBMI
implication ,X86 ,AVX512VBMI_VL ,AVX512BW_VL
+implication ,X86 ,VectorT128 ,SSE2
+implication ,X86 ,VectorT256 ,AVX2
+implication ,X86 ,VectorT512 ,AVX512F
+
+; While the AVX-512 ISAs can be individually lit-up, they really
+; need F, BW, CD, DQ, and VL to be fully functional without adding
+; significant complexity into the JIT. Additionally, unlike AVX/AVX2
+; there was never really any hardware that didn't provide all 5 at
+; once, with the notable exception being Knight's Landing which
+; provided a similar but not quite the same feature.
+implication ,X86 ,AVX512F ,AVX512BW_VL
+implication ,X86 ,AVX512F ,AVX512CD_VL
+implication ,X86 ,AVX512F ,AVX512DQ_VL
; Definition of X64 instruction sets
definearch ,X64 ,64Bit ,X64, X64
@@ -133,19 +153,20 @@ copyinstructionsets,X86 ,X64
; Definition of Arm64 instruction sets
definearch ,ARM64 ,64Bit ,Arm64, Arm64
-instructionset ,ARM64 ,ArmBase , ,16 ,ArmBase ,base
-instructionset ,ARM64 ,AdvSimd , ,17 ,AdvSimd ,neon
-instructionset ,ARM64 ,Aes , ,9 ,Aes ,aes
-instructionset ,ARM64 ,Crc32 , ,18 ,Crc32 ,crc
-instructionset ,ARM64 ,Dp , ,23 ,Dp ,dotprod
-instructionset ,ARM64 ,Rdm , ,24 ,Rdm ,rdma
-instructionset ,ARM64 ,Sha1 , ,19 ,Sha1 ,sha1
-instructionset ,ARM64 ,Sha256 , ,20 ,Sha256 ,sha2
-instructionset ,ARM64 , ,Atomics ,21 ,Atomics ,lse
-instructionset ,ARM64 , , , ,Vector64 ,
-instructionset ,ARM64 , , , ,Vector128,
-instructionset ,ARM64 , , , ,Dczva ,
-instructionset ,ARM64 , ,Rcpc ,26 ,Rcpc ,rcpc
+instructionset ,ARM64 ,ArmBase , ,16 ,ArmBase ,base
+instructionset ,ARM64 ,AdvSimd , ,17 ,AdvSimd ,neon
+instructionset ,ARM64 ,Aes , ,9 ,Aes ,aes
+instructionset ,ARM64 ,Crc32 , ,18 ,Crc32 ,crc
+instructionset ,ARM64 ,Dp , ,23 ,Dp ,dotprod
+instructionset ,ARM64 ,Rdm , ,24 ,Rdm ,rdma
+instructionset ,ARM64 ,Sha1 , ,19 ,Sha1 ,sha1
+instructionset ,ARM64 ,Sha256 , ,20 ,Sha256 ,sha2
+instructionset ,ARM64 , ,Atomics ,21 ,Atomics ,lse
+instructionset ,ARM64 , , , ,Vector64 ,
+instructionset ,ARM64 , , , ,Vector128 ,
+instructionset ,ARM64 , , , ,Dczva ,
+instructionset ,ARM64 , ,Rcpc ,26 ,Rcpc ,rcpc
+instructionset ,ARM64 ,VectorT128 , ,39 ,VectorT128 ,vectort128
instructionset64bit,ARM64 ,ArmBase
instructionset64bit,ARM64 ,AdvSimd
@@ -159,16 +180,16 @@ instructionset64bit,ARM64 ,Sha256
vectorinstructionset,ARM64,Vector64
vectorinstructionset,ARM64,Vector128
-implication ,ARM64 ,AdvSimd ,ArmBase
-implication ,ARM64 ,Aes ,ArmBase
-implication ,ARM64 ,Crc32 ,ArmBase
-implication ,ARM64 ,Dp ,AdvSimd
-implication ,ARM64 ,Rdm ,AdvSimd
-implication ,ARM64 ,Sha1 ,ArmBase
-implication ,ARM64 ,Sha256 ,ArmBase
-implication ,ARM64 ,Vector64 ,AdvSimd
-implication ,ARM64 ,Vector128 ,AdvSimd
-
+implication ,ARM64 ,AdvSimd ,ArmBase
+implication ,ARM64 ,Aes ,ArmBase
+implication ,ARM64 ,Crc32 ,ArmBase
+implication ,ARM64 ,Dp ,AdvSimd
+implication ,ARM64 ,Rdm ,AdvSimd
+implication ,ARM64 ,Sha1 ,ArmBase
+implication ,ARM64 ,Sha256 ,ArmBase
+implication ,ARM64 ,Vector64 ,AdvSimd
+implication ,ARM64 ,Vector128 ,AdvSimd
+implication ,ARM64 ,VectorT128 ,AdvSimd
; ,name and aliases ,archs ,lower baselines included by implication
;
@@ -176,7 +197,7 @@ instructionsetgroup ,x86-x64 ,X64 X86 ,sse2
instructionsetgroup ,x86-x64-v2 ,X64 X86 ,sse4.2 popcnt
instructionsetgroup ,x86-x64-v3 ,X64 X86 ,x86-x64-v2 avx2 bmi bmi2 lzcnt movbe fma
instructionsetgroup ,skylake ,X64 X86 ,x86-x64-v3
-instructionsetgroup ,x86-x64-v4 ,X64 X86 ,x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl
+instructionsetgroup ,x86-x64-v4 ,X64 X86 ,x86-x64-v3 avx512f avx512f_vl avx512bw avx512bw_vl avx512cd avx512cd_vl avx512dq avx512dq_vl
instructionsetgroup ,armv8-a ,ARM64 ,neon
instructionsetgroup ,armv8.1-a ,ARM64 ,armv8-a lse crc rdma
diff --git a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetGenerator.cs b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetGenerator.cs
index 4547b91e2fdc70..3ce3bf5a795dbd 100644
--- a/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetGenerator.cs
+++ b/src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetGenerator.cs
@@ -95,7 +95,7 @@ public InstructionSetImplication(string architecture, InstructionSetImplication
private Dictionary _64BitVariantArchitectureManagedNameSuffix = new Dictionary();
// This represents the number of flags fields we currently track
- private const int FlagsFieldCount = 1;
+ private const int FlagsFieldCount = 2;
private void ArchitectureEncountered(string arch)
{
diff --git a/src/coreclr/tools/Common/TypeSystem/Common/DefType.FieldLayout.cs b/src/coreclr/tools/Common/TypeSystem/Common/DefType.FieldLayout.cs
index 99305804f53519..fb4cd6bf17157f 100644
--- a/src/coreclr/tools/Common/TypeSystem/Common/DefType.FieldLayout.cs
+++ b/src/coreclr/tools/Common/TypeSystem/Common/DefType.FieldLayout.cs
@@ -73,6 +73,11 @@ private static class FieldLayoutFlags
/// True if the type transitively has an Int128 in it or is an Int128
///
public const int IsInt128OrHasInt128Fields = 0x800;
+
+ ///
+ /// True if the type transitively has a Vector in it or is Vector
+ ///
+ public const int IsVectorTOrHasVectorTFields = 0x1000;
}
private sealed class StaticBlockInfo
@@ -153,6 +158,21 @@ public virtual bool IsInt128OrHasInt128Fields
}
}
+ ///
+ /// Is a type Vector or transitively have any fields of a type Vector.
+ ///
+ public virtual bool IsVectorTOrHasVectorTFields
+ {
+ get
+ {
+ if (!_fieldLayoutFlags.HasFlags(FieldLayoutFlags.ComputedInstanceTypeLayout))
+ {
+ ComputeInstanceLayout(InstanceLayoutKind.TypeAndFields);
+ }
+ return _fieldLayoutFlags.HasFlags(FieldLayoutFlags.IsVectorTOrHasVectorTFields);
+ }
+ }
+
///
/// The number of bytes required to hold a field of this type
///
@@ -451,6 +471,10 @@ public void ComputeInstanceLayout(InstanceLayoutKind layoutKind)
{
_fieldLayoutFlags.AddFlags(FieldLayoutFlags.IsInt128OrHasInt128Fields);
}
+ if (computedLayout.IsVectorTOrHasVectorTFields)
+ {
+ _fieldLayoutFlags.AddFlags(FieldLayoutFlags.IsVectorTOrHasVectorTFields);
+ }
if (computedLayout.Offsets != null)
{
diff --git a/src/coreclr/tools/Common/TypeSystem/Common/FieldLayoutAlgorithm.cs b/src/coreclr/tools/Common/TypeSystem/Common/FieldLayoutAlgorithm.cs
index 53388c915b85d8..31a46ec47f6416 100644
--- a/src/coreclr/tools/Common/TypeSystem/Common/FieldLayoutAlgorithm.cs
+++ b/src/coreclr/tools/Common/TypeSystem/Common/FieldLayoutAlgorithm.cs
@@ -84,6 +84,7 @@ public struct ComputedInstanceFieldLayout
public bool LayoutAbiStable; // Is the layout stable such that it can safely be used in function calling conventions
public bool IsAutoLayoutOrHasAutoLayoutFields;
public bool IsInt128OrHasInt128Fields;
+ public bool IsVectorTOrHasVectorTFields;
///
/// If Offsets is non-null, then all field based layout is complete.
diff --git a/src/coreclr/tools/Common/TypeSystem/Common/MetadataFieldLayoutAlgorithm.cs b/src/coreclr/tools/Common/TypeSystem/Common/MetadataFieldLayoutAlgorithm.cs
index 0fc9064fb00a81..d56bfa19bbf210 100644
--- a/src/coreclr/tools/Common/TypeSystem/Common/MetadataFieldLayoutAlgorithm.cs
+++ b/src/coreclr/tools/Common/TypeSystem/Common/MetadataFieldLayoutAlgorithm.cs
@@ -110,6 +110,7 @@ out instanceByteSizeAndAlignment
LayoutAbiStable = true,
IsAutoLayoutOrHasAutoLayoutFields = false,
IsInt128OrHasInt128Fields = false,
+ IsVectorTOrHasVectorTFields = false,
};
if (numInstanceFields > 0)
@@ -211,7 +212,7 @@ public override ComputedStaticFieldLayout ComputeStaticFieldLayout(DefType defTy
}
ref StaticsBlock block = ref GetStaticsBlockForField(ref result, field);
- SizeAndAlignment sizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout: false, context.Target.DefaultPackingSize, out bool _, out bool _, out bool _);
+ SizeAndAlignment sizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout: false, context.Target.DefaultPackingSize, out bool _, out bool _, out bool _, out bool _);
block.Size = LayoutInt.AlignUp(block.Size, sizeAndAlignment.Alignment, context.Target);
result.Offsets[index] = new FieldAndOffset(field, block.Size);
@@ -303,18 +304,27 @@ protected ComputedInstanceFieldLayout ComputeExplicitFieldLayout(MetadataType ty
int fieldOrdinal = 0;
bool layoutAbiStable = true;
bool hasAutoLayoutField = false;
- bool hasInt128Field = type.BaseType == null ? false : type.BaseType.IsInt128OrHasInt128Fields;
+ bool hasInt128Field = false;
+ bool hasVectorTField = false;
+
+ if (type.BaseType is not null)
+ {
+ hasInt128Field = type.BaseType.IsInt128OrHasInt128Fields;
+ hasVectorTField = type.BaseType.IsVectorTOrHasVectorTFields;
+ }
foreach (var fieldAndOffset in layoutMetadata.Offsets)
{
TypeDesc fieldType = fieldAndOffset.Field.FieldType;
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType.UnderlyingType, hasLayout: true, packingSize, out bool fieldLayoutAbiStable, out bool fieldHasAutoLayout, out bool fieldHasInt128Field);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType.UnderlyingType, hasLayout: true, packingSize, out bool fieldLayoutAbiStable, out bool fieldHasAutoLayout, out bool fieldHasInt128Field, out bool fieldHasVectorTField);
if (!fieldLayoutAbiStable)
layoutAbiStable = false;
if (fieldHasAutoLayout)
hasAutoLayoutField = true;
if (fieldHasInt128Field)
hasInt128Field = true;
+ if (fieldHasVectorTField)
+ hasVectorTField = true;
largestAlignmentRequired = LayoutInt.Max(fieldSizeAndAlignment.Alignment, largestAlignmentRequired);
@@ -367,6 +377,7 @@ protected ComputedInstanceFieldLayout ComputeExplicitFieldLayout(MetadataType ty
{
IsAutoLayoutOrHasAutoLayoutFields = hasAutoLayoutField,
IsInt128OrHasInt128Fields = hasInt128Field,
+ IsVectorTOrHasVectorTFields = hasVectorTField,
};
computedLayout.FieldAlignment = instanceSizeAndAlignment.Alignment;
computedLayout.FieldSize = instanceSizeAndAlignment.Size;
@@ -402,20 +413,29 @@ protected ComputedInstanceFieldLayout ComputeSequentialFieldLayout(MetadataType
int packingSize = ComputePackingSize(type, layoutMetadata);
bool layoutAbiStable = true;
bool hasAutoLayoutField = false;
- bool hasInt128Field = type.BaseType == null ? false : type.BaseType.IsInt128OrHasInt128Fields;
+ bool hasInt128Field = false;
+ bool hasVectorTField = false;
+
+ if (type.BaseType is not null)
+ {
+ hasInt128Field = type.BaseType.IsInt128OrHasInt128Fields;
+ hasVectorTField = type.BaseType.IsVectorTOrHasVectorTFields;
+ }
foreach (var field in type.GetFields())
{
if (field.IsStatic)
continue;
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(field.FieldType.UnderlyingType, hasLayout: true, packingSize, out bool fieldLayoutAbiStable, out bool fieldHasAutoLayout, out bool fieldHasInt128Field);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(field.FieldType.UnderlyingType, hasLayout: true, packingSize, out bool fieldLayoutAbiStable, out bool fieldHasAutoLayout, out bool fieldHasInt128Field, out bool fieldHasVectorTField);
if (!fieldLayoutAbiStable)
layoutAbiStable = false;
if (fieldHasAutoLayout)
hasAutoLayoutField = true;
if (fieldHasInt128Field)
hasInt128Field = true;
+ if (fieldHasVectorTField)
+ hasVectorTField = true;
largestAlignmentRequirement = LayoutInt.Max(fieldSizeAndAlignment.Alignment, largestAlignmentRequirement);
@@ -443,6 +463,7 @@ protected ComputedInstanceFieldLayout ComputeSequentialFieldLayout(MetadataType
{
IsAutoLayoutOrHasAutoLayoutFields = hasAutoLayoutField,
IsInt128OrHasInt128Fields = hasInt128Field,
+ IsVectorTOrHasVectorTFields = hasVectorTField,
};
computedLayout.FieldAlignment = instanceSizeAndAlignment.Alignment;
computedLayout.FieldSize = instanceSizeAndAlignment.Size;
@@ -517,6 +538,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
int instanceGCPointerFieldsCount = 0;
int[] instanceNonGCPointerFieldsCount = new int[maxLog2Size + 1];
bool hasInt128Field = false;
+ bool hasVectorTField = false;
foreach (var field in type.GetFields())
{
@@ -531,6 +553,8 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
instanceValueClassFieldCount++;
if (((DefType)fieldType).IsInt128OrHasInt128Fields)
hasInt128Field = true;
+ if (((DefType)fieldType).IsVectorTOrHasVectorTFields)
+ hasVectorTField = true;
}
else if (fieldType.IsGCPointer)
{
@@ -540,7 +564,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
{
Debug.Assert(fieldType.IsPrimitive || fieldType.IsPointer || fieldType.IsFunctionPointer || fieldType.IsEnum || fieldType.IsByRef);
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout, packingSize, out bool _, out bool _, out bool _);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout, packingSize, out bool _, out bool _, out bool _, out bool _);
instanceNonGCPointerFieldsCount[CalculateLog2(fieldSizeAndAlignment.Size.AsInt)]++;
}
}
@@ -577,7 +601,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
TypeDesc fieldType = field.FieldType;
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout, packingSize, out bool fieldLayoutAbiStable, out bool _, out bool _);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(fieldType, hasLayout, packingSize, out bool fieldLayoutAbiStable, out bool _, out bool _, out bool _);
if (!fieldLayoutAbiStable)
layoutAbiStable = false;
@@ -747,7 +771,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
for (int i = 0; i < instanceValueClassFieldsArr.Length; i++)
{
// Align the cumulative field offset to the indeterminate value
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(instanceValueClassFieldsArr[i].FieldType, hasLayout, packingSize, out bool fieldLayoutAbiStable, out bool _, out bool _);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(instanceValueClassFieldsArr[i].FieldType, hasLayout, packingSize, out bool fieldLayoutAbiStable, out bool _, out bool _, out bool _);
if (!fieldLayoutAbiStable)
layoutAbiStable = false;
@@ -804,6 +828,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
{
IsAutoLayoutOrHasAutoLayoutFields = true,
IsInt128OrHasInt128Fields = hasInt128Field,
+ IsVectorTOrHasVectorTFields = hasVectorTField,
};
computedLayout.FieldAlignment = instanceSizeAndAlignment.Alignment;
computedLayout.FieldSize = instanceSizeAndAlignment.Size;
@@ -817,7 +842,7 @@ protected ComputedInstanceFieldLayout ComputeAutoFieldLayout(MetadataType type,
private static void PlaceInstanceField(FieldDesc field, bool hasLayout, int packingSize, FieldAndOffset[] offsets, ref LayoutInt instanceFieldPos, ref int fieldOrdinal, LayoutInt offsetBias)
{
- var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(field.FieldType, hasLayout, packingSize, out bool _, out bool _, out bool _);
+ var fieldSizeAndAlignment = ComputeFieldSizeAndAlignment(field.FieldType, hasLayout, packingSize, out bool _, out bool _, out bool _, out bool _);
instanceFieldPos = AlignUpInstanceFieldOffset(instanceFieldPos, fieldSizeAndAlignment.Alignment, field.Context.Target);
offsets[fieldOrdinal] = new FieldAndOffset(field, instanceFieldPos + offsetBias);
@@ -877,12 +902,13 @@ public LayoutInt CalculateFieldBaseOffset(MetadataType type, bool requiresAlign8
return cumulativeInstanceFieldPos;
}
- private static SizeAndAlignment ComputeFieldSizeAndAlignment(TypeDesc fieldType, bool hasLayout, int packingSize, out bool layoutAbiStable, out bool fieldTypeHasAutoLayout, out bool fieldTypeHasInt128Field)
+ private static SizeAndAlignment ComputeFieldSizeAndAlignment(TypeDesc fieldType, bool hasLayout, int packingSize, out bool layoutAbiStable, out bool fieldTypeHasAutoLayout, out bool fieldTypeHasInt128Field, out bool fieldTypeHasVectorTField)
{
SizeAndAlignment result;
layoutAbiStable = true;
fieldTypeHasAutoLayout = true;
fieldTypeHasInt128Field = false;
+ fieldTypeHasVectorTField = false;
if (fieldType.IsDefType)
{
@@ -894,6 +920,7 @@ private static SizeAndAlignment ComputeFieldSizeAndAlignment(TypeDesc fieldType,
layoutAbiStable = defType.LayoutAbiStable;
fieldTypeHasAutoLayout = defType.IsAutoLayoutOrHasAutoLayoutFields;
fieldTypeHasInt128Field = defType.IsInt128OrHasInt128Fields;
+ fieldTypeHasVectorTField = defType.IsVectorTOrHasVectorTFields;
}
else
{
diff --git a/src/coreclr/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs b/src/coreclr/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs
index cb80143d94250d..3aa6ca2db11a16 100644
--- a/src/coreclr/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs
+++ b/src/coreclr/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs
@@ -422,6 +422,12 @@ internal static MarshallerKind GetMarshallerKind(
return MarshallerKind.Invalid;
}
+ if (!isField && ((DefType)type).IsVectorTOrHasVectorTFields)
+ {
+ // Vector types or structs that contain them cannot be passed by value
+ return MarshallerKind.Invalid;
+ }
+
if (MarshalUtils.IsBlittableType(type))
{
if (nativeType != NativeTypeKind.Default && nativeType != NativeTypeKind.Struct)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj b/src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj
index 3ddbde70bd2f08..4512bbfd88e7d0 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj
@@ -17,7 +17,6 @@
-
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/CompilerGeneratedInteropStubManager.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/CompilerGeneratedInteropStubManager.cs
index b56ca367618e9a..16cb8dcc6e8bb8 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/CompilerGeneratedInteropStubManager.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/CompilerGeneratedInteropStubManager.cs
@@ -32,10 +32,10 @@ public sealed override PInvokeILProvider CreatePInvokeILProvider()
public sealed override void AddToReadyToRunHeader(ReadyToRunHeaderNode header, NodeFactory nodeFactory, ExternalReferencesTableNode commonFixupsTableNode)
{
var delegateMapNode = new DelegateMarshallingStubMapNode(commonFixupsTableNode, _interopStateManager);
- header.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.DelegateMarshallingStubMap), delegateMapNode, delegateMapNode, delegateMapNode.EndSymbol);
+ header.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.DelegateMarshallingStubMap), delegateMapNode);
var structMapNode = new StructMarshallingStubMapNode(commonFixupsTableNode, _interopStateManager);
- header.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.StructMarshallingStubMap), structMapNode, structMapNode, structMapNode.EndSymbol);
+ header.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.StructMarshallingStubMap), structMapNode);
}
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs
index d32757d162d80b..c2132ce7aa6d1f 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs
@@ -239,30 +239,24 @@ private void ReportWarningsForReflectionAccess(in MessageOrigin origin, TypeSyst
// This is because reflection access is actually problematic on all members which are in a "requires" scope
// so for example even instance methods. See for example https://github.com/dotnet/linker/issues/3140 - it's possible
// to call a method on a "null" instance via reflection.
- if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresUnreferencedCodeAttribute, out CustomAttributeValue? requiresAttribute))
- {
- if (!ShouldSkipWarningsForOverride(entity, accessKind))
+ if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresUnreferencedCodeAttribute, out CustomAttributeValue? requiresAttribute) &&
+ ShouldProduceRequiresWarningForReflectionAccess(entity, accessKind))
ReportRequires(origin, entity, DiagnosticUtilities.RequiresUnreferencedCodeAttribute, requiresAttribute.Value);
- }
- if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresAssemblyFilesAttribute, out requiresAttribute))
- {
- if (!ShouldSkipWarningsForOverride(entity, accessKind))
+ if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresAssemblyFilesAttribute, out requiresAttribute) &&
+ ShouldProduceRequiresWarningForReflectionAccess(entity, accessKind))
ReportRequires(origin, entity, DiagnosticUtilities.RequiresAssemblyFilesAttribute, requiresAttribute.Value);
- }
- if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresDynamicCodeAttribute, out requiresAttribute))
- {
- if (!ShouldSkipWarningsForOverride(entity, accessKind))
+ if (_logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresDynamicCodeAttribute, out requiresAttribute) &&
+ ShouldProduceRequiresWarningForReflectionAccess(entity, accessKind))
ReportRequires(origin, entity, DiagnosticUtilities.RequiresDynamicCodeAttribute, requiresAttribute.Value);
- }
// Below is about accessing DAM annotated members, so only RUC is applicable as a suppression scope
if (_logger.ShouldSuppressAnalysisWarningsForRequires(origin.MemberDefinition, DiagnosticUtilities.RequiresUnreferencedCodeAttribute))
return;
bool isReflectionAccessCoveredByDAM = Annotations.ShouldWarnWhenAccessedForReflection(entity);
- if (isReflectionAccessCoveredByDAM && !ShouldSkipWarningsForOverride(entity, accessKind))
+ if (isReflectionAccessCoveredByDAM && ShouldProduceRequiresWarningForReflectionAccess(entity, accessKind))
{
if (entity is MethodDesc)
_logger.LogWarning(origin, DiagnosticId.DynamicallyAccessedMembersMethodAccessedViaReflection, entity.GetDisplayName());
@@ -273,16 +267,16 @@ private void ReportWarningsForReflectionAccess(in MessageOrigin origin, TypeSyst
// We decided to not warn on reflection access to compiler-generated methods:
// https://github.com/dotnet/runtime/issues/85042
- // All override methods should have the same annotations as their base methods
- // (else we will produce warning IL2046 or IL2092 or some other warning).
- // When marking override methods via DynamicallyAccessedMembers, we should only issue a warning for the base method.
- // PERF: Avoid precomputing this as this method is relatively expensive. Only call it once we're about to produce a warning.
- static bool ShouldSkipWarningsForOverride(TypeSystemEntity entity, AccessKind accessKind)
+ static bool ShouldProduceRequiresWarningForReflectionAccess(TypeSystemEntity entity, AccessKind accessKind)
{
- if (accessKind != AccessKind.DynamicallyAccessedMembersMark || entity is not MethodDesc method || !method.IsVirtual)
- return false;
+ bool isCompilerGenerated = CompilerGeneratedState.IsNestedFunctionOrStateMachineMember(entity);
- return MetadataVirtualMethodAlgorithm.FindSlotDefiningMethodForVirtualMethod(method) != method;
+ // Compiler generated code accessed via a token is considered a "hard" reference
+ // even though we also have to treat it as reflection access.
+ // So we need to enforce RUC check/warn in this case.
+ bool forceRequiresWarning = accessKind == AccessKind.TokenAccess;
+
+ return !isCompilerGenerated || forceRequiresWarning;
}
}
@@ -319,7 +313,8 @@ static bool IsDeclaredWithinType(TypeSystemEntity member, TypeDesc type)
// causing problems is pretty low.
bool isReflectionAccessCoveredByRUC = _logger.ShouldSuppressAnalysisWarningsForRequires(entity, DiagnosticUtilities.RequiresUnreferencedCodeAttribute, out CustomAttributeValue? requiresUnreferencedCodeAttribute);
- if (isReflectionAccessCoveredByRUC && !ShouldSkipWarningsForOverride(entity))
+ bool isCompilerGenerated = CompilerGeneratedState.IsNestedFunctionOrStateMachineMember(entity);
+ if (isReflectionAccessCoveredByRUC && !isCompilerGenerated)
{
var id = reportOnMember ? DiagnosticId.DynamicallyAccessedMembersOnTypeReferencesMemberWithRequiresUnreferencedCode : DiagnosticId.DynamicallyAccessedMembersOnTypeReferencesMemberOnBaseWithRequiresUnreferencedCode;
_logger.LogWarning(origin, id, _typeHierarchyDataFlowOrigin.GetDisplayName(),
@@ -329,7 +324,7 @@ static bool IsDeclaredWithinType(TypeSystemEntity member, TypeDesc type)
}
bool isReflectionAccessCoveredByDAM = Annotations.ShouldWarnWhenAccessedForReflection(entity);
- if (isReflectionAccessCoveredByDAM && !ShouldSkipWarningsForOverride(entity))
+ if (isReflectionAccessCoveredByDAM && !isCompilerGenerated)
{
var id = reportOnMember ? DiagnosticId.DynamicallyAccessedMembersOnTypeReferencesMemberWithDynamicallyAccessedMembers : DiagnosticId.DynamicallyAccessedMembersOnTypeReferencesMemberOnBaseWithDynamicallyAccessedMembers;
_logger.LogWarning(origin, id, _typeHierarchyDataFlowOrigin.GetDisplayName(), entity.GetDisplayName());
@@ -337,17 +332,6 @@ static bool IsDeclaredWithinType(TypeSystemEntity member, TypeDesc type)
// We decided to not warn on reflection access to compiler-generated methods:
// https://github.com/dotnet/runtime/issues/85042
-
- // All override methods should have the same annotations as their base methods
- // (else we will produce warning IL2046 or IL2092 or some other warning).
- // When marking override methods via DynamicallyAccessedMembers, we should only issue a warning for the base method.
- static bool ShouldSkipWarningsForOverride(TypeSystemEntity entity)
- {
- if (entity is not MethodDesc method || !method.IsVirtual)
- return false;
-
- return MetadataVirtualMethodAlgorithm.FindSlotDefiningMethodForVirtualMethod(method) != method;
- }
}
private void ReportRequires(in MessageOrigin origin, TypeSystemEntity entity, string requiresAttributeName, in CustomAttributeValue requiresAttribute)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayMapNode.cs
index 6bf674b28296a4..f8207e8454f9c5 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayMapNode.cs
@@ -12,18 +12,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of array types generated into the image.
///
- internal sealed class ArrayMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ArrayMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
- private ExternalReferencesTableNode _externalReferences;
+ private readonly ExternalReferencesTableNode _externalReferences;
+ private int? _size;
public ArrayMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__array_type_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -68,9 +67,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
index 2e75de7ead91e9..f63dab431e7190 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
@@ -5,24 +5,22 @@
namespace ILCompiler.DependencyAnalysis
{
- public interface IHasStartSymbol
- {
- ObjectAndOffsetSymbolNode StartSymbol { get; }
- }
-
///
/// Represents an array of nodes. The contents of this node will be emitted
/// by placing a starting symbol, followed by contents of nodes (optionally
/// sorted using provided comparer), followed by ending symbol.
///
- public class ArrayOfEmbeddedDataNode : EmbeddedDataContainerNode, IHasStartSymbol
+ public class ArrayOfEmbeddedDataNode : EmbeddedDataContainerNode, INodeWithSize
where TEmbedded : EmbeddedObjectNode
{
+ private int? _size;
private HashSet _nestedNodes = new HashSet();
private List _nestedNodesList = new List();
private IComparer _sorter;
- public ArrayOfEmbeddedDataNode(string startSymbolMangledName, string endSymbolMangledName, IComparer nodeSorter) : base(startSymbolMangledName, endSymbolMangledName)
+ int INodeWithSize.Size => _size.Value;
+
+ public ArrayOfEmbeddedDataNode(string mangledName, IComparer nodeSorter) : base(mangledName)
{
_sorter = nodeSorter;
}
@@ -35,11 +33,10 @@ public void AddEmbeddedObject(TEmbedded symbol)
{
_nestedNodesList.Add(symbol);
}
- symbol.ContainingNode = this;
}
}
- protected override string GetName(NodeFactory factory) => $"Region {StartSymbol.GetMangledName(factory.NameMangler)}";
+ protected override string GetName(NodeFactory factory) => $"Region {this.GetMangledName(factory.NameMangler)}";
public override ObjectNodeSection GetSection(NodeFactory factory) => ObjectNodeSection.DataSection;
public override bool IsShareable => false;
@@ -84,12 +81,11 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly)
if (_sorter != null)
_nestedNodesList.Sort(_sorter);
- builder.AddSymbol(StartSymbol);
+ builder.AddSymbol(this);
GetElementDataForNodes(ref builder, factory, relocsOnly);
- EndSymbol.SetSymbolOffset(builder.CountBytes);
- builder.AddSymbol(EndSymbol);
+ _size = builder.CountBytes;
ObjectData objData = builder.ToObjectData();
return objData;
@@ -100,15 +96,6 @@ public override bool ShouldSkipEmittingObjectNode(NodeFactory factory)
return _nestedNodesList.Count == 0;
}
- protected override DependencyList ComputeNonRelocationBasedDependencies(NodeFactory factory)
- {
- DependencyList dependencies = new DependencyList();
- dependencies.Add(StartSymbol, "StartSymbol");
- dependencies.Add(EndSymbol, "EndSymbol");
-
- return dependencies;
- }
-
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
public override int ClassCode => (int)ObjectNodeOrder.ArrayOfEmbeddedDataNode;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
index ba9e73eccfbfee..a2bc613dcbd4f6 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
@@ -24,13 +24,12 @@ public sealed class ArrayOfEmbeddedPointersNode : ArrayOfEmbeddedDataNo
///
public delegate void OnMarkedDelegate(EmbeddedPointerIndirectionNode embeddedObject);
- public ArrayOfEmbeddedPointersNode(string startSymbolMangledName, string endSymbolMangledName, IComparer nodeSorter)
+ public ArrayOfEmbeddedPointersNode(string mangledName, IComparer nodeSorter)
: base(
- startSymbolMangledName,
- endSymbolMangledName,
+ mangledName,
nodeSorter != null ? new PointerIndirectionNodeComparer(nodeSorter) : null)
{
- _startSymbolMangledName = startSymbolMangledName;
+ _startSymbolMangledName = mangledName;
}
public EmbeddedObjectNode NewNode(TTarget target)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfFrozenObjectsNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfFrozenObjectsNode.cs
index a83b93447fa67e..e6ca20b6507d27 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfFrozenObjectsNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ArrayOfFrozenObjectsNode.cs
@@ -8,15 +8,10 @@
namespace ILCompiler.DependencyAnalysis
{
- public class ArrayOfFrozenObjectsNode : DehydratableObjectNode, ISymbolDefinitionNode
+ public class ArrayOfFrozenObjectsNode : DehydratableObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
- public ISymbolNode EndSymbol => _endSymbol;
-
- public ArrayOfFrozenObjectsNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__FrozenSegmentEnd", true);
- }
+ private int? _size;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
=> sb.Append(nameMangler.CompilationUnitPrefix).Append("__FrozenSegmentStart");
@@ -61,8 +56,7 @@ protected override ObjectData GetDehydratableData(NodeFactory factory, bool relo
AlignNextObject(ref builder, factory);
builder.EmitZeroPointer();
- _endSymbol.SetSymbolOffset(builder.CountBytes);
- builder.AddSymbol(_endSymbol);
+ _size = builder.CountBytes;
return builder.ToObjectData();
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ByRefTypeMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ByRefTypeMapNode.cs
index 8adf1643302b73..b1a97f9d5bf7f1 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ByRefTypeMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ByRefTypeMapNode.cs
@@ -11,18 +11,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of ByRef types generated into the image.
///
- internal sealed class ByRefTypeMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ByRefTypeMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly ExternalReferencesTableNode _externalReferences;
public ByRefTypeMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__byref_type_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -62,9 +61,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ClassConstructorContextMap.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ClassConstructorContextMap.cs
index 2f84a816e0fffb..ec0e162367693f 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ClassConstructorContextMap.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ClassConstructorContextMap.cs
@@ -11,18 +11,17 @@
namespace ILCompiler.DependencyAnalysis
{
- internal sealed class ClassConstructorContextMap : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ClassConstructorContextMap : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public ClassConstructorContextMap(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__type_to_cctorContext_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -74,9 +73,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DehydratedDataNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DehydratedDataNode.cs
index fcaf3c4ea9a3be..4e526961690e17 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DehydratedDataNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DehydratedDataNode.cs
@@ -28,14 +28,9 @@ namespace ILCompiler.DependencyAnalysis
/// * Generate N bytes of zeros.
/// * Generate a relocation to Nth entry in the lookup table that supplements the dehydrated stream.
///
- internal sealed class DehydratedDataNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class DehydratedDataNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
-
- public DehydratedDataNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__dehydrated_data_End", true);
- }
+ private int? _size;
public override bool IsShareable => false;
@@ -47,7 +42,7 @@ public DehydratedDataNode()
public int Offset => 0;
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -87,7 +82,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
if (firstSymbol != null)
builder.EmitReloc(firstSymbol, RelocType.IMAGE_REL_BASED_RELPTR32, -firstSymbol.Offset);
else
- return new ObjectData(Array.Empty(), Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(Array.Empty(), Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
// Sort the reloc targets and create reloc lookup table.
KeyValuePair[] relocSort = new List>(relocOccurences).ToArray();
@@ -312,8 +307,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
dehydratedSegmentPosition += o.Data.Length;
}
- _endSymbol.SetSymbolOffset(builder.CountBytes);
- builder.AddSymbol(_endSymbol);
+ _size = builder.CountBytes;
// Dehydrated data is followed by the reloc lookup table.
for (int i = 0; i < relocSort.Length; i++)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DelegateMarshallingStubMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DelegateMarshallingStubMapNode.cs
index 1176f0e1bf4a5a..2111f081a697d9 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DelegateMarshallingStubMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/DelegateMarshallingStubMapNode.cs
@@ -12,20 +12,19 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of delegate marshalling stub types generated into the image.
///
- internal sealed class DelegateMarshallingStubMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class DelegateMarshallingStubMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly ExternalReferencesTableNode _externalReferences;
private readonly InteropStateManager _interopStateManager;
public DelegateMarshallingStubMapNode(ExternalReferencesTableNode externalReferences, InteropStateManager interopStateManager)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__delegate_marshalling_stub_map_End", true);
_externalReferences = externalReferences;
_interopStateManager = interopStateManager;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -71,9 +70,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
index 511f91cbb56f22..33b7594383c49a 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
@@ -16,8 +16,6 @@ public abstract class EmbeddedObjectNode : SortableDependencyNode
private int _offset;
private int _index;
- public IHasStartSymbol ContainingNode { get; set; }
-
public EmbeddedObjectNode()
{
_offset = InvalidOffset;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExactMethodInstantiationsNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExactMethodInstantiationsNode.cs
index 3d52e0a3a9c22a..46d38a7b9857b1 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExactMethodInstantiationsNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExactMethodInstantiationsNode.cs
@@ -13,14 +13,13 @@ namespace ILCompiler.DependencyAnalysis
///
/// Hashtable of all exact (non-canonical) generic method instantiations compiled in the module.
///
- public sealed class ExactMethodInstantiationsNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class ExactMethodInstantiationsNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public ExactMethodInstantiationsNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__exact_method_instantiations_End", true);
_externalReferences = externalReferences;
}
@@ -29,7 +28,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__exact_method_instantiations");
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -99,9 +98,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
public static void GetExactMethodInstantiationDependenciesForMethod(ref DependencyList dependencies, NodeFactory factory, MethodDesc method)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExternalReferencesTableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExternalReferencesTableNode.cs
index 3c2c3d29289724..86e2d14a0f3c29 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExternalReferencesTableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ExternalReferencesTableNode.cs
@@ -13,9 +13,9 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a node that points to various symbols and can be sequentially addressed.
///
- public sealed class ExternalReferencesTableNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class ExternalReferencesTableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly string _blobName;
private readonly NodeFactory _nodeFactory;
@@ -25,11 +25,10 @@ public sealed class ExternalReferencesTableNode : ObjectNode, ISymbolDefinitionN
public ExternalReferencesTableNode(string blobName, NodeFactory nodeFactory)
{
_blobName = blobName;
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__external_" + blobName + "_references_End", true);
_nodeFactory = nodeFactory;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -103,10 +102,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
}
}
- _endSymbol.SetSymbolOffset(builder.CountBytes);
+ _size = builder.CountBytes;
builder.AddSymbol(this);
- builder.AddSymbol(_endSymbol);
return builder.ToObjectData();
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/FunctionPointerMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/FunctionPointerMapNode.cs
index f2651f9069f1bf..979b8d4ef618a7 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/FunctionPointerMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/FunctionPointerMapNode.cs
@@ -12,18 +12,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of function pointer types generated into the image.
///
- internal sealed class FunctionPointerMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class FunctionPointerMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly ExternalReferencesTableNode _externalReferences;
public FunctionPointerMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__fnptr_type_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -71,9 +70,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsHashtableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsHashtableNode.cs
index 9e69b3d98d6770..aeae01fa7cd2c6 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsHashtableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsHashtableNode.cs
@@ -13,14 +13,13 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hashtable of all compiled generic method instantiations
///
- public sealed class GenericMethodsHashtableNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class GenericMethodsHashtableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public GenericMethodsHashtableNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__generic_methods_hashtable_End", true);
_externalReferences = externalReferences;
}
@@ -29,7 +28,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__generic_methods_hashtable");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -92,9 +91,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
public static void GetGenericMethodsHashtableDependenciesForMethod(ref DependencyList dependencies, NodeFactory factory, MethodDesc method)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsTemplateMap.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsTemplateMap.cs
index 971c81d1ef89db..6230518cf2b258 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsTemplateMap.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericMethodsTemplateMap.cs
@@ -13,14 +13,13 @@ namespace ILCompiler.DependencyAnalysis
///
/// Hashtable of all generic method templates used by the TypeLoader at runtime
///
- public sealed class GenericMethodsTemplateMap : ObjectNode, ISymbolDefinitionNode
+ public sealed class GenericMethodsTemplateMap : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public GenericMethodsTemplateMap(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__GenericMethodsTemplateMap_End", true);
_externalReferences = externalReferences;
}
@@ -29,7 +28,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__GenericMethodsTemplateMap");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -73,9 +72,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
public static void GetTemplateMethodDependencies(ref DependencyList dependencies, NodeFactory factory, MethodDesc method)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesHashtableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesHashtableNode.cs
index c1f06e34ef5a3d..fa1a70b3544acc 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesHashtableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesHashtableNode.cs
@@ -12,14 +12,13 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hashtable of all compiled generic type instantiations
///
- public sealed class GenericTypesHashtableNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class GenericTypesHashtableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public GenericTypesHashtableNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__generic_types_hashtable_End", true);
_externalReferences = externalReferences;
}
@@ -28,7 +27,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__generic_types_hashtable");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -68,9 +67,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesTemplateMap.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesTemplateMap.cs
index 9a329b0a74140d..80f44e60948d45 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesTemplateMap.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericTypesTemplateMap.cs
@@ -13,14 +13,13 @@ namespace ILCompiler.DependencyAnalysis
///
/// Hashtable of all generic type templates used by the TypeLoader at runtime
///
- public sealed class GenericTypesTemplateMap : ObjectNode, ISymbolDefinitionNode
+ public sealed class GenericTypesTemplateMap : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public GenericTypesTemplateMap(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__GenericTypesTemplateMap_End", true);
_externalReferences = externalReferences;
}
@@ -29,7 +28,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__GenericTypesTemplateMap");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -68,9 +67,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
public static void GetTemplateTypeDependencies(ref DependencyList dependencies, NodeFactory factory, TypeDesc type)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericVirtualMethodTableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericVirtualMethodTableNode.cs
index c1847c5d06f5f4..7769b2b6a60872 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericVirtualMethodTableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/GenericVirtualMethodTableNode.cs
@@ -14,15 +14,14 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a map of generic virtual method implementations.
///
- public sealed class GenericVirtualMethodTableNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class GenericVirtualMethodTableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
private Dictionary> _gvmImplementations;
public GenericVirtualMethodTableNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__gvm_table_End", true);
_externalReferences = externalReferences;
_gvmImplementations = new Dictionary>();
}
@@ -32,7 +31,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__gvm_table");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -135,9 +134,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeFormatWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/InterfaceGenericVirtualMethodTableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/InterfaceGenericVirtualMethodTableNode.cs
index b5855ce04e43d2..74de063bbd706d 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/InterfaceGenericVirtualMethodTableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/InterfaceGenericVirtualMethodTableNode.cs
@@ -15,16 +15,15 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a map between reflection metadata and generated method bodies.
///
- public sealed class InterfaceGenericVirtualMethodTableNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class InterfaceGenericVirtualMethodTableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
private Dictionary> _interfaceGvmSlots;
private Dictionary>> _interfaceImpls;
public InterfaceGenericVirtualMethodTableNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__interface_gvm_table_End", true);
_externalReferences = externalReferences;
_interfaceGvmSlots = new Dictionary>();
_interfaceImpls = new Dictionary>>();
@@ -34,7 +33,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
sb.Append(nameMangler.CompilationUnitPrefix).Append("__interface_gvm_table");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -225,9 +224,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] streamBytes = nativeFormatWriter.Save();
- _endSymbol.SetSymbolOffset(streamBytes.Length);
+ _size = streamBytes.Length;
- return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(streamBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs
index d17ecfb8126d8b..ad84d1bd81ed6a 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs
@@ -11,16 +11,11 @@ namespace ILCompiler.DependencyAnalysis
/// Represents a blob of native metadata describing assemblies, the types in them, and their members.
/// The data is used at runtime to e.g. support reflection.
///
- public sealed class MetadataNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class MetadataNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
- public MetadataNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__embedded_metadata_End", true);
- }
-
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -42,7 +37,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
return new ObjectData(Array.Empty(), Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
byte[] blob = factory.MetadataManager.GetMetadataBlob(factory);
- _endSymbol.SetSymbolOffset(blob.Length);
+ _size = blob.Length;
return new ObjectData(
blob,
@@ -50,8 +45,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
1,
new ISymbolDefinitionNode[]
{
- this,
- _endSymbol
+ this
});
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ModuleInitializerListNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ModuleInitializerListNode.cs
index fe53861e70ca44..185274addeb0d9 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ModuleInitializerListNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ModuleInitializerListNode.cs
@@ -10,16 +10,11 @@
namespace ILCompiler.DependencyAnalysis
{
- internal sealed class ModuleInitializerListNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ModuleInitializerListNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
- public ModuleInitializerListNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__module_initializers_End", true);
- }
-
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -137,7 +132,6 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
ObjectDataBuilder builder = new ObjectDataBuilder(factory, relocsOnly);
builder.RequireInitialAlignment(factory.Target.PointerSize);
builder.AddSymbol(this);
- builder.AddSymbol(_endSymbol);
foreach (var module in sortedModules)
{
@@ -150,7 +144,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
var result = builder.ToObjectData();
- _endSymbol.SetSymbolOffset(result.Data.Length);
+ _size = result.Data.Length;
return result;
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NativeLayoutInfoNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NativeLayoutInfoNode.cs
index e9bc60848daf05..95e73168567505 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NativeLayoutInfoNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NativeLayoutInfoNode.cs
@@ -12,9 +12,9 @@ namespace ILCompiler.DependencyAnalysis
///
/// Native layout info blob.
///
- public sealed class NativeLayoutInfoNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class NativeLayoutInfoNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
private ExternalReferencesTableNode _staticsReferences;
@@ -29,7 +29,6 @@ public sealed class NativeLayoutInfoNode : ObjectNode, ISymbolDefinitionNode
public NativeLayoutInfoNode(ExternalReferencesTableNode externalReferences, ExternalReferencesTableNode staticsReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__nativelayoutinfo_End", true);
_externalReferences = externalReferences;
_staticsReferences = staticsReferences;
@@ -45,7 +44,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
sb.Append(nameMangler.CompilationUnitPrefix).Append("__nativelayoutinfo");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -87,9 +86,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
SaveNativeLayoutInfoWriter(factory);
- _endSymbol.SetSymbolOffset(_writerSavedBytes.Length);
+ _size = _writerSavedBytes.Length;
- return new ObjectData(_writerSavedBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(_writerSavedBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs
index 53b44d32b142c7..5d734957ff6e75 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs
@@ -1250,18 +1250,15 @@ public string GetSymbolAlternateName(ISymbolNode node)
}
public ArrayOfEmbeddedPointersNode GCStaticsRegion = new ArrayOfEmbeddedPointersNode(
- "__GCStaticRegionStart",
- "__GCStaticRegionEnd",
+ "__GCStaticRegion",
new SortableDependencyNode.ObjectNodeComparer(CompilerComparer.Instance));
public ArrayOfEmbeddedDataNode ThreadStaticsRegion = new ArrayOfEmbeddedDataNode(
- "__ThreadStaticRegionStart",
- "__ThreadStaticRegionEnd",
+ "__ThreadStaticRegion",
new SortableDependencyNode.EmbeddedObjectNodeComparer(CompilerComparer.Instance));
public ArrayOfEmbeddedPointersNode EagerCctorTable = new ArrayOfEmbeddedPointersNode(
- "__EagerCctorStart",
- "__EagerCctorEnd",
+ "__EagerCctor",
null);
public ArrayOfFrozenObjectsNode FrozenSegmentRegion = new ArrayOfFrozenObjectsNode();
@@ -1296,18 +1293,18 @@ public virtual void AttachToDependencyGraph(DependencyAnalyzerBase
graph.AddRoot(_inlinedThreadStatiscNode, "Inlined threadstatics are used if present");
}
- ReadyToRunHeader.Add(ReadyToRunSectionType.GCStaticRegion, GCStaticsRegion, GCStaticsRegion.StartSymbol, GCStaticsRegion.EndSymbol);
- ReadyToRunHeader.Add(ReadyToRunSectionType.ThreadStaticRegion, ThreadStaticsRegion, ThreadStaticsRegion.StartSymbol, ThreadStaticsRegion.EndSymbol);
- ReadyToRunHeader.Add(ReadyToRunSectionType.EagerCctor, EagerCctorTable, EagerCctorTable.StartSymbol, EagerCctorTable.EndSymbol);
- ReadyToRunHeader.Add(ReadyToRunSectionType.TypeManagerIndirection, TypeManagerIndirection, TypeManagerIndirection);
- ReadyToRunHeader.Add(ReadyToRunSectionType.FrozenObjectRegion, FrozenSegmentRegion, FrozenSegmentRegion, FrozenSegmentRegion.EndSymbol);
- ReadyToRunHeader.Add(ReadyToRunSectionType.ModuleInitializerList, ModuleInitializerList, ModuleInitializerList, ModuleInitializerList.EndSymbol);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.GCStaticRegion, GCStaticsRegion);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.ThreadStaticRegion, ThreadStaticsRegion);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.EagerCctor, EagerCctorTable);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.TypeManagerIndirection, TypeManagerIndirection);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.FrozenObjectRegion, FrozenSegmentRegion);
+ ReadyToRunHeader.Add(ReadyToRunSectionType.ModuleInitializerList, ModuleInitializerList);
var commonFixupsTableNode = new ExternalReferencesTableNode("CommonFixupsTable", this);
InteropStubManager.AddToReadyToRunHeader(ReadyToRunHeader, this, commonFixupsTableNode);
MetadataManager.AddToReadyToRunHeader(ReadyToRunHeader, this, commonFixupsTableNode);
MetadataManager.AttachToDependencyGraph(graph);
- ReadyToRunHeader.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.CommonFixupsTable), commonFixupsTableNode, commonFixupsTableNode, commonFixupsTableNode.EndSymbol);
+ ReadyToRunHeader.Add(MetadataManager.BlobIdToReadyToRunSection(ReflectionMapBlob.CommonFixupsTable), commonFixupsTableNode);
}
protected struct MethodKey : IEquatable
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/PointerTypeMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/PointerTypeMapNode.cs
index 497c1f58092cb0..bf2ecfb4c71616 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/PointerTypeMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/PointerTypeMapNode.cs
@@ -11,18 +11,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of pointer types generated into the image.
///
- internal sealed class PointerTypeMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class PointerTypeMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly ExternalReferencesTableNode _externalReferences;
public PointerTypeMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__ptr_type_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -62,9 +61,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReadyToRunHeaderNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReadyToRunHeaderNode.cs
index 199b5a1e342dd6..e92c6804bbef2b 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReadyToRunHeaderNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReadyToRunHeaderNode.cs
@@ -5,7 +5,8 @@
using Internal.Runtime;
using Internal.Text;
-using Internal.TypeSystem;
+
+using Debug = System.Diagnostics.Debug;
namespace ILCompiler.DependencyAnalysis
{
@@ -13,25 +14,21 @@ public class ReadyToRunHeaderNode : ObjectNode, ISymbolDefinitionNode
{
private struct HeaderItem
{
- public HeaderItem(ReadyToRunSectionType id, ObjectNode node, ISymbolNode startSymbol, ISymbolNode endSymbol)
+ public HeaderItem(ReadyToRunSectionType id, ObjectNode node)
{
Id = id;
Node = node;
- StartSymbol = startSymbol;
- EndSymbol = endSymbol;
}
public readonly ReadyToRunSectionType Id;
public readonly ObjectNode Node;
- public readonly ISymbolNode StartSymbol;
- public readonly ISymbolNode EndSymbol;
}
private List _items = new List();
- public void Add(ReadyToRunSectionType id, ObjectNode node, ISymbolNode startSymbol, ISymbolNode endSymbol = null)
+ public void Add(ReadyToRunSectionType id, T node) where T : ObjectNode, ISymbolDefinitionNode
{
- _items.Add(new HeaderItem(id, node, startSymbol, endSymbol));
+ _items.Add(new HeaderItem(id, node));
}
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
@@ -93,17 +90,17 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
builder.EmitInt((int)item.Id);
ModuleInfoFlags flags = 0;
- if (item.EndSymbol != null)
+ if (item.Node is INodeWithSize)
{
flags |= ModuleInfoFlags.HasEndPointer;
}
builder.EmitInt((int)flags);
- builder.EmitPointerReloc(item.StartSymbol);
+ builder.EmitPointerReloc((ISymbolNode)item.Node);
- if (item.EndSymbol != null)
+ if (!relocsOnly && item.Node is INodeWithSize nodeWithSize)
{
- builder.EmitPointerReloc(item.EndSymbol);
+ builder.EmitPointerReloc((ISymbolNode)item.Node, nodeWithSize.Size);
}
else
{
@@ -117,6 +114,12 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
return builder.ToObjectData();
}
- public override int ClassCode => -534800244;
+ protected internal override int Phase => (int)ObjectNodePhase.Late;
+ public override int ClassCode => 0x7db08464;
+ }
+
+ public interface INodeWithSize
+ {
+ public int Size { get; }
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionFieldMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionFieldMapNode.cs
index 3044fd4d979d07..8ea831f1efdb8f 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionFieldMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionFieldMapNode.cs
@@ -14,18 +14,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a map between reflection metadata and native field offsets.
///
- internal sealed class ReflectionFieldMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ReflectionFieldMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public ReflectionFieldMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__field_to_offset_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -183,9 +182,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs
index 171a3f83ac227e..b46f2303c20f03 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs
@@ -15,24 +15,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a map between reflection metadata and generated method bodies.
///
- internal sealed class ReflectionInvokeMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ReflectionInvokeMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public ReflectionInvokeMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__method_to_entrypoint_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolNode EndSymbol
- {
- get
- {
- return _endSymbol;
- }
- }
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -240,9 +233,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionVirtualInvokeMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionVirtualInvokeMapNode.cs
index e28f18387bf954..9e62841554db2f 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionVirtualInvokeMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionVirtualInvokeMapNode.cs
@@ -17,14 +17,13 @@ namespace ILCompiler.DependencyAnalysis
/// Represents a map containing the necessary information needed to resolve
/// a virtual method target called through reflection.
///
- internal sealed class ReflectionVirtualInvokeMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ReflectionVirtualInvokeMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public ReflectionVirtualInvokeMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__VirtualInvokeMap_End", true);
_externalReferences = externalReferences;
}
@@ -33,7 +32,7 @@ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
sb.Append(nameMangler.CompilationUnitPrefix).Append("__VirtualInvokeMap");
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public int Offset => 0;
public override bool IsShareable => false;
public override ObjectNodeSection GetSection(NodeFactory factory) => _externalReferences.GetSection(factory);
@@ -207,9 +206,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceDataNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceDataNode.cs
index 903d4c21ffb9e1..6248fbd8c806c9 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceDataNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceDataNode.cs
@@ -18,21 +18,17 @@ namespace ILCompiler.DependencyAnalysis
/// Resources are simply copied from the inputs and concatenated into this blob.
/// All format information is provided by
///
- internal sealed class ResourceDataNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ResourceDataNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
+ private int? _size;
+
///
/// Resource index information generated while extracting resources into the data blob
///
private List _indexData;
private int _totalLength;
- public ResourceDataNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__embedded_resourcedata_End", true);
- }
-
- private ObjectAndOffsetSymbolNode _endSymbol;
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public override bool IsShareable => false;
@@ -62,8 +58,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
1,
new ISymbolDefinitionNode[]
{
- this,
- EndSymbol
+ this
});
}
@@ -141,7 +136,7 @@ private byte[] GenerateResourceBlob(NodeFactory factory)
currentPos += resourceData.Length;
}
- _endSymbol.SetSymbolOffset(resourceBlob.Length);
+ _size = resourceBlob.Length;
return resourceBlob;
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceIndexNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceIndexNode.cs
index 21504fd1e0e48e..6d312258a7ded0 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceIndexNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ResourceIndexNode.cs
@@ -11,19 +11,18 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of resources within the resource blob in the image.
///
- internal sealed class ResourceIndexNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class ResourceIndexNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
private ResourceDataNode _resourceDataNode;
public ResourceIndexNode(ResourceDataNode resourceDataNode)
{
_resourceDataNode = resourceDataNode;
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__embedded_resourceindex_End", true);
}
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public override bool IsShareable => false;
@@ -53,8 +52,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
1,
new ISymbolDefinitionNode[]
{
- this,
- EndSymbol
+ this
});
}
@@ -90,7 +88,7 @@ private byte[] GenerateIndexBlob(NodeFactory factory)
}
byte[] blob = nativeWriter.Save();
- _endSymbol.SetSymbolOffset(blob.Length);
+ _size = blob.Length;
return blob;
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StackTraceMethodMappingNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StackTraceMethodMappingNode.cs
index 9a277122333f42..2a7fb40f17c359 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StackTraceMethodMappingNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StackTraceMethodMappingNode.cs
@@ -10,15 +10,11 @@ namespace ILCompiler.DependencyAnalysis
///
/// BlobIdStackTraceMethodRvaToTokenMapping - list of 8-byte pairs (method RVA-method token)
///
- public sealed class StackTraceMethodMappingNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class StackTraceMethodMappingNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- public StackTraceMethodMappingNode()
- {
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "_stacktrace_methodRVA_to_token_mapping_End", true);
- }
+ private int? _size;
- private ObjectAndOffsetSymbolNode _endSymbol;
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public override bool IsShareable => false;
@@ -50,7 +46,6 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
ObjectDataBuilder objData = new ObjectDataBuilder(factory, relocsOnly);
objData.RequireInitialPointerAlignment();
objData.AddSymbol(this);
- objData.AddSymbol(_endSymbol);
foreach (var mappingEntry in factory.MetadataManager.GetStackTraceMapping(factory))
{
@@ -58,7 +53,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
objData.EmitInt(mappingEntry.MetadataHandle);
}
- _endSymbol.SetSymbolOffset(objData.CountBytes);
+ _size = objData.CountBytes;
return objData.ToObjectData();
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StaticsInfoHashtableNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StaticsInfoHashtableNode.cs
index bfa05c4dd69b0b..4a1d8d94f3e90e 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StaticsInfoHashtableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StaticsInfoHashtableNode.cs
@@ -13,20 +13,19 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hashtable with information about all statics regions for all compiled generic types.
///
- internal sealed class StaticsInfoHashtableNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class StaticsInfoHashtableNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
private ExternalReferencesTableNode _nativeStaticsReferences;
public StaticsInfoHashtableNode(ExternalReferencesTableNode externalReferences, ExternalReferencesTableNode nativeStaticsReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "_StaticsInfoHashtableNode_End", true);
_externalReferences = externalReferences;
_nativeStaticsReferences = nativeStaticsReferences;
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -108,9 +107,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StructMarshallingStubMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StructMarshallingStubMapNode.cs
index c2f5ddf791b9e8..ed0c278aa11504 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StructMarshallingStubMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/StructMarshallingStubMapNode.cs
@@ -14,20 +14,19 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a hash table of struct marshalling stub types generated into the image.
///
- internal sealed class StructMarshallingStubMapNode : ObjectNode, ISymbolDefinitionNode
+ internal sealed class StructMarshallingStubMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private readonly ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private readonly ExternalReferencesTableNode _externalReferences;
private readonly InteropStateManager _interopStateManager;
public StructMarshallingStubMapNode(ExternalReferencesTableNode externalReferences, InteropStateManager interopStateManager)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__struct_marshalling_stub_map_End", true);
_externalReferences = externalReferences;
_interopStateManager = interopStateManager;
}
- public ISymbolDefinitionNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -115,9 +114,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/TypeMetadataMapNode.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/TypeMetadataMapNode.cs
index 2c638a08bbdfe2..a23f5e4540dfaf 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/TypeMetadataMapNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/TypeMetadataMapNode.cs
@@ -11,18 +11,17 @@ namespace ILCompiler.DependencyAnalysis
///
/// Represents a map between EETypes and metadata records within the .
///
- public sealed class TypeMetadataMapNode : ObjectNode, ISymbolDefinitionNode
+ public sealed class TypeMetadataMapNode : ObjectNode, ISymbolDefinitionNode, INodeWithSize
{
- private ObjectAndOffsetSymbolNode _endSymbol;
+ private int? _size;
private ExternalReferencesTableNode _externalReferences;
public TypeMetadataMapNode(ExternalReferencesTableNode externalReferences)
{
- _endSymbol = new ObjectAndOffsetSymbolNode(this, 0, "__type_to_metadata_map_End", true);
_externalReferences = externalReferences;
}
- public ISymbolNode EndSymbol => _endSymbol;
+ int INodeWithSize.Size => _size.Value;
public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
{
@@ -70,9 +69,9 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
byte[] hashTableBytes = writer.Save();
- _endSymbol.SetSymbolOffset(hashTableBytes.Length);
+ _size = hashTableBytes.Length;
- return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this, _endSymbol });
+ return new ObjectData(hashTableBytes, Array.Empty(), 1, new ISymbolDefinitionNode[] { this });
}
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/FeatureSwitchManager.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/FeatureSwitchManager.cs
index 7d406ce0be2963..a1dfaeca37930c 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/FeatureSwitchManager.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/FeatureSwitchManager.cs
@@ -28,7 +28,13 @@ public class FeatureSwitchManager : ILProvider
public FeatureSwitchManager(ILProvider nestedILProvider, Logger logger, IEnumerable> switchValues)
{
_nestedILProvider = nestedILProvider;
- _hashtable = new FeatureSwitchHashtable(logger, new Dictionary(switchValues));
+
+ // Last setting wins
+ var dictionary = new Dictionary();
+ foreach ((string name, bool value) in switchValues)
+ dictionary[name] = value;
+
+ _hashtable = new FeatureSwitchHashtable(logger, dictionary);
}
private BodySubstitution GetSubstitution(MethodDesc method)
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs
index 1411ca7b6c7941..cd7a616c0698b3 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/HardwareIntrinsicHelpers.Aot.cs
@@ -111,6 +111,10 @@ private static class XArchIntrinsicConstants
public const int Avx512dq_vl = 0x400000;
public const int Avx512Vbmi = 0x800000;
public const int Avx512Vbmi_vl = 0x1000000;
+ public const int Serialize = 0x2000000;
+ public const int VectorT128 = 0x4000000;
+ public const int VectorT256 = 0x8000000;
+ public const int VectorT512 = 0x10000000;
public static int FromInstructionSet(InstructionSet instructionSet)
{
@@ -120,6 +124,7 @@ public static int FromInstructionSet(InstructionSet instructionSet)
return instructionSet switch
{
+ // Optional ISAs - only available via opt-in or opportunistic light-up
InstructionSet.X64_AES => Aes,
InstructionSet.X64_AES_X64 => Aes,
InstructionSet.X64_PCLMULQDQ => Pclmulqdq,
@@ -170,8 +175,10 @@ public static int FromInstructionSet(InstructionSet instructionSet)
InstructionSet.X64_AVX512VBMI_X64 => Avx512Vbmi,
InstructionSet.X64_AVX512VBMI_VL => Avx512Vbmi_vl,
InstructionSet.X64_AVX512VBMI_VL_X64 => Avx512Vbmi_vl,
+ InstructionSet.X64_X86Serialize => Serialize,
+ InstructionSet.X64_X86Serialize_X64 => Serialize,
- // SSE and SSE2 are baseline ISAs - they're always available
+ // Baseline ISAs - they're always available
InstructionSet.X64_SSE => 0,
InstructionSet.X64_SSE_X64 => 0,
InstructionSet.X64_SSE2 => 0,
@@ -180,6 +187,11 @@ public static int FromInstructionSet(InstructionSet instructionSet)
InstructionSet.X64_X86Base => 0,
InstructionSet.X64_X86Base_X64 => 0,
+ // Vector Sizes
+ InstructionSet.X64_VectorT128 => VectorT128,
+ InstructionSet.X64_VectorT256 => VectorT256,
+ InstructionSet.X64_VectorT512 => VectorT512,
+
_ => throw new NotSupportedException(((InstructionSet_X64)instructionSet).ToString())
};
}
@@ -196,13 +208,20 @@ private static class Arm64IntrinsicConstants
public const int Sha256 = 0x0040;
public const int Atomics = 0x0080;
public const int Rcpc = 0x0100;
+ public const int VectorT128 = 0x0200;
public static int FromInstructionSet(InstructionSet instructionSet)
{
return instructionSet switch
{
+
+ // Baseline ISAs - they're always available
+ InstructionSet.ARM64_ArmBase => 0,
+ InstructionSet.ARM64_ArmBase_Arm64 => 0,
InstructionSet.ARM64_AdvSimd => AdvSimd,
InstructionSet.ARM64_AdvSimd_Arm64 => AdvSimd,
+
+ // Optional ISAs - only available via opt-in or opportunistic light-up
InstructionSet.ARM64_Aes => Aes,
InstructionSet.ARM64_Aes_Arm64 => Aes,
InstructionSet.ARM64_Crc32 => Crc32,
@@ -218,8 +237,8 @@ public static int FromInstructionSet(InstructionSet instructionSet)
InstructionSet.ARM64_Atomics => Atomics,
InstructionSet.ARM64_Rcpc => Rcpc,
- InstructionSet.ARM64_ArmBase => 0,
- InstructionSet.ARM64_ArmBase_Arm64 => 0,
+ // Vector Sizes
+ InstructionSet.ARM64_VectorT128 => VectorT128,
_ => throw new NotSupportedException(((InstructionSet_ARM64)instructionSet).ToString())
};
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs
index 10d7ae4edf0335..26dc3ff32e73cc 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs
@@ -122,83 +122,83 @@ internal static ReadyToRunSectionType BlobIdToReadyToRunSection(ReflectionMapBlo
public virtual void AddToReadyToRunHeader(ReadyToRunHeaderNode header, NodeFactory nodeFactory, ExternalReferencesTableNode commonFixupsTableNode)
{
var metadataNode = new MetadataNode();
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.EmbeddedMetadata), metadataNode, metadataNode, metadataNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.EmbeddedMetadata), metadataNode);
var nativeReferencesTableNode = new ExternalReferencesTableNode("NativeReferences", nodeFactory);
var nativeStaticsTableNode = new ExternalReferencesTableNode("NativeStatics", nodeFactory);
var resourceDataNode = new ResourceDataNode();
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdResourceData), resourceDataNode, resourceDataNode, resourceDataNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdResourceData), resourceDataNode);
var resourceIndexNode = new ResourceIndexNode(resourceDataNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdResourceIndex), resourceIndexNode, resourceIndexNode, resourceIndexNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdResourceIndex), resourceIndexNode);
var typeMapNode = new TypeMetadataMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.TypeMap), typeMapNode, typeMapNode, typeMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.TypeMap), typeMapNode);
var cctorContextMapNode = new ClassConstructorContextMap(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.CCtorContextMap), cctorContextMapNode, cctorContextMapNode, cctorContextMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.CCtorContextMap), cctorContextMapNode);
var invokeMapNode = new ReflectionInvokeMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.InvokeMap), invokeMapNode, invokeMapNode, invokeMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.InvokeMap), invokeMapNode);
var arrayMapNode = new ArrayMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ArrayMap), arrayMapNode, arrayMapNode, arrayMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ArrayMap), arrayMapNode);
var byRefMapNode = new ByRefTypeMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ByRefTypeMap), byRefMapNode, byRefMapNode, byRefMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ByRefTypeMap), byRefMapNode);
var pointerMapNode = new PointerTypeMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.PointerTypeMap), pointerMapNode, pointerMapNode, pointerMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.PointerTypeMap), pointerMapNode);
var functionPointerMapNode = new FunctionPointerMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.FunctionPointerTypeMap), functionPointerMapNode, functionPointerMapNode, functionPointerMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.FunctionPointerTypeMap), functionPointerMapNode);
var fieldMapNode = new ReflectionFieldMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.FieldAccessMap), fieldMapNode, fieldMapNode, fieldMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.FieldAccessMap), fieldMapNode);
NativeLayoutInfo = new NativeLayoutInfoNode(nativeReferencesTableNode, nativeStaticsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeLayoutInfo), NativeLayoutInfo, NativeLayoutInfo, NativeLayoutInfo.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeLayoutInfo), NativeLayoutInfo);
var exactMethodInstantiations = new ExactMethodInstantiationsNode(nativeReferencesTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ExactMethodInstantiationsHashtable), exactMethodInstantiations, exactMethodInstantiations, exactMethodInstantiations.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.ExactMethodInstantiationsHashtable), exactMethodInstantiations);
var genericsTypesHashtableNode = new GenericTypesHashtableNode(nativeReferencesTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericsHashtable), genericsTypesHashtableNode, genericsTypesHashtableNode, genericsTypesHashtableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericsHashtable), genericsTypesHashtableNode);
var genericMethodsHashtableNode = new GenericMethodsHashtableNode(nativeReferencesTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericMethodsHashtable), genericMethodsHashtableNode, genericMethodsHashtableNode, genericMethodsHashtableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericMethodsHashtable), genericMethodsHashtableNode);
var genericVirtualMethodTableNode = new GenericVirtualMethodTableNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericVirtualMethodTable), genericVirtualMethodTableNode, genericVirtualMethodTableNode, genericVirtualMethodTableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericVirtualMethodTable), genericVirtualMethodTableNode);
var interfaceGenericVirtualMethodTableNode = new InterfaceGenericVirtualMethodTableNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.InterfaceGenericVirtualMethodTable), interfaceGenericVirtualMethodTableNode, interfaceGenericVirtualMethodTableNode, interfaceGenericVirtualMethodTableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.InterfaceGenericVirtualMethodTable), interfaceGenericVirtualMethodTableNode);
var genericMethodsTemplatesMapNode = new GenericMethodsTemplateMap(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericMethodsTemplateMap), genericMethodsTemplatesMapNode, genericMethodsTemplatesMapNode, genericMethodsTemplatesMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.GenericMethodsTemplateMap), genericMethodsTemplatesMapNode);
var genericTypesTemplatesMapNode = new GenericTypesTemplateMap(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.TypeTemplateMap), genericTypesTemplatesMapNode, genericTypesTemplatesMapNode, genericTypesTemplatesMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.TypeTemplateMap), genericTypesTemplatesMapNode);
var staticsInfoHashtableNode = new StaticsInfoHashtableNode(nativeReferencesTableNode, nativeStaticsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.StaticsInfoHashtable), staticsInfoHashtableNode, staticsInfoHashtableNode, staticsInfoHashtableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.StaticsInfoHashtable), staticsInfoHashtableNode);
var virtualInvokeMapNode = new ReflectionVirtualInvokeMapNode(commonFixupsTableNode);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.VirtualInvokeMap), virtualInvokeMapNode, virtualInvokeMapNode, virtualInvokeMapNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.VirtualInvokeMap), virtualInvokeMapNode);
var stackTraceMethodMappingNode = new StackTraceMethodMappingNode();
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdStackTraceMethodRvaToTokenMapping), stackTraceMethodMappingNode, stackTraceMethodMappingNode, stackTraceMethodMappingNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.BlobIdStackTraceMethodRvaToTokenMapping), stackTraceMethodMappingNode);
// The external references tables should go last
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeReferences), nativeReferencesTableNode, nativeReferencesTableNode, nativeReferencesTableNode.EndSymbol);
- header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeStatics), nativeStaticsTableNode, nativeStaticsTableNode, nativeStaticsTableNode.EndSymbol);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeReferences), nativeReferencesTableNode);
+ header.Add(BlobIdToReadyToRunSection(ReflectionMapBlob.NativeStatics), nativeStaticsTableNode);
if (IsDataDehydrated)
{
var dehydratedDataNode = new DehydratedDataNode();
- header.Add(ReadyToRunSectionType.DehydratedData, dehydratedDataNode, dehydratedDataNode, dehydratedDataNode.EndSymbol);
+ header.Add(ReadyToRunSectionType.DehydratedData, dehydratedDataNode);
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeConfigurationRootProvider.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeConfigurationRootProvider.cs
index 122067c0587103..de64f9eb277ce2 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeConfigurationRootProvider.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeConfigurationRootProvider.cs
@@ -2,6 +2,11 @@
// The .NET Foundation licenses this file to you under the MIT license.
using System.Collections.Generic;
+using System.Text;
+
+using ILCompiler.DependencyAnalysis;
+
+using Internal.Text;
namespace ILCompiler
{
@@ -11,47 +16,115 @@ namespace ILCompiler
///
public class RuntimeConfigurationRootProvider : ICompilationRootProvider
{
- private readonly IEnumerable _runtimeOptions;
+ private readonly string _blobName;
+ private readonly IReadOnlyCollection _runtimeOptions;
- public RuntimeConfigurationRootProvider(IEnumerable runtimeOptions)
+ public RuntimeConfigurationRootProvider(string blobName, IReadOnlyCollection runtimeOptions)
{
+ _blobName = blobName;
_runtimeOptions = runtimeOptions;
}
void ICompilationRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider)
{
- rootProvider.RootReadOnlyDataBlob(GetRuntimeOptionsBlob(), 4, "Runtime configuration information", "g_compilerEmbeddedSettingsBlob");
+ rootProvider.AddCompilationRoot(new RuntimeConfigurationBlobNode(_blobName, _runtimeOptions), "Runtime configuration");
}
- protected byte[] GetRuntimeOptionsBlob()
+ private sealed class RuntimeConfigurationBlobNode : ObjectNode, ISymbolDefinitionNode
{
- const int HeaderSize = 4;
+ private readonly string _blobName;
+ private readonly IReadOnlyCollection _runtimeOptions;
+
+ public RuntimeConfigurationBlobNode(string blobName, IReadOnlyCollection runtimeOptions)
+ {
+ _blobName = blobName;
+ _runtimeOptions = runtimeOptions;
+ }
- ArrayBuilder options = default(ArrayBuilder);
+ public int Offset => 0;
- // Reserve space for the header
- options.ZeroExtend(HeaderSize);
+ public override bool IsShareable => false;
- foreach (string option in _runtimeOptions)
- {
- byte[] optionBytes = System.Text.Encoding.ASCII.GetBytes(option);
- options.Append(optionBytes);
+ public override int ClassCode => 7864454;
- // Emit a null to separate the next option
- options.Add(0);
+ public override bool StaticDependenciesAreComputed => true;
+
+ public void AppendMangledName(NameMangler nameMangler, Utf8StringBuilder sb)
+ {
+ sb.Append(_blobName);
}
- byte[] result = options.ToArray();
+ public override ObjectNodeSection GetSection(NodeFactory factory) =>
+ factory.Target.IsWindows ? ObjectNodeSection.ReadOnlyDataSection : ObjectNodeSection.DataSection;
+
+ protected override string GetName(NodeFactory factory) => this.GetMangledName(factory.NameMangler);
+
+ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
+ {
+ var builder = new ObjectDataBuilder(factory.TypeSystemContext.Target, relocsOnly);
+ builder.AddSymbol(this);
- int length = options.Count - HeaderSize;
+ var settings = new Dictionary();
- // Encode the size of the blob into the header
- result[0] = (byte)length;
- result[1] = (byte)(length >> 8);
- result[2] = (byte)(length >> 0x10);
- result[3] = (byte)(length >> 0x18);
+ // Put values in a dictionary - we expect many "true" strings, for example.
+ var valueDict = new Dictionary();
+ int valueIndex = 0;
+ foreach (string line in _runtimeOptions)
+ {
+ int indexOfEquals = line.IndexOf("=");
+ if (indexOfEquals > 0)
+ {
+ string key = line.Substring(0, indexOfEquals);
+ string value = line.Substring(indexOfEquals + 1);
- return result;
+ if (!valueDict.TryGetValue(value, out ISymbolNode valueNode))
+ {
+ valueNode = factory.ReadOnlyDataBlob(
+ new Utf8String(_blobName + "_value_" + valueIndex++),
+ Utf8NullTerminatedBytes(value),
+ alignment: 1);
+ valueDict.Add(value, valueNode);
+ }
+
+ settings[key] = valueNode;
+ }
+ }
+
+ // The format is:
+ // * Number of entries (T)
+ // * N times pointer to key
+ // * N times pointer to value
+ builder.EmitNaturalInt(settings.Count);
+
+ int i = 0;
+ foreach (string key in settings.Keys)
+ {
+ ISymbolNode node = factory.ReadOnlyDataBlob(
+ new Utf8String(_blobName + "_key_" + i++),
+ Utf8NullTerminatedBytes(key),
+ alignment: 1);
+ builder.EmitPointerReloc(node);
+ }
+
+ foreach (ISymbolNode value in settings.Values)
+ {
+ builder.EmitPointerReloc(value);
+ }
+
+ static byte[] Utf8NullTerminatedBytes(string s)
+ {
+ byte[] result = new byte[Encoding.UTF8.GetByteCount(s) + 1];
+ Encoding.UTF8.GetBytes(s, result);
+ return result;
+ }
+
+ return builder.ToObjectData();
+ }
+
+ public override int CompareToImpl(ISortableNode other, CompilerComparer comparer)
+ {
+ return _blobName.CompareTo(((RuntimeConfigurationBlobNode)other)._blobName);
+ }
}
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeKnobsRootProvider.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeKnobsRootProvider.cs
deleted file mode 100644
index db1d0236a1556e..00000000000000
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/RuntimeKnobsRootProvider.cs
+++ /dev/null
@@ -1,57 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-using System.Collections.Generic;
-
-namespace ILCompiler
-{
- ///
- /// A root provider that provides a runtime configuration blob that influences runtime behaviors.
- /// See RhConfigValues.h for allowed values.
- ///
- public class RuntimeKnobsRootProvider : ICompilationRootProvider
- {
- private readonly IEnumerable _runtimeKnobs;
-
- public RuntimeKnobsRootProvider(IEnumerable runtimeKnobs)
- {
- _runtimeKnobs = runtimeKnobs;
- }
-
- void ICompilationRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider)
- {
- rootProvider.RootReadOnlyDataBlob(GetRuntimeKnobsBlob(), 4, "Runtime configuration knobs", "g_compilerEmbeddedKnobsBlob");
- }
-
- protected byte[] GetRuntimeKnobsBlob()
- {
- const int HeaderSize = 4;
-
- ArrayBuilder options = default(ArrayBuilder);
-
- // Reserve space for the header
- options.ZeroExtend(HeaderSize);
-
- foreach (string option in _runtimeKnobs)
- {
- byte[] optionBytes = System.Text.Encoding.UTF8.GetBytes(option);
- options.Append(optionBytes);
-
- // Emit a null to separate the next option
- options.Add(0);
- }
-
- byte[] result = options.ToArray();
-
- int length = options.Count - HeaderSize;
-
- // Encode the size of the blob into the header
- result[0] = (byte)length;
- result[1] = (byte)(length >> 8);
- result[2] = (byte)(length >> 0x10);
- result[3] = (byte)(length >> 0x18);
-
- return result;
- }
- }
-}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/VectorOfTFieldLayoutAlgorithm.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/VectorOfTFieldLayoutAlgorithm.cs
index 1a1eef14d55820..cfa4dc2524815b 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/VectorOfTFieldLayoutAlgorithm.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/VectorOfTFieldLayoutAlgorithm.cs
@@ -25,6 +25,7 @@ public override ComputedInstanceFieldLayout ComputeInstanceLayout(DefType defTyp
TargetDetails targetDetails = defType.Context.Target;
ComputedInstanceFieldLayout layoutFromMetadata = _fallbackAlgorithm.ComputeInstanceLayout(defType, layoutKind);
+ layoutFromMetadata.IsVectorTOrHasVectorTFields = true;
LayoutInt instanceFieldSize;
@@ -53,6 +54,7 @@ public override ComputedInstanceFieldLayout ComputeInstanceLayout(DefType defTyp
FieldAlignment = layoutFromMetadata.FieldAlignment,
FieldSize = instanceFieldSize,
Offsets = layoutFromMetadata.Offsets,
+ IsVectorTOrHasVectorTFields = true,
};
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/ILImporter.Scanner.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/IL/ILImporter.Scanner.cs
index bda68c3c44d141..8ea05d0f37b189 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/ILImporter.Scanner.cs
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/IL/ILImporter.Scanner.cs
@@ -929,12 +929,32 @@ private void ImportLdToken(int token)
nextBasicBlock = _basicBlocks[_currentOffset + 5];
if (nextBasicBlock == null)
{
+ // We expect pattern:
+ //
+ // ldtoken Foo
+ // call GetTypeFromHandle
+ // ldtoken Bar
+ // call GetTypeFromHandle
+ // call Equals
+ //
+ // We check for both ldtoken cases
if ((ILOpcode)_ilBytes[_currentOffset + 5] == ILOpcode.call)
{
methodToken = ReadILTokenAt(_currentOffset + 6);
method = (MethodDesc)_methodIL.GetObject(methodToken);
isTypeEquals = IsTypeEquals(method);
}
+ else if ((ILOpcode)_ilBytes[_currentOffset + 5] == ILOpcode.ldtoken
+ && _basicBlocks[_currentOffset + 10] == null
+ && (ILOpcode)_ilBytes[_currentOffset + 10] == ILOpcode.call
+ && methodToken == ReadILTokenAt(_currentOffset + 11)
+ && _basicBlocks[_currentOffset + 15] == null
+ && (ILOpcode)_ilBytes[_currentOffset + 15] == ILOpcode.call)
+ {
+ methodToken = ReadILTokenAt(_currentOffset + 16);
+ method = (MethodDesc)_methodIL.GetObject(methodToken);
+ isTypeEquals = IsTypeEquals(method);
+ }
}
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.Sorting.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.Sorting.cs
deleted file mode 100644
index 27d6287aef7e73..00000000000000
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.Sorting.cs
+++ /dev/null
@@ -1,21 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-using Internal.TypeSystem;
-
-using Debug = System.Diagnostics.Debug;
-
-namespace Internal.IL.Stubs.StartupCode
-{
- public partial class AppContextInitializerMethod
- {
- protected override int ClassCode => 15749517;
-
- protected override int CompareToImpl(MethodDesc other, TypeSystemComparer comparer)
- {
- // Should be a singleton
- Debug.Assert(this == other);
- return 0;
- }
- }
-}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.cs b/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.cs
deleted file mode 100644
index 92d9eccdaee950..00000000000000
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/IL/Stubs/StartupCode/AppContextInitializerMethod.cs
+++ /dev/null
@@ -1,98 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-using System;
-using System.Collections.Generic;
-
-using Internal.TypeSystem;
-
-namespace Internal.IL.Stubs.StartupCode
-{
- public sealed partial class AppContextInitializerMethod : ILStubMethod
- {
- private TypeDesc _owningType;
- private MethodSignature _signature;
- private IReadOnlyCollection> _switches;
-
- public AppContextInitializerMethod(TypeDesc owningType, IEnumerable args)
- {
- _owningType = owningType;
- var switches = new List>();
-
- foreach (string s in args)
- {
- int index = s.IndexOf('=');
- if (index <= 0)
- throw new ArgumentException($"String '{s}' in unexpected format. Expected 'Key=Value'");
- switches.Add(KeyValuePair.Create(s.Substring(0, index), s.Substring(index + 1)));
- }
-
- _switches = switches;
- }
-
- public override TypeSystemContext Context
- {
- get
- {
- return _owningType.Context;
- }
- }
-
- public override TypeDesc OwningType
- {
- get
- {
- return _owningType;
- }
- }
-
- public override string Name
- {
- get
- {
- return "SetAppContextSwitches";
- }
- }
-
- public override string DiagnosticName
- {
- get
- {
- return "SetAppContextSwitches";
- }
- }
-
- public override MethodIL EmitIL()
- {
- ILEmitter emitter = new ILEmitter();
- ILCodeStream codeStream = emitter.NewCodeStream();
-
- MetadataType appContextType = Context.SystemModule.GetKnownType("System", "AppContext");
- MethodDesc setDataMethod = appContextType.GetKnownMethod("SetData", null);
- ILToken setDataToken = emitter.NewToken(setDataMethod);
-
- foreach (KeyValuePair keyValue in _switches)
- {
- codeStream.Emit(ILOpcode.ldstr, emitter.NewToken(keyValue.Key));
- codeStream.Emit(ILOpcode.ldstr, emitter.NewToken(keyValue.Value));
- codeStream.Emit(ILOpcode.call, setDataToken);
- }
-
- codeStream.Emit(ILOpcode.ret);
-
- return emitter.Link(this);
- }
-
- public override MethodSignature Signature
- {
- get
- {
- _signature ??= new MethodSignature(MethodSignatureFlags.Static, 0,
- Context.GetWellKnownType(WellKnownType.Void),
- TypeDesc.EmptyTypes);
-
- return _signature;
- }
- }
- }
-}
diff --git a/src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj b/src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj
index 724df1c41ccfcc..e0bce1b1eb82c7 100644
--- a/src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj
+++ b/src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj
@@ -284,7 +284,6 @@
-
@@ -579,7 +578,6 @@
-
@@ -624,8 +622,6 @@
-
-
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
index 36c558a19f5983..801bd7dbe2ccbd 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedDataNode.cs
@@ -10,24 +10,19 @@
namespace ILCompiler.DependencyAnalysis
{
- public interface IHasStartSymbol
- {
- ObjectAndOffsetSymbolNode StartSymbol { get; }
- }
-
///
/// Represents an array of nodes. The contents of this node will be emitted
/// by placing a starting symbol, followed by contents of nodes (optionally
/// sorted using provided comparer), followed by ending symbol.
///
- public class ArrayOfEmbeddedDataNode : EmbeddedDataContainerNode, IHasStartSymbol
+ public class ArrayOfEmbeddedDataNode : EmbeddedDataContainerNode
where TEmbedded : EmbeddedObjectNode
{
private HashSet _nestedNodes = new HashSet();
private List _nestedNodesList = new List();
private IComparer _sorter;
- public ArrayOfEmbeddedDataNode(string startSymbolMangledName, string endSymbolMangledName, IComparer nodeSorter) : base(startSymbolMangledName, endSymbolMangledName)
+ public ArrayOfEmbeddedDataNode(string mangledName, IComparer nodeSorter) : base(mangledName)
{
_sorter = nodeSorter;
}
@@ -40,11 +35,10 @@ public void AddEmbeddedObject(TEmbedded symbol)
{
_nestedNodesList.Add(symbol);
}
- symbol.ContainingNode = this;
}
}
- protected override string GetName(NodeFactory factory) => $"Region {StartSymbol.GetMangledName(factory.NameMangler)}";
+ protected override string GetName(NodeFactory factory) => $"Region {this.GetMangledName(factory.NameMangler)}";
public override ObjectNodeSection GetSection(NodeFactory factory) => ObjectNodeSection.DataSection;
public override bool IsShareable => false;
@@ -89,13 +83,10 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly)
if (_sorter != null)
_nestedNodesList.MergeSort(_sorter);
- builder.AddSymbol(StartSymbol);
+ builder.AddSymbol(this);
GetElementDataForNodes(ref builder, factory, relocsOnly);
- EndSymbol.SetSymbolOffset(builder.CountBytes);
- builder.AddSymbol(EndSymbol);
-
ObjectData objData = builder.ToObjectData();
return objData;
}
@@ -105,15 +96,6 @@ public override bool ShouldSkipEmittingObjectNode(NodeFactory factory)
return _nestedNodesList.Count == 0;
}
- protected override DependencyList ComputeNonRelocationBasedDependencies(NodeFactory factory)
- {
- DependencyList dependencies = new DependencyList();
- dependencies.Add(StartSymbol, "StartSymbol");
- dependencies.Add(EndSymbol, "EndSymbol");
-
- return dependencies;
- }
-
protected internal override int Phase => (int)ObjectNodePhase.Ordered;
public override int ClassCode => (int)ObjectNodeOrder.ArrayOfEmbeddedDataNode;
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
index 8fba1bae53a39c..aae7ccddd7e5fd 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ArrayOfEmbeddedPointersNode.cs
@@ -30,13 +30,12 @@ public sealed class ArrayOfEmbeddedPointersNode : ArrayOfEmbeddedDataNo
///
public delegate void OnMarkedDelegate(EmbeddedPointerIndirectionNode embeddedObject);
- public ArrayOfEmbeddedPointersNode(string startSymbolMangledName, string endSymbolMangledName, IComparer> nodeSorter)
+ public ArrayOfEmbeddedPointersNode(string mangledName, IComparer> nodeSorter)
: base(
- startSymbolMangledName,
- endSymbolMangledName,
+ mangledName,
nodeSorter)
{
- _startSymbolMangledName = startSymbolMangledName;
+ _startSymbolMangledName = mangledName;
}
public EmbeddedObjectNode NewNode(TTarget target)
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
index 511f91cbb56f22..33b7594383c49a 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/EmbeddedObjectNode.cs
@@ -16,8 +16,6 @@ public abstract class EmbeddedObjectNode : SortableDependencyNode
private int _offset;
private int _index;
- public IHasStartSymbol ContainingNode { get; set; }
-
public EmbeddedObjectNode()
{
_offset = InvalidOffset;
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionNode.cs
index b0020b0ec001c7..45dc3fb4e240ab 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionNode.cs
@@ -12,7 +12,7 @@ public class ImportSectionNode : EmbeddedObjectNode
{
private class ImportTable : ArrayOfEmbeddedDataNode
{
- public ImportTable(string startSymbol, string endSymbol) : base(startSymbol, endSymbol, nodeSorter: new EmbeddedObjectNodeComparer(CompilerComparer.Instance)) {}
+ public ImportTable(string symbol) : base(symbol, nodeSorter: new EmbeddedObjectNodeComparer(CompilerComparer.Instance)) {}
public override bool ShouldSkipEmittingObjectNode(NodeFactory factory) => false;
@@ -44,8 +44,8 @@ public ImportSectionNode(string name, ReadyToRunImportSectionType importType, Re
_emitPrecode = emitPrecode;
_emitGCRefMap = emitGCRefMap;
- _imports = new ImportTable(_name + "_ImportBegin", _name + "_ImportEnd");
- _signatures = new ArrayOfEmbeddedPointersNode(_name + "_SigBegin", _name + "_SigEnd", new EmbeddedObjectNodeComparer(CompilerComparer.Instance));
+ _imports = new ImportTable(_name + "_ImportBegin");
+ _signatures = new ArrayOfEmbeddedPointersNode(_name + "_SigBegin", new EmbeddedObjectNodeComparer(CompilerComparer.Instance));
_signatureList = new List();
_gcRefMap = _emitGCRefMap ? new GCRefMapNode(this) : null;
}
@@ -101,7 +101,7 @@ public override void EncodeData(ref ObjectDataBuilder dataBuilder, NodeFactory f
{
if (!_imports.ShouldSkipEmittingObjectNode(factory))
{
- dataBuilder.EmitReloc(_imports.StartSymbol, RelocType.IMAGE_REL_BASED_ADDR32NB, 0);
+ dataBuilder.EmitReloc(_imports, RelocType.IMAGE_REL_BASED_ADDR32NB, 0);
}
else
{
@@ -110,7 +110,7 @@ public override void EncodeData(ref ObjectDataBuilder dataBuilder, NodeFactory f
if (!relocsOnly)
{
- dataBuilder.EmitReloc(_imports.StartSymbol, RelocType.IMAGE_REL_SYMBOL_SIZE);
+ dataBuilder.EmitReloc(_imports, RelocType.IMAGE_REL_SYMBOL_SIZE);
dataBuilder.EmitShort((short)_flags);
dataBuilder.EmitByte((byte)_type);
dataBuilder.EmitByte(_entrySize);
@@ -118,7 +118,7 @@ public override void EncodeData(ref ObjectDataBuilder dataBuilder, NodeFactory f
if (!_signatures.ShouldSkipEmittingObjectNode(factory))
{
- dataBuilder.EmitReloc(_signatures.StartSymbol, RelocType.IMAGE_REL_BASED_ADDR32NB, 0);
+ dataBuilder.EmitReloc(_signatures, RelocType.IMAGE_REL_BASED_ADDR32NB, 0);
}
else
{
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionsTableNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionsTableNode.cs
index 61b5e394030dce..9163ac18784d5b 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionsTableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportSectionsTableNode.cs
@@ -12,7 +12,7 @@ public class ImportSectionsTableNode : ArrayOfEmbeddedDataNode EncodeDataCore(NodeFactory factory)
yield return new GCInfoComponent(_methodNode, 0);
yield return new GCInfoComponent(_methodNode, _methodNode.Size);
// TODO: Is this correct?
- yield return new GCInfoComponent(factory.RuntimeFunctionsGCInfo.StartSymbol, this.OffsetFromBeginningOfArray);
+ yield return new GCInfoComponent(factory.RuntimeFunctionsGCInfo, this.OffsetFromBeginningOfArray);
}
else
{
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsGCInfoNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsGCInfoNode.cs
index 8d69be7c2d0d34..d65a01c827a303 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsGCInfoNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsGCInfoNode.cs
@@ -8,7 +8,7 @@ namespace ILCompiler.DependencyAnalysis.ReadyToRun
public class RuntimeFunctionsGCInfoNode : ArrayOfEmbeddedDataNode
{
public RuntimeFunctionsGCInfoNode()
- : base("RuntimeFunctionsGCInfo_Begin", "RuntimeFunctionsGCInfo_End", new EmbeddedObjectNodeComparer(CompilerComparer.Instance))
+ : base("RuntimeFunctionsGCInfo", new EmbeddedObjectNodeComparer(CompilerComparer.Instance))
{
}
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsTableNode.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsTableNode.cs
index 941f671c4d9a2b..d25829f7acdaa8 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsTableNode.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/RuntimeFunctionsTableNode.cs
@@ -126,7 +126,7 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
Debug.Assert(frameInfo.StartOffset != frameInfo.EndOffset);
runtimeFunctionsBuilder.EmitReloc(symbol, RelocType.IMAGE_REL_BASED_ADDR32NB, delta: frameInfo.EndOffset);
}
- runtimeFunctionsBuilder.EmitReloc(factory.RuntimeFunctionsGCInfo.StartSymbol, RelocType.IMAGE_REL_BASED_ADDR32NB, funcletOffsets[frameIndex]);
+ runtimeFunctionsBuilder.EmitReloc(factory.RuntimeFunctionsGCInfo, RelocType.IMAGE_REL_BASED_ADDR32NB, funcletOffsets[frameIndex]);
runtimeFunctionIndex++;
}
}
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/TypeFixupSignature.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/TypeFixupSignature.cs
index a0b9e801f2495d..e379651ff5e2cb 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/TypeFixupSignature.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/TypeFixupSignature.cs
@@ -36,14 +36,20 @@ public override ObjectData GetData(NodeFactory factory, bool relocsOnly = false)
if (!relocsOnly)
{
+ ReadyToRunFixupKind fixupKind = _fixupKind;
dataBuilder.AddSymbol(this);
+ if ((fixupKind == ReadyToRunFixupKind.Verify_TypeLayout) && ((MetadataType)_typeDesc).IsVectorTOrHasVectorTFields)
+ {
+ fixupKind = ReadyToRunFixupKind.Check_TypeLayout;
+ }
+
IEcmaModule targetModule = factory.SignatureContext.GetTargetModule(_typeDesc);
- SignatureContext innerContext = dataBuilder.EmitFixup(factory, _fixupKind, targetModule, factory.SignatureContext);
+ SignatureContext innerContext = dataBuilder.EmitFixup(factory, fixupKind, targetModule, factory.SignatureContext);
dataBuilder.EmitTypeSignature(_typeDesc, innerContext);
- if ((_fixupKind == ReadyToRunFixupKind.Check_TypeLayout) ||
- (_fixupKind == ReadyToRunFixupKind.Verify_TypeLayout))
+ if ((fixupKind == ReadyToRunFixupKind.Check_TypeLayout) ||
+ (fixupKind == ReadyToRunFixupKind.Verify_TypeLayout))
{
EncodeTypeLayout(dataBuilder, _typeDesc);
}
@@ -92,7 +98,7 @@ private static void EncodeTypeLayout(ObjectDataSignatureBuilder dataBuilder, Typ
};
dataBuilder.EmitUInt((uint)hfaElementType);
}
-
+
if (alignment != pointerSize)
{
dataBuilder.EmitUInt((uint)alignment);
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRunCodegenNodeFactory.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRunCodegenNodeFactory.cs
index 5ab12a41e6150e..14dc927fc5555d 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRunCodegenNodeFactory.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRunCodegenNodeFactory.cs
@@ -742,7 +742,7 @@ public void AttachToDependencyGraph(DependencyAnalyzerBase graph, I
Header.Add(Internal.Runtime.ReadyToRunSectionType.InstanceMethodEntryPoints, InstanceEntryPointTable, InstanceEntryPointTable);
ImportSectionsTable = new ImportSectionsTableNode(this);
- Header.Add(Internal.Runtime.ReadyToRunSectionType.ImportSections, ImportSectionsTable, ImportSectionsTable.StartSymbol);
+ Header.Add(Internal.Runtime.ReadyToRunSectionType.ImportSections, ImportSectionsTable, ImportSectionsTable);
DebugInfoTable = new DebugInfoTableNode();
Header.Add(Internal.Runtime.ReadyToRunSectionType.DebugInfo, DebugInfoTable, DebugInfoTable);
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs
index 8b1b8d7c30e2e0..3644241dcf85b7 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs
@@ -475,12 +475,18 @@ private bool IsLayoutFixedInCurrentVersionBubbleInternal(TypeDesc type)
return true;
}
- if (!(type is MetadataType defType))
+ if (type is not MetadataType defType)
{
// Non metadata backed types have layout defined in all version bubbles
return true;
}
+ if (VectorOfTFieldLayoutAlgorithm.IsVectorOfTType(defType))
+ {
+ // Vector always needs a layout check
+ return false;
+ }
+
if (!NodeFactory.CompilationModuleGroup.VersionsWithModule(defType.Module))
{
// Valuetypes with non-versionable attribute are candidates for fixed layout. Reject the rest.
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilerContext.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilerContext.cs
index 6eed36223b0992..c6d40d4ee7b42b 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilerContext.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilerContext.cs
@@ -220,6 +220,7 @@ public override ComputedInstanceFieldLayout ComputeInstanceLayout(DefType type,
ByteCountAlignment = LayoutInt.Indeterminate,
Offsets = fieldsAndOffsets.ToArray(),
LayoutAbiStable = false,
+ IsVectorTOrHasVectorTFields = true,
};
return instanceLayout;
}
@@ -238,6 +239,7 @@ public override ComputedInstanceFieldLayout ComputeInstanceLayout(DefType type,
FieldSize = layoutFromSimilarIntrinsicVector.FieldSize,
Offsets = layoutFromMetadata.Offsets,
LayoutAbiStable = _vectorAbiIsStable,
+ IsVectorTOrHasVectorTFields = true,
};
#else
return new ComputedInstanceFieldLayout
@@ -248,6 +250,7 @@ public override ComputedInstanceFieldLayout ComputeInstanceLayout(DefType type,
FieldSize = layoutFromSimilarIntrinsicVector.FieldSize,
Offsets = layoutFromMetadata.Offsets,
LayoutAbiStable = _vectorAbiIsStable,
+ IsVectorTOrHasVectorTFields = true,
};
#endif
}
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj
index 874104ecb86821..3bb44554265c6c 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj
@@ -76,7 +76,6 @@
-
diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/JitInterface/CorInfoImpl.ReadyToRun.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/JitInterface/CorInfoImpl.ReadyToRun.cs
index bb23738a4dc621..6744603dcb2288 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/JitInterface/CorInfoImpl.ReadyToRun.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/JitInterface/CorInfoImpl.ReadyToRun.cs
@@ -3128,7 +3128,7 @@ private bool getStringChar(CORINFO_OBJECT_STRUCT_* strObj, int index, ushort* va
{
return false;
}
-
+
private CORINFO_OBJECT_STRUCT_* getRuntimeTypePointer(CORINFO_CLASS_STRUCT_* cls)
{
return null;
diff --git a/src/coreclr/tools/aot/ILCompiler.TypeSystem.Tests/ILCompiler.TypeSystem.Tests.csproj b/src/coreclr/tools/aot/ILCompiler.TypeSystem.Tests/ILCompiler.TypeSystem.Tests.csproj
index 1f6b33ff18ba39..d62215c7901a56 100644
--- a/src/coreclr/tools/aot/ILCompiler.TypeSystem.Tests/ILCompiler.TypeSystem.Tests.csproj
+++ b/src/coreclr/tools/aot/ILCompiler.TypeSystem.Tests/ILCompiler.TypeSystem.Tests.csproj
@@ -20,7 +20,6 @@
-
diff --git a/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj b/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
index 9e93415789c3c1..7e5f930584abfc 100644
--- a/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
+++ b/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
@@ -10,8 +10,10 @@
$(RuntimeBinDir)ilc-published/
- false
- false
+
+ false
+
+ falsetrue$(ROOTFS_DIR)true
diff --git a/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs b/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs
index ff80a245bcea6f..574bfb78c461ca 100644
--- a/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs
+++ b/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs
@@ -88,8 +88,6 @@ internal sealed class ILCompilerRootCommand : RootCommand
new(new[] { "--methodbodyfolding" }, "Fold identical method bodies");
public Option InitAssemblies { get; } =
new(new[] { "--initassembly" }, Array.Empty, "Assembly(ies) with a library initializer");
- public Option AppContextSwitches { get; } =
- new(new[] { "--appcontextswitch" }, Array.Empty, "System.AppContext switches to set (format: 'Key=Value')");
public Option FeatureSwitches { get; } =
new(new[] { "--feature" }, Array.Empty, "Feature switches to apply (format: 'Namespace.Name=[true|false]'");
public Option RuntimeOptions { get; } =
@@ -114,6 +112,8 @@ internal sealed class ILCompilerRootCommand : RootCommand
}, true, "Maximum number of threads to use during compilation");
public Option InstructionSet { get; } =
new(new[] { "--instruction-set" }, "Instruction set to allow or disallow");
+ public Option MaxVectorTBitWidth { get; } =
+ new(new[] { "--max-vectort-bitwidth" }, "Maximum width, in bits, that Vector is allowed to be");
public Option Guard { get; } =
new(new[] { "--guard" }, "Enable mitigations. Options: 'cf': CFG (Control Flow Guard, Windows only)");
public Option Dehydrate { get; } =
@@ -207,12 +207,12 @@ public ILCompilerRootCommand(string[] args) : base(".NET Native IL Compiler")
AddOption(EmitStackTraceData);
AddOption(MethodBodyFolding);
AddOption(InitAssemblies);
- AddOption(AppContextSwitches);
AddOption(FeatureSwitches);
AddOption(RuntimeOptions);
AddOption(RuntimeKnobs);
AddOption(Parallelism);
AddOption(InstructionSet);
+ AddOption(MaxVectorTBitWidth);
AddOption(Guard);
AddOption(Dehydrate);
AddOption(PreinitStatics);
@@ -270,9 +270,11 @@ public ILCompilerRootCommand(string[] args) : base(".NET Native IL Compiler")
// + the original command line arguments
// + a rsp file that should work to directly run out of the zip file
+#pragma warning disable CA1861 // Avoid constant arrays as arguments. Only executed once during the execution of the program.
Helpers.MakeReproPackage(makeReproPath, context.ParseResult.GetValue(OutputFilePath), args, context.ParseResult,
inputOptions : new[] { "r", "reference", "m", "mibc", "rdxml", "directpinvokelist", "descriptor" },
outputOptions : new[] { "o", "out", "exportsfile" });
+#pragma warning restore CA1861 // Avoid constant arrays as arguments
}
context.ExitCode = new Program(this).Run();
diff --git a/src/coreclr/tools/aot/ILCompiler/Program.cs b/src/coreclr/tools/aot/ILCompiler/Program.cs
index 36493522a44041..f83ad617dda8d2 100644
--- a/src/coreclr/tools/aot/ILCompiler/Program.cs
+++ b/src/coreclr/tools/aot/ILCompiler/Program.cs
@@ -29,6 +29,7 @@ namespace ILCompiler
internal sealed class Program
{
private readonly ILCompilerRootCommand _command;
+ private static readonly char[] s_separator = new char[] { ',', ';', ' ' };
public Program(ILCompilerRootCommand command)
{
@@ -41,7 +42,7 @@ public Program(ILCompilerRootCommand command)
}
}
- private List CreateInitializerList(CompilerTypeSystemContext context)
+ private IReadOnlyCollection CreateInitializerList(CompilerTypeSystemContext context)
{
List assembliesWithInitializers = new List();
@@ -55,18 +56,7 @@ private List CreateInitializerList(CompilerTypeSystemContext context
var libraryInitializers = new LibraryInitializers(context, assembliesWithInitializers);
- List initializerList = new List(libraryInitializers.LibraryInitializerMethods);
-
- // If there are any AppContext switches the user wishes to enable, generate code that sets them.
- string[] appContextSwitches = Get(_command.AppContextSwitches);
- if (appContextSwitches.Length > 0)
- {
- MethodDesc appContextInitMethod = new Internal.IL.Stubs.StartupCode.AppContextInitializerMethod(
- context.GeneratedAssembly.GetGlobalModuleType(), appContextSwitches);
- initializerList.Add(appContextInitMethod);
- }
-
- return initializerList;
+ return libraryInitializers.LibraryInitializerMethods;
}
public int Run()
@@ -77,7 +67,7 @@ public int Run()
TargetArchitecture targetArchitecture = Get(_command.TargetArchitecture);
TargetOS targetOS = Get(_command.TargetOS);
- InstructionSetSupport instructionSetSupport = Helpers.ConfigureInstructionSetSupport(Get(_command.InstructionSet), targetArchitecture, targetOS,
+ InstructionSetSupport instructionSetSupport = Helpers.ConfigureInstructionSetSupport(Get(_command.InstructionSet), Get(_command.MaxVectorTBitWidth), targetArchitecture, targetOS,
"Unrecognized instruction set {0}", "Unsupported combination of instruction sets: {0}/{1}");
string systemModuleName = Get(_command.SystemModuleName);
@@ -208,6 +198,8 @@ public int Run()
compilationGroup = new SingleFileCompilationModuleGroup();
}
+ const string settingsBlobName = "g_compilerEmbeddedSettingsBlob";
+ const string knobsBlobName = "g_compilerEmbeddedKnobsBlob";
string[] runtimeOptions = Get(_command.RuntimeOptions);
string[] runtimeKnobs = Get(_command.RuntimeKnobs);
if (nativeLib)
@@ -215,8 +207,8 @@ public int Run()
// Set owning module of generated native library startup method to compiler generated module,
// to ensure the startup method is included in the object file during multimodule mode build
compilationRoots.Add(new NativeLibraryInitializerRootProvider(typeSystemContext.GeneratedAssembly, CreateInitializerList(typeSystemContext)));
- compilationRoots.Add(new RuntimeConfigurationRootProvider(runtimeOptions));
- compilationRoots.Add(new RuntimeKnobsRootProvider(runtimeKnobs));
+ compilationRoots.Add(new RuntimeConfigurationRootProvider(settingsBlobName, runtimeOptions));
+ compilationRoots.Add(new RuntimeConfigurationRootProvider(knobsBlobName, runtimeKnobs));
compilationRoots.Add(new ExpectedIsaFeaturesRootProvider(instructionSetSupport));
if (SplitExeInitialization)
{
@@ -226,8 +218,8 @@ public int Run()
else if (entrypointModule != null)
{
compilationRoots.Add(new MainMethodRootProvider(entrypointModule, CreateInitializerList(typeSystemContext), generateLibraryAndModuleInitializers: !SplitExeInitialization));
- compilationRoots.Add(new RuntimeConfigurationRootProvider(runtimeOptions));
- compilationRoots.Add(new RuntimeKnobsRootProvider(runtimeKnobs));
+ compilationRoots.Add(new RuntimeConfigurationRootProvider(settingsBlobName, runtimeOptions));
+ compilationRoots.Add(new RuntimeConfigurationRootProvider(knobsBlobName, runtimeKnobs));
compilationRoots.Add(new ExpectedIsaFeaturesRootProvider(instructionSetSupport));
if (SplitExeInitialization)
{
@@ -691,7 +683,7 @@ private static IEnumerable ProcessWarningCodes(IEnumerable warningC
{
foreach (string value in warningCodes)
{
- string[] values = value.Split(new char[] { ',', ';', ' ' }, StringSplitOptions.RemoveEmptyEntries);
+ string[] values = value.Split(s_separator, StringSplitOptions.RemoveEmptyEntries);
foreach (string id in values)
{
if (!id.StartsWith("IL", StringComparison.Ordinal) || !ushort.TryParse(id.AsSpan(2), out ushort code))
diff --git a/src/coreclr/tools/aot/ILCompiler/repro/repro.csproj b/src/coreclr/tools/aot/ILCompiler/repro/repro.csproj
index 26c34d3c5d4283..131ddbd3b22576 100644
--- a/src/coreclr/tools/aot/ILCompiler/repro/repro.csproj
+++ b/src/coreclr/tools/aot/ILCompiler/repro/repro.csproj
@@ -23,6 +23,7 @@
+
diff --git a/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs b/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs
index e3918372e42dc9..ef384a1abece68 100644
--- a/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs
+++ b/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs
@@ -23,6 +23,8 @@ internal class Crossgen2RootCommand : RootCommand
new(new[] { "--reference", "-r" }, result => Helpers.BuildPathDictionary(result.Tokens, false), true, SR.ReferenceFiles);
public Option InstructionSet { get; } =
new(new[] { "--instruction-set" }, SR.InstructionSets);
+ public Option MaxVectorTBitWidth { get; } =
+ new(new[] { "--max-vectort-bitwidth" }, SR.MaxVectorTBitWidths);
public Option MibcFilePaths { get; } =
new(new[] { "--mibc", "-m" }, Array.Empty, SR.MibcFiles);
public Option OutputFilePath { get; } =
@@ -193,6 +195,7 @@ public Crossgen2RootCommand(string[] args) : base(SR.Crossgen2BannerText)
AddOption(UnrootedInputFilePaths);
AddOption(ReferenceFilePaths);
AddOption(InstructionSet);
+ AddOption(MaxVectorTBitWidth);
AddOption(MibcFilePaths);
AddOption(OutputFilePath);
AddOption(CompositeRootPath);
diff --git a/src/coreclr/tools/aot/crossgen2/Program.cs b/src/coreclr/tools/aot/crossgen2/Program.cs
index 639e5551e96940..1d43fc8c7a9150 100644
--- a/src/coreclr/tools/aot/crossgen2/Program.cs
+++ b/src/coreclr/tools/aot/crossgen2/Program.cs
@@ -76,7 +76,7 @@ public int Run()
TargetArchitecture targetArchitecture = Get(_command.TargetArchitecture);
TargetOS targetOS = Get(_command.TargetOS);
- InstructionSetSupport instructionSetSupport = Helpers.ConfigureInstructionSetSupport(Get(_command.InstructionSet), targetArchitecture, targetOS,
+ InstructionSetSupport instructionSetSupport = Helpers.ConfigureInstructionSetSupport(Get(_command.InstructionSet), Get(_command.MaxVectorTBitWidth), targetArchitecture, targetOS,
SR.InstructionSetMustNotBe, SR.InstructionSetInvalidImplication);
SharedGenericsMode genericsMode = SharedGenericsMode.CanonicalReferenceTypes;
var targetDetails = new TargetDetails(targetArchitecture, targetOS, Crossgen2RootCommand.IsArmel ? TargetAbi.NativeAotArmel : TargetAbi.NativeAot, instructionSetSupport.GetVectorTSimdVector());
diff --git a/src/coreclr/tools/aot/crossgen2/Properties/Resources.resx b/src/coreclr/tools/aot/crossgen2/Properties/Resources.resx
index b899f77a8f1147..a737ea6aeb7706 100644
--- a/src/coreclr/tools/aot/crossgen2/Properties/Resources.resx
+++ b/src/coreclr/tools/aot/crossgen2/Properties/Resources.resx
@@ -180,6 +180,9 @@
Instruction set '{0}' implies support for instruction set '{1}'
+
+ The maximum width, in bits, for System.Numerics.Vector<T>. For example '128', '256', or '512'.
+
Input files without automatic rooting of all methods
diff --git a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
index 464aa2bc429075..ceeb4d63ad0c1e 100644
--- a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
+++ b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
@@ -8,7 +8,7 @@
falsetrue
- linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64;win-arm
+ linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64$(PackageRID)falsetrue
diff --git a/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj b/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj
index c3d3fc03766719..ac3720d4f7e059 100644
--- a/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj
+++ b/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj
@@ -13,7 +13,7 @@
truetruedotnet-pgo
- win-x64;win-x86;win-arm;osx-x64
+ win-x64;win-x86;osx-x64$(OutputPath)false.NET Performance Guided Optimization Tool
diff --git a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp
index aa491393c414bd..70378310e449a3 100644
--- a/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp
+++ b/src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp
@@ -1221,8 +1221,6 @@ const char* CorJitFlagToString(CORJIT_FLAGS::CorJitFlag flag)
return "CORJIT_FLAG_BBINSTR_IF_LOOPS";
case CORJIT_FLAGS::CorJitFlag::CORJIT_FLAG_PUBLISH_SECRET_PARAM:
return "CORJIT_FLAG_PUBLISH_SECRET_PARAM";
- case CORJIT_FLAGS::CorJitFlag::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND:
- return "CORJIT_FLAG_SAMPLING_JIT_BACKGROUND";
case CORJIT_FLAGS::CorJitFlag::CORJIT_FLAG_USE_PINVOKE_HELPERS:
return "CORJIT_FLAG_USE_PINVOKE_HELPERS";
case CORJIT_FLAGS::CorJitFlag::CORJIT_FLAG_REVERSE_PINVOKE:
diff --git a/src/coreclr/tools/superpmi/superpmi-shared/spmidumphelper.cpp b/src/coreclr/tools/superpmi/superpmi-shared/spmidumphelper.cpp
index d4dde380b7e236..46d05496066c7c 100644
--- a/src/coreclr/tools/superpmi/superpmi-shared/spmidumphelper.cpp
+++ b/src/coreclr/tools/superpmi/superpmi-shared/spmidumphelper.cpp
@@ -259,7 +259,6 @@ std::string SpmiDumpHelper::DumpJitFlags(unsigned long long flags)
AddFlag(BBINSTR_IF_LOOPS);
AddFlag(PUBLISH_SECRET_PARAM);
- AddFlag(SAMPLING_JIT_BACKGROUND);
AddFlag(USE_PINVOKE_HELPERS);
AddFlag(REVERSE_PINVOKE);
AddFlag(TRACK_TRANSITIONS);
diff --git a/src/coreclr/vm/CMakeLists.txt b/src/coreclr/vm/CMakeLists.txt
index fada96c3a4a290..bf6b91acbf4bd4 100644
--- a/src/coreclr/vm/CMakeLists.txt
+++ b/src/coreclr/vm/CMakeLists.txt
@@ -578,7 +578,6 @@ if(CLR_CMAKE_TARGET_WIN32)
dwreport.cpp
eventreporter.cpp
rtlfunctions.cpp
- stacksampler.cpp
)
list(APPEND VM_HEADERS_WKS
@@ -587,7 +586,6 @@ if(CLR_CMAKE_TARGET_WIN32)
dwreport.h
eventreporter.h
rtlfunctions.h
- stacksampler.h
)
# COM interop scenarios
diff --git a/src/coreclr/vm/ClrEtwAllMeta.lst b/src/coreclr/vm/ClrEtwAllMeta.lst
index 3c145a93bc1ab6..a4ea6e8a4cdb0a 100644
--- a/src/coreclr/vm/ClrEtwAllMeta.lst
+++ b/src/coreclr/vm/ClrEtwAllMeta.lst
@@ -190,7 +190,7 @@ nostack:Contention:::ContentionStop
nomac:Contention:::ContentionStop
nostack:Contention:::ContentionStop_V1
nomac:Contention:::ContentionStop_V1
-nomac:Contention:::LockCreated
+nomac:Contention:::ContentionLockCreated
##################
# StackWalk events
diff --git a/src/coreclr/vm/array.cpp b/src/coreclr/vm/array.cpp
index 54ecd0d7e7e7a3..aaa203491c5afa 100644
--- a/src/coreclr/vm/array.cpp
+++ b/src/coreclr/vm/array.cpp
@@ -665,7 +665,10 @@ MethodTable* Module::CreateArrayMethodTable(TypeHandle elemTypeHnd, CorElementTy
IDS_CLASSLOAD_VALUECLASSTOOLARGE);
}
- val_serie_item *val_item = &(pSeries->val_serie[-index]);
+ // pSeries->val_serie is a fixed sized array.
+ // Use pointer arithmetic instead of direct array access to avoid compilers, specifically GCC,
+ // to discover undefined behavior and generate unintended code when optimization is turned on.
+ val_serie_item *val_item = pSeries->val_serie - index;
val_item->set_val_serie_item (NumPtrs, (unsigned short)skip);
}
diff --git a/src/coreclr/vm/ceemain.cpp b/src/coreclr/vm/ceemain.cpp
index 162c0aa7bf9cc6..1fd898684cf1c1 100644
--- a/src/coreclr/vm/ceemain.cpp
+++ b/src/coreclr/vm/ceemain.cpp
@@ -170,10 +170,6 @@
#include "stringarraylist.h"
#include "stubhelpers.h"
-#ifdef FEATURE_STACK_SAMPLING
-#include "stacksampler.h"
-#endif
-
#ifdef FEATURE_COMINTEROP
#include "runtimecallablewrapper.h"
#include "mngstdinterfaces.h"
@@ -932,10 +928,6 @@ void EEStartupHelper()
SystemDomain::System()->DefaultDomain()->SetupSharedStatics();
-#ifdef FEATURE_STACK_SAMPLING
- StackSampler::Init();
-#endif
-
#ifdef FEATURE_MINIMETADATA_IN_TRIAGEDUMPS
// retrieve configured max size for the mini-metadata buffer (defaults to 64KB)
g_MiniMetaDataBuffMaxSize = CLRConfig::GetConfigValue(CLRConfig::INTERNAL_MiniMdBufferCapacity);
diff --git a/src/coreclr/vm/codeman.cpp b/src/coreclr/vm/codeman.cpp
index eba0c2cd1e88fb..51eec2b9dae9a2 100644
--- a/src/coreclr/vm/codeman.cpp
+++ b/src/coreclr/vm/codeman.cpp
@@ -1338,6 +1338,9 @@ void EEJitManager::SetCpuInfo()
CORJIT_FLAGS CPUCompileFlags;
+ // Get the maximum bitwidth of Vector, rounding down to the nearest multiple of 128-bits
+ uint32_t maxVectorTBitWidth = (CLRConfig::GetConfigValue(CLRConfig::EXTERNAL_MaxVectorTBitWidth) / 128) * 128;
+
#if defined(TARGET_X86) || defined(TARGET_AMD64)
CPUCompileFlags.Set(InstructionSet_X86Base);
@@ -1347,84 +1350,6 @@ void EEJitManager::SetCpuInfo()
// AMD64 Architecture Programmer’s Manual. Volume 3
// For more information, please refer to the CPUID instruction in the respective manuals
- // We will set the following flags:
- // CORJIT_FLAG_USE_SSE2 is required
- // SSE - EDX bit 25
- // SSE2 - EDX bit 26
- // CORJIT_FLAG_USE_AES
- // CORJIT_FLAG_USE_SSE2
- // AES - ECX bit 25
- // CORJIT_FLAG_USE_PCLMULQDQ
- // CORJIT_FLAG_USE_SSE2
- // PCLMULQDQ - ECX bit 1
- // CORJIT_FLAG_USE_SSE3 if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSE2
- // SSE3 - ECX bit 0
- // CORJIT_FLAG_USE_SSSE3 if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSE3
- // SSSE3 - ECX bit 9
- // CORJIT_FLAG_USE_SSE41 if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSSE3
- // SSE4.1 - ECX bit 19
- // CORJIT_FLAG_USE_SSE42 if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSE41
- // SSE4.2 - ECX bit 20
- // CORJIT_FLAG_USE_MOVBE if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSE42
- // MOVBE - ECX bit 22
- // CORJIT_FLAG_USE_POPCNT if the following feature bits are set (input EAX of 1)
- // CORJIT_FLAG_USE_SSE42
- // POPCNT - ECX bit 23
- // CORJIT_FLAG_USE_AVX if the following feature bits are set (input EAX of 1), and xmmYmmStateSupport returns 1:
- // CORJIT_FLAG_USE_SSE42
- // OSXSAVE - ECX bit 27
- // AVX - ECX bit 28
- // XGETBV - XCR0[2:1] 11b
- // CORJIT_FLAG_USE_FMA if the following feature bits are set (input EAX of 1), and xmmYmmStateSupport returns 1:
- // CORJIT_FLAG_USE_AVX
- // FMA - ECX bit 12
- // CORJIT_FLAG_USE_AVX2 if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX
- // AVX2 - EBX bit 5
- // CORJIT_FLAG_USE_AVXVNNI if the following feature bit is set (input EAX of 0x07 and input ECX of 1):
- // CORJIT_FLAG_USE_AVX2
- // AVXVNNI - EAX bit 4
- // CORJIT_FLAG_USE_AVX_512F if the following feature bit is set (input EAX of 0x07 and input ECX of 0), and avx512StateSupport returns 1:
- // CORJIT_FLAG_USE_AVX2
- // AVX512F - EBX bit 16
- // XGETBV - XRC0[7:5] 111b
- // CORJIT_FLAG_USE_AVX_512F_VL if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F
- // AVX512VL - EBX bit 31
- // CORJIT_FLAG_USE_AVX_512BW if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F
- // AVX512BW - EBX bit 30
- // CORJIT_FLAG_USE_AVX_512BW_VL if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F_VL
- // CORJIT_FLAG_USE_AVX_512BW
- // CORJIT_FLAG_USE_AVX_512CD if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F
- // AVX512CD - EBX bit 28
- // CORJIT_FLAG_USE_AVX_512CD_VL if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F_VL
- // CORJIT_FLAG_USE_AVX_512CD
- // CORJIT_FLAG_USE_AVX_512DQ if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F
- // AVX512DQ - EBX bit 7
- // CORJIT_FLAG_USE_AVX_512DQ_VL if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F_VL
- // CORJIT_FLAG_USE_AVX_512DQ
- // CORJIT_FLAG_USE_AVX_512VBMI if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // CORJIT_FLAG_USE_AVX512F
- // AVX512VBMI - ECX bit 1
- // CORJIT_FLAG_USE_BMI1 if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // BMI1 - EBX bit 3
- // CORJIT_FLAG_USE_BMI2 if the following feature bit is set (input EAX of 0x07 and input ECX of 0):
- // BMI2 - EBX bit 8
- // CORJIT_FLAG_USE_LZCNT if the following feature bits are set (input EAX of 80000001H)
- // LZCNT - ECX bit 5
- // synchronously updating VM and JIT.
-
union XarchCpuInfo
{
struct {
@@ -1479,6 +1404,7 @@ void EEJitManager::SetCpuInfo()
CPUCompileFlags.Set(InstructionSet_SSE);
CPUCompileFlags.Set(InstructionSet_SSE2);
+ CPUCompileFlags.Set(InstructionSet_VectorT128);
if ((cpuidInfo[CPUID_ECX] & (1 << 25)) != 0) // AESNI
{
@@ -1516,9 +1442,12 @@ void EEJitManager::SetCpuInfo()
CPUCompileFlags.Set(InstructionSet_POPCNT);
}
- if (((cpuidInfo[CPUID_ECX] & (1 << 27)) != 0) && ((cpuidInfo[CPUID_ECX] & (1 << 28)) != 0)) // OSXSAVE & AVX
+ const int requiredAvxEcxFlags = (1 << 27) // OSXSAVE
+ | (1 << 28); // AVX
+
+ if ((cpuidInfo[CPUID_ECX] & requiredAvxEcxFlags) == requiredAvxEcxFlags)
{
- if(DoesOSSupportAVX() && (xmmYmmStateSupport() == 1)) // XGETBV == 11
+ if(DoesOSSupportAVX() && (xmmYmmStateSupport() == 1)) // XGETBV == 11
{
CPUCompileFlags.Set(InstructionSet_AVX);
@@ -1535,50 +1464,60 @@ void EEJitManager::SetCpuInfo()
{
CPUCompileFlags.Set(InstructionSet_AVX2);
- if (DoesOSSupportAVX512() && (avx512StateSupport() == 1)) // XGETBV XRC0[7:5] == 111
+ if ((maxVectorTBitWidth == 0) || (maxVectorTBitWidth >= 256))
+ {
+ // We allow 256-bit Vector by default
+ CPUCompileFlags.Clear(InstructionSet_VectorT128);
+ CPUCompileFlags.Set(InstructionSet_VectorT256);
+ }
+
+ if (DoesOSSupportAVX512() && (avx512StateSupport() == 1)) // XGETBV XRC0[7:5] == 111
{
- if ((cpuidInfo[CPUID_EBX] & (1 << 16)) != 0) // AVX512F
+ if ((cpuidInfo[CPUID_EBX] & (1 << 16)) != 0) // AVX512F
{
CPUCompileFlags.Set(InstructionSet_AVX512F);
+ // TODO-XArch: Add support for 512-bit Vector
+ assert(!CPUCompileFlags.IsSet(InstructionSet_VectorT512));
+
bool isAVX512_VLSupported = false;
- if ((cpuidInfo[CPUID_EBX] & (1 << 31)) != 0) // AVX512VL
+ if ((cpuidInfo[CPUID_EBX] & (1 << 31)) != 0) // AVX512VL
{
CPUCompileFlags.Set(InstructionSet_AVX512F_VL);
isAVX512_VLSupported = true;
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 30)) != 0) // AVX512BW
+ if ((cpuidInfo[CPUID_EBX] & (1 << 30)) != 0) // AVX512BW
{
CPUCompileFlags.Set(InstructionSet_AVX512BW);
- if (isAVX512_VLSupported) // AVX512BW_VL
+ if (isAVX512_VLSupported) // AVX512BW_VL
{
CPUCompileFlags.Set(InstructionSet_AVX512BW_VL);
}
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 28)) != 0) // AVX512CD
+ if ((cpuidInfo[CPUID_EBX] & (1 << 28)) != 0) // AVX512CD
{
CPUCompileFlags.Set(InstructionSet_AVX512CD);
- if (isAVX512_VLSupported) // AVX512CD_VL
+ if (isAVX512_VLSupported) // AVX512CD_VL
{
CPUCompileFlags.Set(InstructionSet_AVX512CD_VL);
}
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 17)) != 0) // AVX512DQ
+ if ((cpuidInfo[CPUID_EBX] & (1 << 17)) != 0) // AVX512DQ
{
CPUCompileFlags.Set(InstructionSet_AVX512DQ);
- if (isAVX512_VLSupported) // AVX512DQ_VL
+ if (isAVX512_VLSupported) // AVX512DQ_VL
{
CPUCompileFlags.Set(InstructionSet_AVX512DQ_VL);
}
}
- if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // AVX512VBMI
+ if ((cpuidInfo[CPUID_ECX] & (1 << 1)) != 0) // AVX512VBMI
{
CPUCompileFlags.Set(InstructionSet_AVX512VBMI);
- if (isAVX512_VLSupported) // AVX512VBMI_VL
+ if (isAVX512_VLSupported) // AVX512VBMI_VL
{
CPUCompileFlags.Set(InstructionSet_AVX512VBMI_VL);
}
@@ -1601,28 +1540,23 @@ void EEJitManager::SetCpuInfo()
}
}
- if (CLRConfig::GetConfigValue(CLRConfig::INTERNAL_SIMD16ByteOnly) != 0)
- {
- CPUCompileFlags.Clear(InstructionSet_AVX2);
- }
-
if (maxCpuId >= 0x07)
{
__cpuidex(cpuidInfo, 0x00000007, 0x00000000);
- if ((cpuidInfo[CPUID_EBX] & (1 << 3)) != 0) // BMI1
+ if ((cpuidInfo[CPUID_EBX] & (1 << 3)) != 0) // BMI1
{
CPUCompileFlags.Set(InstructionSet_BMI1);
}
- if ((cpuidInfo[CPUID_EBX] & (1 << 8)) != 0) // BMI2
+ if ((cpuidInfo[CPUID_EBX] & (1 << 8)) != 0) // BMI2
{
CPUCompileFlags.Set(InstructionSet_BMI2);
}
if ((cpuidInfo[CPUID_EDX] & (1 << 14)) != 0)
{
- CPUCompileFlags.Set(InstructionSet_X86Serialize); // SERIALIZE
+ CPUCompileFlags.Set(InstructionSet_X86Serialize); // SERIALIZE
}
}
@@ -1633,7 +1567,7 @@ void EEJitManager::SetCpuInfo()
{
__cpuid(cpuidInfo, 0x80000001);
- if ((cpuidInfo[CPUID_ECX] & (1 << 5)) != 0) // LZCNT
+ if ((cpuidInfo[CPUID_ECX] & (1 << 5)) != 0) // LZCNT
{
CPUCompileFlags.Set(InstructionSet_LZCNT);
}
@@ -1656,6 +1590,7 @@ void EEJitManager::SetCpuInfo()
// FP and SIMD support are enabled by default
CPUCompileFlags.Set(InstructionSet_ArmBase);
CPUCompileFlags.Set(InstructionSet_AdvSimd);
+ CPUCompileFlags.Set(InstructionSet_VectorT128);
// PF_ARM_V8_CRYPTO_INSTRUCTIONS_AVAILABLE (30)
if (IsProcessorFeaturePresent(PF_ARM_V8_CRYPTO_INSTRUCTIONS_AVAILABLE))
@@ -1867,7 +1802,6 @@ void EEJitManager::SetCpuInfo()
{
CPUCompileFlags.Clear(InstructionSet_X86Serialize);
}
-
#elif defined(TARGET_ARM64)
if (!CLRConfig::GetConfigValue(CLRConfig::EXTERNAL_EnableHWIntrinsic))
{
diff --git a/src/coreclr/vm/comutilnative.cpp b/src/coreclr/vm/comutilnative.cpp
index 6d457a40248503..f2985222815e20 100644
--- a/src/coreclr/vm/comutilnative.cpp
+++ b/src/coreclr/vm/comutilnative.cpp
@@ -1523,59 +1523,6 @@ FCIMPL3_IVV(INT64, COMInterlocked::CompareExchange64, INT64* location, INT64 val
}
FCIMPLEND
-FCIMPL2_IV(float,COMInterlocked::ExchangeFloat, float *location, float value)
-{
- FCALL_CONTRACT;
-
- if( NULL == location) {
- FCThrow(kNullReferenceException);
- }
-
- LONG ret = InterlockedExchange((LONG *) location, *(LONG*)&value);
- return *(float*)&ret;
-}
-FCIMPLEND
-
-FCIMPL2_IV(double,COMInterlocked::ExchangeDouble, double *location, double value)
-{
- FCALL_CONTRACT;
-
- if( NULL == location) {
- FCThrow(kNullReferenceException);
- }
-
-
- INT64 ret = InterlockedExchange64((INT64 *) location, *(INT64*)&value);
- return *(double*)&ret;
-}
-FCIMPLEND
-
-FCIMPL3_IVV(float,COMInterlocked::CompareExchangeFloat, float *location, float value, float comparand)
-{
- FCALL_CONTRACT;
-
- if( NULL == location) {
- FCThrow(kNullReferenceException);
- }
-
- LONG ret = (LONG)InterlockedCompareExchange((LONG*) location, *(LONG*)&value, *(LONG*)&comparand);
- return *(float*)&ret;
-}
-FCIMPLEND
-
-FCIMPL3_IVV(double,COMInterlocked::CompareExchangeDouble, double *location, double value, double comparand)
-{
- FCALL_CONTRACT;
-
- if( NULL == location) {
- FCThrow(kNullReferenceException);
- }
-
- INT64 ret = (INT64)InterlockedCompareExchange64((INT64*) location, *(INT64*)&value, *(INT64*)&comparand);
- return *(double*)&ret;
-}
-FCIMPLEND
-
FCIMPL2(LPVOID,COMInterlocked::ExchangeObject, LPVOID*location, LPVOID value)
{
FCALL_CONTRACT;
@@ -1637,24 +1584,6 @@ FCIMPL2_IV(INT64,COMInterlocked::ExchangeAdd64, INT64 *location, INT64 value)
}
FCIMPLEND
-FCIMPL0(void, COMInterlocked::FCMemoryBarrier)
-{
- FCALL_CONTRACT;
-
- MemoryBarrier();
- FC_GC_POLL();
-}
-FCIMPLEND
-
-FCIMPL0(void, COMInterlocked::FCMemoryBarrierLoad)
-{
- FCALL_CONTRACT;
-
- VolatileLoadBarrier();
- FC_GC_POLL();
-}
-FCIMPLEND
-
#include
extern "C" void QCALLTYPE Interlocked_MemoryBarrierProcessWide()
diff --git a/src/coreclr/vm/comutilnative.h b/src/coreclr/vm/comutilnative.h
index 0d6d3f8a5f95fe..37e19a14f6ec3a 100644
--- a/src/coreclr/vm/comutilnative.h
+++ b/src/coreclr/vm/comutilnative.h
@@ -227,17 +227,10 @@ class COMInterlocked
static FCDECL2_IV(INT64, Exchange64, INT64 *location, INT64 value);
static FCDECL3(INT32, CompareExchange, INT32* location, INT32 value, INT32 comparand);
static FCDECL3_IVV(INT64, CompareExchange64, INT64* location, INT64 value, INT64 comparand);
- static FCDECL2_IV(float, ExchangeFloat, float *location, float value);
- static FCDECL2_IV(double, ExchangeDouble, double *location, double value);
- static FCDECL3_IVV(float, CompareExchangeFloat, float *location, float value, float comparand);
- static FCDECL3_IVV(double, CompareExchangeDouble, double *location, double value, double comparand);
static FCDECL2(LPVOID, ExchangeObject, LPVOID* location, LPVOID value);
static FCDECL3(LPVOID, CompareExchangeObject, LPVOID* location, LPVOID value, LPVOID comparand);
static FCDECL2(INT32, ExchangeAdd32, INT32 *location, INT32 value);
static FCDECL2_IV(INT64, ExchangeAdd64, INT64 *location, INT64 value);
-
- static FCDECL0(void, FCMemoryBarrier);
- static FCDECL0(void, FCMemoryBarrierLoad);
};
extern "C" void QCALLTYPE Interlocked_MemoryBarrierProcessWide();
diff --git a/src/coreclr/vm/ecalllist.h b/src/coreclr/vm/ecalllist.h
index acda36bace331d..9fc399c31cd5a3 100644
--- a/src/coreclr/vm/ecalllist.h
+++ b/src/coreclr/vm/ecalllist.h
@@ -509,18 +509,12 @@ FCFuncEnd()
FCFuncStart(gInterlockedFuncs)
FCFuncElementSig("Exchange", &gsig_SM_RefInt_Int_RetInt, COMInterlocked::Exchange)
FCFuncElementSig("Exchange", &gsig_SM_RefLong_Long_RetLong, COMInterlocked::Exchange64)
- FCFuncElementSig("Exchange", &gsig_SM_RefDbl_Dbl_RetDbl, COMInterlocked::ExchangeDouble)
- FCFuncElementSig("Exchange", &gsig_SM_RefFlt_Flt_RetFlt, COMInterlocked::ExchangeFloat)
FCFuncElementSig("Exchange", &gsig_SM_RefObj_Obj_RetObj, COMInterlocked::ExchangeObject)
FCFuncElementSig("CompareExchange", &gsig_SM_RefInt_Int_Int_RetInt, COMInterlocked::CompareExchange)
FCFuncElementSig("CompareExchange", &gsig_SM_RefLong_Long_Long_RetLong, COMInterlocked::CompareExchange64)
- FCFuncElementSig("CompareExchange", &gsig_SM_RefDbl_Dbl_Dbl_RetDbl, COMInterlocked::CompareExchangeDouble)
- FCFuncElementSig("CompareExchange", &gsig_SM_RefFlt_Flt_Flt_RetFlt, COMInterlocked::CompareExchangeFloat)
FCFuncElementSig("CompareExchange", &gsig_SM_RefObj_Obj_Obj_RetObj, COMInterlocked::CompareExchangeObject)
FCFuncElementSig("ExchangeAdd", &gsig_SM_RefInt_Int_RetInt, COMInterlocked::ExchangeAdd32)
FCFuncElementSig("ExchangeAdd", &gsig_SM_RefLong_Long_RetLong, COMInterlocked::ExchangeAdd64)
- FCFuncElement("MemoryBarrier", COMInterlocked::FCMemoryBarrier)
- FCFuncElement("ReadMemoryBarrier", COMInterlocked::FCMemoryBarrierLoad)
FCFuncEnd()
FCFuncStart(gJitInfoFuncs)
diff --git a/src/coreclr/vm/i386/stublinkerx86.cpp b/src/coreclr/vm/i386/stublinkerx86.cpp
index 3203e4f4ce3717..fde6801c289b6d 100644
--- a/src/coreclr/vm/i386/stublinkerx86.cpp
+++ b/src/coreclr/vm/i386/stublinkerx86.cpp
@@ -4558,8 +4558,8 @@ VOID StubLinkerCPU::EmitArrayOpStub(const ArrayOpScript* pArrayOpScript)
for (SSIZE_T __i = 0; __i > cnt; __i--)
{
- HALF_SIZE_T skip = cur->val_serie[__i].skip;
- HALF_SIZE_T nptrs = cur->val_serie[__i].nptrs;
+ HALF_SIZE_T skip = (cur->val_serie + __i)->skip;
+ HALF_SIZE_T nptrs = (cur->val_serie + __i)->nptrs;
total += nptrs*sizeof (Object*);
do
{
diff --git a/src/coreclr/vm/interoputil.cpp b/src/coreclr/vm/interoputil.cpp
index 65710a5227be54..d2dc577b30cf71 100644
--- a/src/coreclr/vm/interoputil.cpp
+++ b/src/coreclr/vm/interoputil.cpp
@@ -3114,7 +3114,7 @@ void IUInvokeDispMethod(
// Validate that the target is valid for the specified type.
if (!IsComTargetValidForType(pRefClassObj, pTarget))
- COMPlusThrow(kTargetException, W("RFLCT.Targ_ITargMismatch"));
+ COMPlusThrow(kTargetException, W("RFLCT_Targ_ITargMismatch"));
// If the invoked type is an interface, make sure it is IDispatch based.
if (pInvokedMT->IsInterface())
diff --git a/src/coreclr/vm/invokeutil.cpp b/src/coreclr/vm/invokeutil.cpp
index ea2278e73e37c4..c4ba804a4c493f 100644
--- a/src/coreclr/vm/invokeutil.cpp
+++ b/src/coreclr/vm/invokeutil.cpp
@@ -681,7 +681,7 @@ void InvokeUtil::ValidateObjectTarget(FieldDesc *pField, TypeHandle enclosingTyp
return;
if (!pField->IsStatic() && !*target)
- COMPlusThrow(kTargetException,W("RFLCT.Targ_StatFldReqTarg"));
+ COMPlusThrow(kTargetException,W("RFLCT_Targ_StatFldReqTarg"));
// Verify that the object is of the proper type...
TypeHandle ty = (*target)->GetTypeHandle();
diff --git a/src/coreclr/vm/jitinterface.cpp b/src/coreclr/vm/jitinterface.cpp
index 85d904b566cdd5..15365a90216a84 100644
--- a/src/coreclr/vm/jitinterface.cpp
+++ b/src/coreclr/vm/jitinterface.cpp
@@ -1800,7 +1800,7 @@ uint32_t CEEInfo::getThreadLocalFieldInfo (CORINFO_FIELD_HANDLE field, bool isG
}
assert(typeIndex != TypeIDProvider::INVALID_TYPE_ID);
-
+
EE_TO_JIT_TRANSITION();
return typeIndex;
}
@@ -1830,9 +1830,9 @@ void CEEInfo::getThreadLocalStaticBlocksInfo (CORINFO_THREAD_STATIC_BLOCKS_INFO*
pInfo->offsetOfThreadStaticBlocks = CEEInfo::ThreadLocalOffset(&t_NonGCThreadStaticBlocks);
pInfo->offsetOfMaxThreadStaticBlocks = CEEInfo::ThreadLocalOffset(&t_NonGCMaxThreadStaticBlocks);
}
-
+
pInfo->offsetOfGCDataPointer = static_cast(PtrArray::GetDataOffset());
-
+
JIT_TO_EE_TRANSITION_LEAF();
}
#else
@@ -1864,7 +1864,7 @@ void CEEInfo::getThreadLocalStaticBlocksInfo (CORINFO_THREAD_STATIC_BLOCKS_INFO*
pInfo->offsetOfThreadStaticBlocks = 0;
pInfo->offsetOfMaxThreadStaticBlocks = 0;
pInfo->offsetOfGCDataPointer = 0;
-
+
JIT_TO_EE_TRANSITION_LEAF();
}
#endif // HOST_WINDOWS
@@ -12453,17 +12453,8 @@ CorJitResult invokeCompileMethodHelper(EEJitManager *jitMgr,
CorJitResult ret = CORJIT_SKIPPED; // Note that CORJIT_SKIPPED is an error exit status code
-#ifdef FEATURE_STACK_SAMPLING
- static ConfigDWORD s_stackSamplingEnabled;
- bool samplingEnabled = (s_stackSamplingEnabled.val(CLRConfig::UNSUPPORTED_StackSamplingEnabled) != 0);
-#endif
-
#if defined(ALLOW_SXS_JIT)
- if (FAILED(ret) && jitMgr->m_alternateJit
-#ifdef FEATURE_STACK_SAMPLING
- && (!samplingEnabled || (jitFlags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND)))
-#endif
- )
+ if (FAILED(ret) && jitMgr->m_alternateJit)
{
CORJIT_FLAGS altJitFlags = jitFlags;
altJitFlags.Set(CORJIT_FLAGS::CORJIT_FLAG_ALT_JIT);
@@ -12474,14 +12465,6 @@ CorJitResult invokeCompileMethodHelper(EEJitManager *jitMgr,
nativeEntry,
nativeSizeOfCode);
-#ifdef FEATURE_STACK_SAMPLING
- if (jitFlags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND))
- {
- // Don't bother with failures if we couldn't collect a trace.
- ret = CORJIT_OK;
- }
-#endif // FEATURE_STACK_SAMPLING
-
// If we failed to jit, then fall back to the primary Jit.
if (FAILED(ret))
{
@@ -13099,14 +13082,7 @@ PCODE UnsafeJitFunction(PrepareCodeConfig* config,
{
jitInfo.WriteCode(jitMgr);
#if defined(DEBUGGING_SUPPORTED)
- // Note: if we're only importing (ie, verifying/
- // checking to make sure we could JIT, but not actually generating code (
- // eg, for inlining), then DON'T TELL THE DEBUGGER about this.
- if (!flags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_MCJIT_BACKGROUND)
-#ifdef FEATURE_STACK_SAMPLING
- && !flags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND)
-#endif // FEATURE_STACK_SAMPLING
- )
+ if (!flags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_MCJIT_BACKGROUND))
{
//
// Notify the debugger that we have successfully jitted the function
diff --git a/src/coreclr/vm/methodtablebuilder.cpp b/src/coreclr/vm/methodtablebuilder.cpp
index 0349ca2684e990..45c585da6d2196 100644
--- a/src/coreclr/vm/methodtablebuilder.cpp
+++ b/src/coreclr/vm/methodtablebuilder.cpp
@@ -1135,6 +1135,7 @@ BOOL MethodTableBuilder::CheckIfSIMDAndUpdateSize()
LPCUTF8 className;
LPCUTF8 nameSpace;
+
if (FAILED(GetMDImport()->GetNameOfTypeDef(bmtInternal->pType->GetTypeDefToken(), &className, &nameSpace)))
return false;
@@ -1144,7 +1145,12 @@ BOOL MethodTableBuilder::CheckIfSIMDAndUpdateSize()
CORJIT_FLAGS CPUCompileFlags = ExecutionManager::GetEEJitManager()->GetCPUCompileFlags();
uint32_t numInstanceFieldBytes = 16;
- if (CPUCompileFlags.IsSet(InstructionSet_AVX2))
+ if (CPUCompileFlags.IsSet(InstructionSet_VectorT512))
+ {
+ // TODO-XARCH: The JIT needs to be updated to support 64-byte Vector
+ numInstanceFieldBytes = 32;
+ }
+ else if (CPUCompileFlags.IsSet(InstructionSet_VectorT256))
{
numInstanceFieldBytes = 32;
}
diff --git a/src/coreclr/vm/prestub.cpp b/src/coreclr/vm/prestub.cpp
index 29965fd0477a28..e555da0436607b 100644
--- a/src/coreclr/vm/prestub.cpp
+++ b/src/coreclr/vm/prestub.cpp
@@ -30,10 +30,6 @@
#include "clrtocomcall.h"
#endif
-#ifdef FEATURE_STACK_SAMPLING
-#include "stacksampler.h"
-#endif
-
#ifdef FEATURE_PERFMAP
#include "perfmap.h"
#endif
@@ -858,10 +854,6 @@ PCODE MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig* pConfig, J
}
-#ifdef FEATURE_STACK_SAMPLING
- StackSampler::RecordJittingInfo(this, flags);
-#endif // FEATURE_STACK_SAMPLING
-
#ifdef PROFILING_SUPPORTED
{
BEGIN_PROFILER_CALLBACK(CORProfilerTrackJITInfo());
diff --git a/src/coreclr/vm/riscv64/stubs.cpp b/src/coreclr/vm/riscv64/stubs.cpp
index 4c40d61acb26e2..612462e0184720 100644
--- a/src/coreclr/vm/riscv64/stubs.cpp
+++ b/src/coreclr/vm/riscv64/stubs.cpp
@@ -914,7 +914,7 @@ UMEntryThunk * UMEntryThunk::Decode(void *pCallback)
// stubs (see UMEntryThunkCode::Encode below) then we'll return NULL. Luckily in these scenarios our
// caller will perform a hash lookup on successful return to verify our result in case random unmanaged
// code happens to look like ours.
- if ((pCode->m_code[0] == 0x00009f97) && // auipc t6, 0
+ if ((pCode->m_code[0] == 0x00000f97) && // auipc t6, 0
(pCode->m_code[1] == 0x018fb383) && // ld t2, 24(t6)
(pCode->m_code[2] == 0x010fbf83) && // ld t6, 16(t6)
(pCode->m_code[3] == 0x000f8067)) // jalr x0, 0(t6)
@@ -934,7 +934,7 @@ void UMEntryThunkCode::Encode(UMEntryThunkCode *pEntryThunkCodeRX, BYTE* pTarget
// m_pTargetCode data
// m_pvSecretParam data
- m_code[0] = 0x00009f97; // auipc t6, 0
+ m_code[0] = 0x00000f97; // auipc t6, 0
m_code[1] = 0x018fb383; // ld t2, 24(t6)
m_code[2] = 0x010fbf83; // ld t6, 16(t6)
m_code[3] = 0x000f8067; // jalr x0, 0(t6)
diff --git a/src/coreclr/vm/riscv64/virtualcallstubcpu.hpp b/src/coreclr/vm/riscv64/virtualcallstubcpu.hpp
index 6edfec858bbd88..53a3c2c0ac9187 100644
--- a/src/coreclr/vm/riscv64/virtualcallstubcpu.hpp
+++ b/src/coreclr/vm/riscv64/virtualcallstubcpu.hpp
@@ -44,7 +44,7 @@ struct LookupHolder
// _resolveWorkerTarget
// _token
- _stub._entryPoint[0] = LOOKUP_STUB_FIRST_DWORD; // auipc t6, 0 //0x00000097
+ _stub._entryPoint[0] = LOOKUP_STUB_FIRST_DWORD; // auipc t6, 0 //0x00000f97
_stub._entryPoint[1] = 0x018fb383; //ld t2, 24(t6)
_stub._entryPoint[2] = 0x010fbf83; //ld t6, 16(t6)
_stub._entryPoint[3] = 0x000f8067; //jalr x0, t6, 0
@@ -226,7 +226,7 @@ struct ResolveHolder
_stub._resolveEntryPoint[n++] = 0x005e0333;
// auipc t0, 0
_stub._resolveEntryPoint[n++] = 0x00000297;
- // addi t0, t0, -16
+ // addi t0, t0, -12
_stub._resolveEntryPoint[n++] = 0xff428293;
// lw t6, 0(t0) #t6 = this._hashedToken
@@ -461,7 +461,7 @@ void VTableCallHolder::Initialize(unsigned slot)
*(DWORD*)p = 0x00036e03 | ((UINT32)dataOffset << 20); p += 4;
// add t4, t4, t3
*(DWORD*)p = 0x01ce8eb3; p += 4;
- // ld t4, offsetOfIndirection(t4)
+ // ld t4, 0(t4)
*(DWORD*)p = 0x000ebe83; p += 4;
}
else
@@ -485,7 +485,7 @@ void VTableCallHolder::Initialize(unsigned slot)
}
else
{
- // ld t4, offsetOfIndirection(t4)
+ // ld t4, offsetAfterIndirection(t4)
*(DWORD*)p = 0x000ebe83 | ((UINT32)offsetAfterIndirection << 20); p += 4;
}
diff --git a/src/coreclr/vm/stackingallocator.h b/src/coreclr/vm/stackingallocator.h
index a937bf942201a3..2753de73908b42 100644
--- a/src/coreclr/vm/stackingallocator.h
+++ b/src/coreclr/vm/stackingallocator.h
@@ -227,7 +227,7 @@ private :
Thread *pThread__ACQUIRE_STACKING_ALLOCATOR = GetThread(); \
StackingAllocator *stackingAllocatorName = pThread__ACQUIRE_STACKING_ALLOCATOR->m_stackLocalAllocator; \
bool allocatorOwner__ACQUIRE_STACKING_ALLOCATOR = false; \
- NewHolder heapAllocatedStackingBuffer__ACQUIRE_STACKING_ALLOCATOR; \
+ NewArrayHolder heapAllocatedStackingBuffer__ACQUIRE_STACKING_ALLOCATOR; \
\
if (stackingAllocatorName == NULL) \
{ \
@@ -237,10 +237,11 @@ private :
} \
else \
{\
- stackingAllocatorName = new (nothrow) StackingAllocator; \
- if (stackingAllocatorName == NULL) \
+ char *pBuffer__ACQUIRE_STACKING_ALLOCATOR = new (nothrow) char[sizeof(StackingAllocator)]; \
+ if (pBuffer__ACQUIRE_STACKING_ALLOCATOR == NULL) \
ThrowOutOfMemory(); \
- heapAllocatedStackingBuffer__ACQUIRE_STACKING_ALLOCATOR = stackingAllocatorName; \
+ heapAllocatedStackingBuffer__ACQUIRE_STACKING_ALLOCATOR = pBuffer__ACQUIRE_STACKING_ALLOCATOR; \
+ stackingAllocatorName = new (pBuffer__ACQUIRE_STACKING_ALLOCATOR) StackingAllocator; \
}\
allocatorOwner__ACQUIRE_STACKING_ALLOCATOR = true; \
} \
diff --git a/src/coreclr/vm/stacksampler.cpp b/src/coreclr/vm/stacksampler.cpp
deleted file mode 100644
index f49218e8590483..00000000000000
--- a/src/coreclr/vm/stacksampler.cpp
+++ /dev/null
@@ -1,433 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-//
-// Summary:
-// --------
-//
-// StackSampler is intended to identify methods where the process is spending most of its time
-// and to re-JIT such methods in the background. Call these methods hot.
-//
-// Identifying hot methods:
-// ========================
-//
-// There is no easy way to tell at a given point in execution whether in the future an unseen
-// or un-hot method will become hot; So we track an evolving list of hot methods.
-//
-// We identify hot methods by suspending the runtime every "m" milliseconds. This operation
-// freezes all the threads. We now get a list of threads that are executing and walk their
-// stacks to get the managed method at the top of their stacks. The sampled list of methods
-// for each thread constitute a single sample. Once we obtain a sample, the threads are thawed.
-//
-// The more a method is present in samples, it is clear that the process is spending its time
-// in that method at several given points in time.
-//
-// We track this information on a per method basis, the count of its occurrences in each sample
-// using a hash map.
-//
-// Note:
-// =====
-// o Using the above technique we have only identified top methods at a given point in the execution.
-// The list of hot methods keeps evolving as we get more samples. Only at the process end can we
-// say that the evolving list of hot methods is THE list of hot methods for the whole process.
-// o Because we get the top managed method in the thread, this includes time spent by that method
-// in helper calls.
-// o If GC is in progress it has suspended the threads, and we would not be able to suspend the threads.
-//
-// Future Consideration:
-// =====================
-// We could track "trending" methods, as methods decay out so we can keep only "trending" variants
-// in the code manager and kick out "past" hot methods.
-//
-// Jitting in the background:
-// ==========================
-// Once we have the hot methods at a given point in time, we JIT them. The decision to JIT is configurable
-// by configuring the number of times a method is seen in samples before we would JIT it.
-// For example, if we are sampling every 10 msec and if we expect methods that spend at least 1 second
-// to be hot, then the number of times to see this method is roughly, 100, it is best to be conservative
-// with this number.
-//
-// Note that we JIT the evolving list of methods, without knowing ahead of time that the methods we JIT
-// will be the final top "n" hot methods due to the lack of knowledge of when the process will end.
-// This means we would over-JIT but this only yields results with false negatives.
-//
-// Currently we JIT in the background only once (with the current goal of getting a trace.)
-//
-// Note:
-// =====
-// o To run the JIT in the background, we try our best to JIT in the same app domain in which the original
-// JITting happened. But if we fail to acquire (ngen'ed method) or enter (unloaded domain) the original domain,
-// we then try to JIT it under the thread's app domain in which the method was last seen to be executing.
-//
-// o The JIT to use is configurable with DOTNET_AltJitName when DOTNET_StackSampling is enabled.
-//
-// o One use case is to collect traces as an .mc file from SuperPMI Shim JIT.
-//
-// Jitting parameters:
-// ==========================
-// The prestub tells us at JITting time using "RecordJittingInfo" to record the parameters used to JIT
-// originally. We use these parameters to JIT in the background when we decide to JIT the method.
-//
-
-
-#include "common.h"
-#include "corjit.h"
-#include "stacksampler.h"
-#include "threadsuspend.h"
-
-#ifdef FEATURE_STACK_SAMPLING
-
-// Global instance of the sampler
-StackSampler* g_pStackSampler = nullptr;
-
-// Create an instance of the stack sampler if sampling is enabled.
-void StackSampler::Init()
-{
- STANDARD_VM_CONTRACT;
-
- bool samplingEnabled = (CLRConfig::GetConfigValue(CLRConfig::UNSUPPORTED_StackSamplingEnabled) != 0);
- if (samplingEnabled)
- {
- g_pStackSampler = new (nothrow) StackSampler();
- }
-}
-
-// ThreadProc for performing sampling and JITting.
-/* static */
-DWORD __stdcall StackSampler::SamplingThreadProc(void* arg)
-{
- WRAPPER_NO_CONTRACT;
-
- StackSampler* pThis = (StackSampler*) arg;
- pThis->ThreadProc();
- return 0;
-}
-
-// Constructor
-StackSampler::StackSampler()
- : m_crstJitInfo(CrstStackSampler, (CrstFlags)(CRST_UNSAFE_ANYMODE))
- , m_nSampleEvery(s_knDefaultSamplingIntervalMsec)
- , m_nSampleAfter(0)
- , m_nNumMethods(s_knDefaultNumMethods)
-{
- // When to start sampling after the thread launch.
- int nSampleAfter = CLRConfig::GetConfigValue(CLRConfig::UNSUPPORTED_StackSamplingAfter);
- if (nSampleAfter != INT_MAX && nSampleAfter >= 0)
- {
- m_nSampleAfter = nSampleAfter;
- }
-
- // How frequently to sample.
- int nSampleEvery = CLRConfig::GetConfigValue(CLRConfig::UNSUPPORTED_StackSamplingEvery);
- if (nSampleEvery != INT_MAX && nSampleEvery > 0)
- {
- m_nSampleEvery = nSampleEvery;
- }
-
- // Max number of methods to track.
- int nNumMethods = CLRConfig::GetConfigValue(CLRConfig::UNSUPPORTED_StackSamplingNumMethods);
- if (nNumMethods != INT_MAX && nNumMethods > 0)
- {
- m_nNumMethods = nNumMethods;
- }
-
- // Launch the thread.
- m_pThread = SetupUnstartedThread();
- m_pThread->SetBackground(TRUE);
-
- // Passing "this" to the thread in the constructor.
- if (m_pThread->CreateNewThread(1*1024*1024, SamplingThreadProc, this))
- {
- m_pThread->StartThread();
- }
-}
-
-// Is "pMD" a good method, that is suitable for tracking as HOT and
-// JITting in the background.
-bool IsGoodMethodDesc(MethodDesc* pMD)
-{
- LIMITED_METHOD_CONTRACT;
- return !(pMD == nullptr || !pMD->IsIL() || pMD->IsUnboxingStub() || pMD->GetMethodTable()->Collectible());
-}
-
-//
-// An opportunity to record the parameters passed to the JIT at the time of JITting this method.
-/* static */
-void StackSampler::RecordJittingInfo(MethodDesc* pMD, CORJIT_FLAGS flags)
-{
- WRAPPER_NO_CONTRACT;
- if (g_pStackSampler == nullptr)
- {
- return;
- }
- // Skip if this is not a good method desc.
- if (!IsGoodMethodDesc(pMD))
- {
- return;
- }
- // Record in the hash map.
- g_pStackSampler->RecordJittingInfoInternal(pMD, flags);
-}
-
-void StackSampler::RecordJittingInfoInternal(MethodDesc* pMD, CORJIT_FLAGS flags)
-{
- JitInfoHashEntry entry(pMD, flags);
-
- // Record the domain in the hash map.
- {
- CrstHolder ch(&m_crstJitInfo);
- m_jitInfo.AddOrReplace(entry);
- }
-}
-
-// Stack walk callback data.
-struct WalkInfo
-{
- StackSampler* pThis;
-
- // The thread in which the walk is happening and the method is executing.
- // Used to obtain the app domain.
- Thread* pMdThread;
-};
-
-// Visitor for stack walk callback.
-StackWalkAction StackSampler::StackWalkCallback(CrawlFrame* pCf, VOID* data)
-{
- WRAPPER_NO_CONTRACT;
-
- WalkInfo* info = (WalkInfo*) data;
- return ((StackSampler*) info->pThis)->CrawlFrameVisitor(pCf, info->pMdThread);
-}
-
-// Stack walk visitor helper to maintain the hash map of method desc, their count
-// and the thread's domain in which the method is executing.
-StackWalkAction StackSampler::CrawlFrameVisitor(CrawlFrame* pCf, Thread* pMdThread)
-{
- CONTRACTL
- {
- THROWS;
- GC_NOTRIGGER;
- MODE_ANY;
- }
- CONTRACTL_END;
-
- MethodDesc* pMD = pCf->GetFunction();
-
- // Filter out methods we don't care about
- if (!IsGoodMethodDesc(pMD))
- {
- return SWA_CONTINUE;
- }
-
- // Lookup the method desc and obtain info.
- CountInfo info;
- m_countInfo.Lookup(pMD, &info);
-
- // Record the current domain ID of the method's thread, i.e.,
- // the method is last known to be executing.
- info.uCount++;
-
- // Put the info back.
- m_countInfo.AddOrReplace(CountInfoHashEntry(pMD, info));
-
- // We got the top good one, skip.
- return SWA_ABORT;
-}
-
-// Thread routine that suspends the runtime, walks the other threads' stacks to get the
-// top managed method. Restarts the runtime after samples are collected. Identifies top
-// methods from the samples and re-JITs them in the background.
-void StackSampler::ThreadProc()
-{
- CONTRACTL
- {
- THROWS;
- GC_TRIGGERS;
- MODE_ANY;
- }
- CONTRACTL_END;
-
- // Complete the thread init.
- if (!m_pThread->HasStarted())
- {
- return;
- }
-
- // User asked us to sample after certain time.
- m_pThread->UserSleep(m_nSampleAfter);
-
- WalkInfo info = { this, nullptr };
-
- while (true)
- {
- EX_TRY
- {
- // Suspend the runtime.
- ThreadSuspend::SuspendEE(ThreadSuspend::SUSPEND_OTHER);
-
- // Walk all other threads.
- Thread* pThread = nullptr;
- while ((pThread = ThreadStore::GetThreadList(pThread)) != nullptr)
- {
- if (pThread == m_pThread)
- {
- continue;
- }
- // TODO: Detect if thread is suspended by user before we suspended and skip.
-
- info.pMdThread = pThread;
-
- // Walk the frames.
- pThread->StackWalkFrames(StackWalkCallback, &info, FUNCTIONSONLY | ALLOW_ASYNC_STACK_WALK);
- }
-
- // Restart the runtime.
- ThreadSuspend::RestartEE(FALSE, TRUE);
-
- // JIT the methods that frequently occur in samples.
- JitFrequentMethodsInSamples();
- }
- EX_CATCH
- {
- }
- EX_END_CATCH(SwallowAllExceptions);
-
- // User asked us to sample every few seconds.
- // TODO: Measure time to JIT using CycleTimer and subtract from the time we sleep every time.
- m_pThread->UserSleep(m_nSampleEvery);
- }
-}
-
-// Find the most frequent method in the samples and JIT them.
-void StackSampler::JitFrequentMethodsInSamples()
-{
- struct Count
- {
- MethodDesc* pMD;
- CountInfo info;
-
- static int __cdecl Decreasing(const void* e1, const void* e2)
- {
- return ((Count*) e2)->info.uCount - ((Count*) e1)->info.uCount;
- }
- };
-
- // We want to keep a max-heap of the top frequent methods in the samples.
- NewHolder freq(new (nothrow) Count[m_nNumMethods]);
-
- //
- // For each element in the samples, call it incoming, add to the "frequent" list
- // if the list has space to hold the incoming element.
- //
- // If the list doesn't have space, replace the min frequent element in the list
- // with the incoming element, if the latter is more frequent.
- //
- unsigned uLength = 0;
- for (CountInfoHash::Iterator iter = m_countInfo.Begin(), end = m_countInfo.End(); iter != end; iter++)
- {
- Count c = { (*iter).Key(), (*iter).Value() };
-
- // Is the list full? Drop the min element if incoming is more frequent.
- if (uLength == m_nNumMethods)
- {
- // Find the min element and the min index.
- unsigned uMinIndex = 0;
- unsigned uMin = freq[0].info.uCount;
- for (unsigned i = 1; i < uLength; ++i)
- {
- if (uMin > freq[i].info.uCount)
- {
- uMin = freq[i].info.uCount;
- uMinIndex = i;
- }
- }
- if (uMin < c.info.uCount)
- {
- freq[uMinIndex] = c;
- }
- }
- // List is not full, just add the incoming element.
- else
- {
- freq[uLength] = c;
- uLength++;
- }
- }
-
- // Sort by most frequent element first.
- qsort(freq, uLength, sizeof(Count), Count::Decreasing);
-
-#ifdef _DEBUG
- LOG((LF_JIT, LL_INFO100000, "-----------HOT METHODS-------\n"));
- for (unsigned i = 0; i < uLength; ++i)
- {
- // printf("%s:%s, %u\n", freq[i].pMD->GetMethodTable()->GetClass()->GetDebugClassName(), freq[i].pMD->GetName(), freq[i].info.uCount);
- LOG((LF_JIT, LL_INFO100000, "%s:%s, %u\n", freq[i].pMD->GetMethodTable()->GetClass()->GetDebugClassName(), freq[i].pMD->GetName(), freq[i].info.uCount));
- }
- LOG((LF_JIT, LL_INFO100000, "-----------------------------\n"));
-#endif
-
- // Do the JITting.
- for (unsigned i = 0; i < uLength; ++i)
- {
- // If not already JITted and the method is frequent enough to be important.
- if (!freq[i].info.fJitted && freq[i].info.uCount > s_knDefaultCountForImportance)
- {
- // Try to get the original app domain ID in which the method was JITTed, if not
- // use the app domain ID the method was last seen executing.
- JitAndCollectTrace(freq[i].pMD);
- }
- }
-}
-
-// Invoke the JIT for the method desc. Switch to the appropriate domain.
-void StackSampler::JitAndCollectTrace(MethodDesc* pMD)
-{
- CONTRACTL
- {
- NOTHROW;
- GC_TRIGGERS;
- MODE_ANY;
- }
- CONTRACTL_END;
-
- // Indicate to the JIT or the JIT interface that we are JITting
- // in the background for stack sampling.
- CORJIT_FLAGS flags(CORJIT_FLAGS::CORJIT_FLAG_SAMPLING_JIT_BACKGROUND);
-
- _ASSERTE(pMD->IsIL());
-
- EX_TRY
- {
- {
- GCX_PREEMP();
-
- COR_ILMETHOD_DECODER::DecoderStatus status;
- NewHolder pDecoder(
- new COR_ILMETHOD_DECODER(pMD->GetILHeader(),
- pMD->GetMDImport(),
- &status));
-
-#ifdef _DEBUG
- LOG((LF_JIT, LL_INFO100000, "Jitting the hot method desc using SuperPMI in the background thread -> "));
- LOG((LF_JIT, LL_INFO100000, "%s:%s\n", pMD->GetMethodTable()->GetClass()->GetDebugClassName(), pMD->GetName()));
-#endif
- NativeCodeVersion natCodeVer(pMD);
- PrepareCodeConfigBuffer cfgBuffer(natCodeVer);
- PCODE pCode = UnsafeJitFunction(cfgBuffer.GetConfig(), pDecoder, flags);
- }
-
- // Update that this method has been already JITted.
- CountInfo info;
- m_countInfo.Lookup(pMD, &info);
- info.fJitted = true;
- m_countInfo.AddOrReplace(CountInfoHashEntry(pMD, info));
- }
- EX_CATCH
- {
- }
- EX_END_CATCH(SwallowAllExceptions)
-
-}
-
-#endif // FEATURE_STACK_SAMPLING
diff --git a/src/coreclr/vm/stacksampler.h b/src/coreclr/vm/stacksampler.h
deleted file mode 100644
index d489d1366ad930..00000000000000
--- a/src/coreclr/vm/stacksampler.h
+++ /dev/null
@@ -1,77 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-/*++
-
-Module Name:
-
- StackSampler.h
-
---*/
-
-#ifndef __STACK_SAMPLER_H
-#define __STACK_SAMPLER_H
-#endif
-
-#ifdef FEATURE_STACK_SAMPLING
-
-class StackSampler
-{
-public:
- // Interface
- static void Init();
- static void RecordJittingInfo(MethodDesc* pMD, CORJIT_FLAGS flags);
-
-private:
-
- // Methods
- StackSampler();
- ~StackSampler();
-
- static DWORD __stdcall SamplingThreadProc(void* arg);
-
- static StackWalkAction StackWalkCallback(CrawlFrame* pCf, VOID* data);
-
- StackWalkAction CrawlFrameVisitor(CrawlFrame* pCf, Thread* pMdThread);
-
- void ThreadProc();
-
- void JitFrequentMethodsInSamples();
-
- void JitAndCollectTrace(MethodDesc* pMD);
-
- void RecordJittingInfoInternal(MethodDesc* pMD, CORJIT_FLAGS flags);
-
-
- // Constants
- static const int s_knDefaultSamplingIntervalMsec = 100;
- static const int s_knDefaultNumMethods = 32;
- static const int s_knDefaultCountForImportance = 0; // TODO: Set to some reasonable value.
-
- // Typedefs
- struct CountInfo;
- typedef MapSHash CountInfoHash;
- typedef CountInfoHash::element_t CountInfoHashEntry;
-
- typedef MapSHash JitInfoHash;
- typedef JitInfoHash::element_t JitInfoHashEntry;
-
- // Nested types
- struct CountInfo
- {
- unsigned uCount;
- bool fJitted;
- CountInfo() : uCount(0), fJitted(false) {}
- };
-
- // Fields
- Crst m_crstJitInfo;
- CountInfoHash m_countInfo;
- JitInfoHash m_jitInfo;
- Thread* m_pThread;
- unsigned m_nSampleEvery;
- unsigned m_nSampleAfter;
- unsigned m_nNumMethods;
-};
-#endif // FEATURE_STACK_SAMPLING
-
diff --git a/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/ILCompilerRIDs.props b/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/ILCompilerRIDs.props
index b7c6ca32c9f14e..1d51a78880ae2d 100644
--- a/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/ILCompilerRIDs.props
+++ b/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/ILCompilerRIDs.props
@@ -1,5 +1,7 @@
+
diff --git a/src/installer/pkg/projects/netcoreappRIDs.props b/src/installer/pkg/projects/netcoreappRIDs.props
index fbc6e4dc4c1a98..4c73b3dcb3c2b5 100644
--- a/src/installer/pkg/projects/netcoreappRIDs.props
+++ b/src/installer/pkg/projects/netcoreappRIDs.props
@@ -17,9 +17,6 @@
x86
-
- arm
- arm64
diff --git a/src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props b/src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
index 14c9aa9b4213dc..3a961b3078c9d1 100644
--- a/src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
+++ b/src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props
@@ -250,6 +250,7 @@
+
diff --git a/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj b/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj
index c28c6ea8fb53ca..f697e4d716a56c 100644
--- a/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj
+++ b/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj
@@ -12,7 +12,7 @@
dotnet-crossgen2crossgen2
- linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64;win-arm
+ linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64falsetools/true
@@ -85,9 +85,9 @@
$(TargetOSComponent)-$(TargetArchitecture)
-
+
- <_Crossgen2SymbolFilesToPackage Include="@(Reference->'$(CoreCLRArtifactsPath)PDB\%(FileName).pdb')" />
+ <_Crossgen2SymbolFilesToPackage Include="@(_CrossgenPublishFiles)" Condition="'%(Extension)' == '.pdb'" />
<_Crossgen2SymbolFilesToPackage Include="@(NativeRuntimeAsset->'$(CoreCLRArtifactsPdbDir)%(FileName).pdb')" Condition="'$(TargetOS)' == 'windows' and '%(FileName)' != 'crossgen2'" />
<_Crossgen2SymbolFilesToPackage Include="@(NativeRuntimeAsset->'$(CoreCLRArtifactsPath)%(FileName)%(Extension)$(SymbolsSuffix)')" Condition="'$(TargetOS)' != 'windows' and '%(FileName)' != 'crossgen2'" />
diff --git a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.kn-IN.resx b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.kn-IN.resx
index b0ec782f9a2a68..04130790681f41 100644
--- a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.kn-IN.resx
+++ b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.kn-IN.resx
@@ -118,6 +118,6 @@
System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
- ನಮಸ್ಕಾರ
+ [kn-IN]
\ No newline at end of file
diff --git a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.resx b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.resx
index 81893ab57f0af6..0fad9b04a4938f 100644
--- a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.resx
+++ b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.resx
@@ -118,6 +118,6 @@
System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
- Hello
+ [default]
\ No newline at end of file
diff --git a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.ta-IN.resx b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.ta-IN.resx
index b42699bbee2136..4acc307a1b73db 100644
--- a/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.ta-IN.resx
+++ b/src/installer/tests/Assets/TestProjects/LocalizedApp/Hello.ta-IN.resx
@@ -118,6 +118,6 @@
System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
- வணக்கம்
+ [ta-IN]
\ No newline at end of file
diff --git a/src/installer/tests/Assets/TestUtils/TestProjects.targets b/src/installer/tests/Assets/TestUtils/TestProjects.targets
index 3aaec8fbd355aa..bb5740e9b240ec 100644
--- a/src/installer/tests/Assets/TestUtils/TestProjects.targets
+++ b/src/installer/tests/Assets/TestUtils/TestProjects.targets
@@ -15,7 +15,7 @@
LatestRuntimeFrameworkVersion="7.0.0"
RuntimeFrameworkName="Microsoft.NETCore.App"
RuntimePackNamePatterns="Microsoft.NETCore.App.Runtime.**RID**"
- RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
+ RuntimePackRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
TargetFramework="$(NetCoreAppCurrent)"
TargetingPackName="Microsoft.NETCore.App.Ref"
TargetingPackVersion="7.0.0"
@@ -25,7 +25,7 @@
ExcludedRuntimeIdentifiers="android"
AppHostPackNamePattern="Microsoft.NETCore.App.Host.**RID**"
AppHostPackVersion="7.0.0"
- AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
+ AppHostRuntimeIdentifiers="linux-arm;linux-arm64;linux-musl-arm64;linux-musl-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;linux-s390x;linux-bionic-arm;linux-bionic-arm64;linux-bionic-x64;linux-bionic-x86"
TargetFramework="$(NetCoreAppCurrent)"
Condition="'@(KnownAppHostPack)' == '' or !@(KnownAppHostPack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))" />
diff --git a/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleLocalizedApp.cs b/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleLocalizedApp.cs
index 633dd4f266d402..13a1ad7e8aec1e 100644
--- a/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleLocalizedApp.cs
+++ b/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleLocalizedApp.cs
@@ -26,20 +26,12 @@ public void Bundled_Localized_App_Run_Succeeds()
var fixture = sharedTestState.TestFixture.Copy();
var singleFile = BundleSelfContainedApp(fixture);
- if (OperatingSystem.IsWindows())
- {
- // Set code page to output unicode characters.
- Command.Create("chcp 65001").Execute();
- }
-
Command.Create(singleFile)
.CaptureStdErr()
.CaptureStdOut()
.Execute()
- .Should()
- .Pass()
- .And
- .HaveStdOutContaining("\u0CA8\u0CAE\u0CB8\u0CCD\u0C95\u0CBE\u0CB0! \u0BB5\u0BA3\u0B95\u0BCD\u0B95\u0BAE\u0BCD! Hello!");
+ .Should().Pass()
+ .And.HaveStdOutContaining("[kn-IN]! [ta-IN]! [default]!");
}
public class SharedTestState : SharedTestStateBase, IDisposable
diff --git a/src/installer/tests/Microsoft.NET.HostModel.Tests/Microsoft.NET.HostModel.AppHost.Tests/AppHostUsedWithSymbolicLinks.cs b/src/installer/tests/Microsoft.NET.HostModel.Tests/Microsoft.NET.HostModel.AppHost.Tests/AppHostUsedWithSymbolicLinks.cs
index 3b945dae016c89..53a4b3a1c94f19 100644
--- a/src/installer/tests/Microsoft.NET.HostModel.Tests/Microsoft.NET.HostModel.AppHost.Tests/AppHostUsedWithSymbolicLinks.cs
+++ b/src/installer/tests/Microsoft.NET.HostModel.Tests/Microsoft.NET.HostModel.AppHost.Tests/AppHostUsedWithSymbolicLinks.cs
@@ -192,7 +192,6 @@ public void Put_app_directory_behind_symlink_and_use_dotnet()
}
[Fact]
- // If enabled, this tests will need to set the console code page to output unicode characters: Command.Create("chcp 65001").Execute();
[SkipOnPlatform(TestPlatforms.Windows, "Creating symbolic links requires administrative privilege on Windows, so skip test.")]
public void Put_satellite_assembly_behind_symlink()
{
@@ -219,7 +218,7 @@ public void Put_satellite_assembly_behind_symlink()
.CaptureStdOut()
.Execute()
.Should().Pass()
- .And.HaveStdOutContaining("\u0CA8\u0CAE\u0CB8\u0CCD\u0C95\u0CBE\u0CB0! \u0BB5\u0BA3\u0B95\u0BCD\u0B95\u0BAE\u0BCD! Hello!");
+ .And.HaveStdOutContaining("[kn-IN]! [ta-IN]! [default]!");
}
public class SharedTestState : IDisposable
diff --git a/src/libraries/Common/src/Interop/Android/System.Security.Cryptography.Native.Android/Interop.Evp.DigestAlgs.cs b/src/libraries/Common/src/Interop/Android/System.Security.Cryptography.Native.Android/Interop.Evp.DigestAlgs.cs
index 8ba0658735a297..d7325ca8e8574c 100644
--- a/src/libraries/Common/src/Interop/Android/System.Security.Cryptography.Native.Android/Interop.Evp.DigestAlgs.cs
+++ b/src/libraries/Common/src/Interop/Android/System.Security.Cryptography.Native.Android/Interop.Evp.DigestAlgs.cs
@@ -47,12 +47,33 @@ internal static IntPtr EvpSha512() =>
internal static IntPtr HashAlgorithmToEvp(string hashAlgorithmId) => hashAlgorithmId switch
{
- nameof(HashAlgorithmName.SHA1) => EvpSha1(),
- nameof(HashAlgorithmName.SHA256) => EvpSha256(),
- nameof(HashAlgorithmName.SHA384) => EvpSha384(),
- nameof(HashAlgorithmName.SHA512) => EvpSha512(),
- nameof(HashAlgorithmName.MD5) => EvpMd5(),
+ HashAlgorithmNames.SHA1 => EvpSha1(),
+ HashAlgorithmNames.SHA256 => EvpSha256(),
+ HashAlgorithmNames.SHA384 => EvpSha384(),
+ HashAlgorithmNames.SHA512 => EvpSha512(),
+ HashAlgorithmNames.MD5 => EvpMd5(),
+ HashAlgorithmNames.SHA3_256 or HashAlgorithmNames.SHA3_384 or HashAlgorithmNames.SHA3_512 =>
+ throw new PlatformNotSupportedException(),
_ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmId))
};
+
+ internal static bool HashAlgorithmSupported(string hashAlgorithmId)
+ {
+ switch (hashAlgorithmId)
+ {
+ case HashAlgorithmNames.SHA1:
+ case HashAlgorithmNames.SHA256:
+ case HashAlgorithmNames.SHA384:
+ case HashAlgorithmNames.SHA512:
+ case HashAlgorithmNames.MD5:
+ return true;
+ case HashAlgorithmNames.SHA3_256:
+ case HashAlgorithmNames.SHA3_384:
+ case HashAlgorithmNames.SHA3_512:
+ return false;
+ default:
+ throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmId));
+ }
+ }
}
}
diff --git a/src/libraries/Common/src/Interop/Browser/Interop.Normalization.cs b/src/libraries/Common/src/Interop/Browser/Interop.Normalization.cs
deleted file mode 100644
index b55cf5307d704f..00000000000000
--- a/src/libraries/Common/src/Interop/Browser/Interop.Normalization.cs
+++ /dev/null
@@ -1,17 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-using System.Runtime.CompilerServices;
-using System.Text;
-
-internal static partial class Interop
-{
- internal static unsafe partial class JsGlobalization
- {
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- internal static extern unsafe int IsNormalized(NormalizationForm normalizationForm, in string source, out int exceptionalResult, out object result);
-
- [MethodImplAttribute(MethodImplOptions.InternalCall)]
- internal static extern unsafe int NormalizeString(NormalizationForm normalizationForm, in string source, char* dstBuffer, int dstBufferCapacity, out int exceptionalResult, out object result);
- }
-}
diff --git a/src/libraries/Common/src/Interop/Interop.Utils.cs b/src/libraries/Common/src/Interop/Interop.Utils.cs
index b7cd0665c96483..498236c4ac4608 100644
--- a/src/libraries/Common/src/Interop/Interop.Utils.cs
+++ b/src/libraries/Common/src/Interop/Interop.Utils.cs
@@ -12,7 +12,7 @@ internal static partial class Interop
/// increasing buffer until the size is big enough.
///
internal static bool CallStringMethod(
- SpanFunc interopCall,
+ SpanFunc interopCall,
TArg1 arg1, TArg2 arg2, TArg3 arg3,
out string? result)
{
@@ -20,19 +20,19 @@ internal static bool CallStringMethod(
const int MaxHeapSize = 1280; // max from previous version of the code, starting at 80 and doubling four times
Span buffer = stackalloc char[InitialSize];
- Interop.Globalization.ResultCode resultCode = interopCall(buffer, arg1, arg2, arg3);
+ Globalization.ResultCode resultCode = interopCall(buffer, arg1, arg2, arg3);
- if (resultCode == Interop.Globalization.ResultCode.Success)
+ if (resultCode == Globalization.ResultCode.Success)
{
result = buffer.Slice(0, buffer.IndexOf('\0')).ToString();
return true;
}
- if (resultCode == Interop.Globalization.ResultCode.InsufficientBuffer)
+ if (resultCode == Globalization.ResultCode.InsufficientBuffer)
{
// Increase the string size and try again
buffer = new char[MaxHeapSize];
- if (interopCall(buffer, arg1, arg2, arg3) == Interop.Globalization.ResultCode.Success)
+ if (interopCall(buffer, arg1, arg2, arg3) == Globalization.ResultCode.Success)
{
result = buffer.Slice(0, buffer.IndexOf('\0')).ToString();
return true;
diff --git a/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.DigestAlgs.cs b/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.DigestAlgs.cs
index abaefab1dab6fc..dec9bdd84a5ddf 100644
--- a/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.DigestAlgs.cs
+++ b/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.DigestAlgs.cs
@@ -14,45 +14,128 @@ internal static partial class Crypto
private static volatile IntPtr s_evpSha256;
private static volatile IntPtr s_evpSha384;
private static volatile IntPtr s_evpSha512;
+ private static volatile IntPtr s_evpSha3_256;
+ private static volatile IntPtr s_evpSha3_384;
+ private static volatile IntPtr s_evpSha3_512;
+ private static volatile bool s_evpSha3_256Cached;
+ private static volatile bool s_evpSha3_384Cached;
+ private static volatile bool s_evpSha3_512Cached;
[LibraryImport(Libraries.CryptoNative)]
private static partial IntPtr CryptoNative_EvpMd5();
- internal static IntPtr EvpMd5() =>
+ private static IntPtr EvpMd5() =>
s_evpMd5 != IntPtr.Zero ? s_evpMd5 : (s_evpMd5 = CryptoNative_EvpMd5());
[LibraryImport(Libraries.CryptoNative)]
- internal static partial IntPtr CryptoNative_EvpSha1();
+ private static partial IntPtr CryptoNative_EvpSha1();
- internal static IntPtr EvpSha1() =>
+ private static IntPtr EvpSha1() =>
s_evpSha1 != IntPtr.Zero ? s_evpSha1 : (s_evpSha1 = CryptoNative_EvpSha1());
[LibraryImport(Libraries.CryptoNative)]
- internal static partial IntPtr CryptoNative_EvpSha256();
+ private static partial IntPtr CryptoNative_EvpSha256();
- internal static IntPtr EvpSha256() =>
+ private static IntPtr EvpSha256() =>
s_evpSha256 != IntPtr.Zero ? s_evpSha256 : (s_evpSha256 = CryptoNative_EvpSha256());
[LibraryImport(Libraries.CryptoNative)]
- internal static partial IntPtr CryptoNative_EvpSha384();
+ private static partial IntPtr CryptoNative_EvpSha384();
- internal static IntPtr EvpSha384() =>
+ private static IntPtr EvpSha384() =>
s_evpSha384 != IntPtr.Zero ? s_evpSha384 : (s_evpSha384 = CryptoNative_EvpSha384());
[LibraryImport(Libraries.CryptoNative)]
- internal static partial IntPtr CryptoNative_EvpSha512();
+ private static partial IntPtr CryptoNative_EvpSha512();
- internal static IntPtr EvpSha512() =>
+ private static IntPtr EvpSha512() =>
s_evpSha512 != IntPtr.Zero ? s_evpSha512 : (s_evpSha512 = CryptoNative_EvpSha512());
- internal static IntPtr HashAlgorithmToEvp(string hashAlgorithmId) => hashAlgorithmId switch
+ [LibraryImport(Libraries.CryptoNative)]
+ private static partial IntPtr CryptoNative_EvpSha3_256();
+
+ private static IntPtr EvpSha3_256()
+ {
+ if (!s_evpSha3_256Cached)
+ {
+ s_evpSha3_256 = CryptoNative_EvpSha3_256();
+ s_evpSha3_256Cached = true;
+ }
+
+ return s_evpSha3_256;
+ }
+
+ [LibraryImport(Libraries.CryptoNative)]
+ private static partial IntPtr CryptoNative_EvpSha3_384();
+
+ private static IntPtr EvpSha3_384()
+ {
+ if (!s_evpSha3_384Cached)
+ {
+ s_evpSha3_384 = CryptoNative_EvpSha3_384();
+ s_evpSha3_384Cached = true;
+ }
+
+ return s_evpSha3_384;
+ }
+
+ [LibraryImport(Libraries.CryptoNative)]
+ private static partial IntPtr CryptoNative_EvpSha3_512();
+
+ private static IntPtr EvpSha3_512()
+ {
+ if (!s_evpSha3_512Cached)
+ {
+ s_evpSha3_512 = CryptoNative_EvpSha3_512();
+ s_evpSha3_512Cached = true;
+ }
+
+ return s_evpSha3_512;
+ }
+
+
+ internal static IntPtr HashAlgorithmToEvp(string hashAlgorithmId)
+ {
+ switch (hashAlgorithmId)
+ {
+ case HashAlgorithmNames.SHA1: return EvpSha1();
+ case HashAlgorithmNames.SHA256: return EvpSha256();
+ case HashAlgorithmNames.SHA384: return EvpSha384();
+ case HashAlgorithmNames.SHA512: return EvpSha512();
+ case HashAlgorithmNames.SHA3_256:
+ IntPtr sha3_256 = EvpSha3_256();
+ return sha3_256 != 0 ? sha3_256 : throw new PlatformNotSupportedException();
+ case HashAlgorithmNames.SHA3_384:
+ IntPtr sha3_384 = EvpSha3_384();
+ return sha3_384 != 0 ? sha3_384 : throw new PlatformNotSupportedException();
+ case HashAlgorithmNames.SHA3_512:
+ IntPtr sha3_512 = EvpSha3_512();
+ return sha3_512 != 0 ? sha3_512 : throw new PlatformNotSupportedException();
+ case nameof(HashAlgorithmName.MD5): return EvpMd5();
+ default:
+ throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmId));
+ };
+ }
+
+ internal static bool HashAlgorithmSupported(string hashAlgorithmId)
{
- nameof(HashAlgorithmName.SHA1) => EvpSha1(),
- nameof(HashAlgorithmName.SHA256) => EvpSha256(),
- nameof(HashAlgorithmName.SHA384) => EvpSha384(),
- nameof(HashAlgorithmName.SHA512) => EvpSha512(),
- nameof(HashAlgorithmName.MD5) => EvpMd5(),
- _ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmId))
- };
+ switch (hashAlgorithmId)
+ {
+ case HashAlgorithmNames.SHA1:
+ case HashAlgorithmNames.SHA256:
+ case HashAlgorithmNames.SHA384:
+ case HashAlgorithmNames.SHA512:
+ case HashAlgorithmNames.MD5:
+ return true;
+ case HashAlgorithmNames.SHA3_256:
+ return EvpSha3_256() != 0;
+ case HashAlgorithmNames.SHA3_384:
+ return EvpSha3_384() != 0;
+ case HashAlgorithmNames.SHA3_512:
+ return EvpSha3_512() != 0;
+ default:
+ throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmId));
+ }
+ }
}
}
diff --git a/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs b/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs
index 14a6066eeb3b75..9cc01436b9138c 100644
--- a/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs
+++ b/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs
@@ -6,6 +6,7 @@
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Diagnostics;
+using System.Globalization;
using System.IO;
using System.Net;
using System.Net.Security;
@@ -84,16 +85,11 @@ private static bool DisableTlsResume
private static int GetCacheSize()
{
- int cacheSize = -1;
string? value = AppContext.GetData(TlsCacheSizeCtxName) as string ?? Environment.GetEnvironmentVariable(TlsCacheSizeEnvironmentVariable);
- try
+ if (!int.TryParse(value, CultureInfo.InvariantCulture, out int cacheSize))
{
- if (value != null)
- {
- cacheSize = int.Parse(value);
- }
+ cacheSize = -1;
}
- catch { };
return cacheSize;
}
diff --git a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventTraceGuidsEx.cs b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventTraceGuidsEx.cs
index 483cfe04b23238..f7d82befca494a 100644
--- a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventTraceGuidsEx.cs
+++ b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventTraceGuidsEx.cs
@@ -45,7 +45,7 @@ internal struct TRACE_ENABLE_INFO
public long MatchAllKeyword;
}
- [LibraryImport(Interop.Libraries.Advapi32)]
+ [LibraryImport(Libraries.Advapi32)]
internal static unsafe partial int EnumerateTraceGuidsEx(
TRACE_QUERY_INFO_CLASS TraceQueryInfoClass,
void* InBuffer,
diff --git a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventWriteTransfer.cs b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventWriteTransfer.cs
index d66fe12d87a9bd..1c1438adb1d020 100644
--- a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventWriteTransfer.cs
+++ b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.EventWriteTransfer.cs
@@ -32,7 +32,7 @@ internal static unsafe int EventWriteTransfer(
return HResult;
}
- [LibraryImport(Interop.Libraries.Advapi32, EntryPoint = "EventWriteTransfer")]
+ [LibraryImport(Libraries.Advapi32, EntryPoint = "EventWriteTransfer")]
private static unsafe partial int EventWriteTransfer_PInvoke(
long registrationHandle,
in EventDescriptor eventDescriptor,
diff --git a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.GetTokenInformation_void.cs b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.GetTokenInformation_void.cs
index e61ce35a91f97a..02f7b03c5812ef 100644
--- a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.GetTokenInformation_void.cs
+++ b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.GetTokenInformation_void.cs
@@ -9,7 +9,7 @@ internal static partial class Interop
{
internal static partial class Advapi32
{
- [LibraryImport(Interop.Libraries.Advapi32, SetLastError = true)]
+ [LibraryImport(Libraries.Advapi32, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static unsafe partial bool GetTokenInformation(
SafeTokenHandle TokenHandle,
diff --git a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.RegCreateKeyEx.cs b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.RegCreateKeyEx.cs
index 9e5e420146e191..e7cde43076f554 100644
--- a/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.RegCreateKeyEx.cs
+++ b/src/libraries/Common/src/Interop/Windows/Advapi32/Interop.RegCreateKeyEx.cs
@@ -22,7 +22,7 @@ internal static partial int RegCreateKeyEx(
string? lpClass,
int dwOptions,
int samDesired,
- ref Interop.Kernel32.SECURITY_ATTRIBUTES secAttrs,
+ ref Kernel32.SECURITY_ATTRIBUTES secAttrs,
out SafeRegistryHandle hkResult,
out int lpdwDisposition);
}
diff --git a/src/libraries/Common/src/Interop/Windows/BCrypt/BCryptAlgorithmCache.cs b/src/libraries/Common/src/Interop/Windows/BCrypt/BCryptAlgorithmCache.cs
index aec34aeb523478..5f060292a666c7 100644
--- a/src/libraries/Common/src/Interop/Windows/BCrypt/BCryptAlgorithmCache.cs
+++ b/src/libraries/Common/src/Interop/Windows/BCrypt/BCryptAlgorithmCache.cs
@@ -13,6 +13,7 @@ internal static partial class BCrypt
internal static class BCryptAlgorithmCache
{
private static readonly ConcurrentDictionary<(string HashAlgorithmId, BCryptOpenAlgorithmProviderFlags Flags), (SafeBCryptAlgorithmHandle Handle, int HashSizeInBytes)> s_handles = new();
+ private static readonly ConcurrentDictionary<(string HashAlgorithmId, BCryptOpenAlgorithmProviderFlags Flags), bool> s_supported = new();
///
/// Returns a SafeBCryptAlgorithmHandle of the desired algorithm and flags. This is a shared handle so do not dispose it!
@@ -43,6 +44,42 @@ public static unsafe SafeBCryptAlgorithmHandle GetCachedBCryptAlgorithmHandle(st
}
}
}
+
+ public static unsafe bool IsBCryptAlgorithmSupported(string hashAlgorithmId, BCryptOpenAlgorithmProviderFlags flags)
+ {
+ var key = (hashAlgorithmId, flags);
+
+ if (s_supported.TryGetValue(key, out bool supported))
+ {
+ return supported;
+ }
+
+ NTSTATUS status = BCryptOpenAlgorithmProvider(
+ out SafeBCryptAlgorithmHandle handle,
+ key.hashAlgorithmId,
+ null,
+ key.flags);
+
+ bool isSupported = status == NTSTATUS.STATUS_SUCCESS;
+
+ if (s_supported.TryAdd(key, isSupported) && isSupported)
+ {
+ // It's a valid algorithm. Let's prime the handle cache while we are here. Presumably it's
+ // going to get used if we're asking if it's supported.
+ int hashSize = BCryptGetDWordProperty(handle, BCryptPropertyStrings.BCRYPT_HASH_LENGTH);
+ Debug.Assert(hashSize > 0);
+
+ if (s_handles.TryAdd(key, (handle, hashSize)))
+ {
+ // If we added the handle to the cache, don't dispose of it and return our answer.
+ return isSupported;
+ }
+ }
+
+ // Either the algorithm isn't supported or we don't need it for priming the cache, so Dispose.
+ handle.Dispose();
+ return isSupported;
+ }
}
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/BCrypt/Interop.BCryptAlgPseudoHandle.cs b/src/libraries/Common/src/Interop/Windows/BCrypt/Interop.BCryptAlgPseudoHandle.cs
index 57c508bf709479..d0a4e878f2764a 100644
--- a/src/libraries/Common/src/Interop/Windows/BCrypt/Interop.BCryptAlgPseudoHandle.cs
+++ b/src/libraries/Common/src/Interop/Windows/BCrypt/Interop.BCryptAlgPseudoHandle.cs
@@ -23,6 +23,12 @@ public enum BCryptAlgPseudoHandle : uint
BCRYPT_HMAC_SHA384_ALG_HANDLE = 0x000000c1,
BCRYPT_HMAC_SHA512_ALG_HANDLE = 0x000000d1,
BCRYPT_PBKDF2_ALG_HANDLE = 0x00000331,
+ BCRYPT_SHA3_256_ALG_HANDLE = 0x000003B1,
+ BCRYPT_SHA3_384_ALG_HANDLE = 0x000003C1,
+ BCRYPT_SHA3_512_ALG_HANDLE = 0x000003D1,
+ BCRYPT_HMAC_SHA3_256_ALG_HANDLE = 0x000003E1,
+ BCRYPT_HMAC_SHA3_384_ALG_HANDLE = 0x000003F1,
+ BCRYPT_HMAC_SHA3_512_ALG_HANDLE = 0x00000401,
}
internal static bool PseudoHandlesSupported { get; } =
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CopyFile.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CopyFile.cs
index 001c7afc08a5ea..8fd8476677f482 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CopyFile.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CopyFile.cs
@@ -10,14 +10,14 @@ internal static partial class Kernel32
{
internal static int CopyFile(string src, string dst, bool failIfExists)
{
- int copyFlags = failIfExists ? Interop.Kernel32.FileOperations.COPY_FILE_FAIL_IF_EXISTS : 0;
+ int copyFlags = failIfExists ? FileOperations.COPY_FILE_FAIL_IF_EXISTS : 0;
int cancel = 0;
- if (!Interop.Kernel32.CopyFileEx(src, dst, IntPtr.Zero, IntPtr.Zero, ref cancel, copyFlags))
+ if (!CopyFileEx(src, dst, IntPtr.Zero, IntPtr.Zero, ref cancel, copyFlags))
{
return Marshal.GetLastPInvokeError();
}
- return Interop.Errors.ERROR_SUCCESS;
+ return Errors.ERROR_SUCCESS;
}
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.FileTimeToSystemTime.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.FileTimeToSystemTime.cs
index 0eac8ca5cff91f..c00eb8655478d8 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.FileTimeToSystemTime.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.FileTimeToSystemTime.cs
@@ -9,6 +9,6 @@ internal static partial class Kernel32
{
[LibraryImport(Libraries.Kernel32)]
[SuppressGCTransition]
- internal static unsafe partial Interop.BOOL FileTimeToSystemTime(ulong* lpFileTime, Interop.Kernel32.SYSTEMTIME* lpSystemTime);
+ internal static unsafe partial BOOL FileTimeToSystemTime(ulong* lpFileTime, SYSTEMTIME* lpSystemTime);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCPInfo.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCPInfo.cs
index 7f4558d05ac7f1..ad9aa0da87ccba 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCPInfo.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCPInfo.cs
@@ -17,6 +17,6 @@ internal unsafe struct CPINFO
}
[LibraryImport(Libraries.Kernel32)]
- internal static unsafe partial Interop.BOOL GetCPInfo(uint codePage, CPINFO* lpCpInfo);
+ internal static unsafe partial BOOL GetCPInfo(uint codePage, CPINFO* lpCpInfo);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCurrentProcessorNumberEx.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCurrentProcessorNumberEx.cs
index 4ca0d0eeb4f8f5..1224f926efc21a 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCurrentProcessorNumberEx.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetCurrentProcessorNumberEx.cs
@@ -8,7 +8,7 @@ internal static partial class Interop
{
internal static partial class Kernel32
{
- [StructLayoutAttribute(LayoutKind.Sequential)]
+ [StructLayout(LayoutKind.Sequential)]
internal struct PROCESSOR_NUMBER
{
public ushort Group;
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetSystemTime.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetSystemTime.cs
index 9b80349ca1b609..a276d59314aa02 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetSystemTime.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.GetSystemTime.cs
@@ -9,6 +9,6 @@ internal static partial class Kernel32
{
[LibraryImport(Libraries.Kernel32)]
[SuppressGCTransition]
- internal static unsafe partial void GetSystemTime(Interop.Kernel32.SYSTEMTIME* lpSystemTime);
+ internal static unsafe partial void GetSystemTime(SYSTEMTIME* lpSystemTime);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.Globalization.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.Globalization.cs
index 99e18caf50b1c4..ca34cca3a34599 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.Globalization.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.Globalization.cs
@@ -117,7 +117,7 @@ internal static unsafe partial bool IsNLSDefinedString(
int cchStr);
[LibraryImport("kernel32.dll")]
- internal static unsafe partial Interop.BOOL GetUserPreferredUILanguages(uint dwFlags, uint* pulNumLanguages, char* pwszLanguagesBuffer, uint* pcchLanguagesBuffer);
+ internal static unsafe partial BOOL GetUserPreferredUILanguages(uint dwFlags, uint* pulNumLanguages, char* pwszLanguagesBuffer, uint* pcchLanguagesBuffer);
[LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16)]
internal static unsafe partial int GetLocaleInfoEx(string lpLocaleName, uint LCType, void* lpLCData, int cchData);
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.OutputDebugString.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.OutputDebugString.cs
index 0012b5de2899d7..684a575ecb9085 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.OutputDebugString.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.OutputDebugString.cs
@@ -7,7 +7,7 @@ internal static partial class Interop
{
internal static partial class Kernel32
{
- [LibraryImport(Interop.Libraries.Kernel32, EntryPoint = "OutputDebugStringW", StringMarshalling = StringMarshalling.Utf16)]
+ [LibraryImport(Libraries.Kernel32, EntryPoint = "OutputDebugStringW", StringMarshalling = StringMarshalling.Utf16)]
internal static partial void OutputDebugString(string message);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.SystemTimeToFileTime.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.SystemTimeToFileTime.cs
index 9b267a480126c2..2cb594e3cdf9ad 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.SystemTimeToFileTime.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.SystemTimeToFileTime.cs
@@ -9,6 +9,6 @@ internal static partial class Kernel32
{
[LibraryImport(Libraries.Kernel32)]
[SuppressGCTransition]
- internal static unsafe partial Interop.BOOL SystemTimeToFileTime(Interop.Kernel32.SYSTEMTIME* lpSystemTime, ulong* lpFileTime);
+ internal static unsafe partial BOOL SystemTimeToFileTime(SYSTEMTIME* lpSystemTime, ulong* lpFileTime);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.TzSpecificLocalTimeToSystemTime.cs b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.TzSpecificLocalTimeToSystemTime.cs
index de38b9e0a96c42..b38ffdb2efe50c 100644
--- a/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.TzSpecificLocalTimeToSystemTime.cs
+++ b/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.TzSpecificLocalTimeToSystemTime.cs
@@ -9,9 +9,9 @@ internal static partial class Interop
internal static partial class Kernel32
{
[LibraryImport(Libraries.Kernel32)]
- internal static unsafe partial Interop.BOOL TzSpecificLocalTimeToSystemTime(
+ internal static unsafe partial BOOL TzSpecificLocalTimeToSystemTime(
IntPtr lpTimeZoneInformation,
- Interop.Kernel32.SYSTEMTIME* lpLocalTime,
- Interop.Kernel32.SYSTEMTIME* lpUniversalTime);
+ SYSTEMTIME* lpLocalTime,
+ SYSTEMTIME* lpUniversalTime);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/NtDll/Interop.NtCreateFile.cs b/src/libraries/Common/src/Interop/Windows/NtDll/Interop.NtCreateFile.cs
index cc80f325702c3c..d1a07c14221ef9 100644
--- a/src/libraries/Common/src/Interop/Windows/NtDll/Interop.NtCreateFile.cs
+++ b/src/libraries/Common/src/Interop/Windows/NtDll/Interop.NtCreateFile.cs
@@ -203,7 +203,7 @@ public enum CreateDisposition : uint
///
/// Open if exists or fail if doesn't exist. Equivalent to OPEN_EXISTING or
- /// .
+ /// .
///
///
/// TruncateExisting also uses Open and then manually truncates the file
@@ -214,25 +214,25 @@ public enum CreateDisposition : uint
///
/// Create if doesn't exist or fail if does exist. Equivalent to CREATE_NEW
- /// or .
+ /// or .
///
FILE_CREATE = 2,
///
/// Open if exists or create if doesn't exist. Equivalent to OPEN_ALWAYS or
- /// .
+ /// .
///
FILE_OPEN_IF = 3,
///
/// Open and overwrite if exists or fail if doesn't exist. Equivalent to
- /// TRUNCATE_EXISTING or .
+ /// TRUNCATE_EXISTING or .
///
FILE_OVERWRITE = 4,
///
/// Open and overwrite if exists or create if doesn't exist. Equivalent to
- /// CREATE_ALWAYS or .
+ /// CREATE_ALWAYS or .
///
FILE_OVERWRITE_IF = 5
}
diff --git a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CLSIDFromProgID.cs b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CLSIDFromProgID.cs
index 5a9b58d6bd72de..cf2108ec6ab020 100644
--- a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CLSIDFromProgID.cs
+++ b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CLSIDFromProgID.cs
@@ -8,7 +8,7 @@ internal static partial class Interop
{
internal static partial class Ole32
{
- [LibraryImport(Interop.Libraries.Ole32, StringMarshalling = StringMarshalling.Utf16)]
+ [LibraryImport(Libraries.Ole32, StringMarshalling = StringMarshalling.Utf16)]
internal static partial int CLSIDFromProgID(string lpszProgID, out Guid lpclsid);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoCreateGuid.cs b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoCreateGuid.cs
index 92b36a053dd533..ad8e77784cf8b2 100644
--- a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoCreateGuid.cs
+++ b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoCreateGuid.cs
@@ -8,7 +8,7 @@ internal static partial class Interop
{
internal static partial class Ole32
{
- [LibraryImport(Interop.Libraries.Ole32)]
+ [LibraryImport(Libraries.Ole32)]
internal static partial int CoCreateGuid(out Guid guid);
}
}
diff --git a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoGetStandardMarshal.cs b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoGetStandardMarshal.cs
index 11d26766d5063c..299c598e061671 100644
--- a/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoGetStandardMarshal.cs
+++ b/src/libraries/Common/src/Interop/Windows/Ole32/Interop.CoGetStandardMarshal.cs
@@ -8,7 +8,7 @@ internal static partial class Interop
{
internal static partial class Ole32
{
- [LibraryImport(Interop.Libraries.Ole32)]
+ [LibraryImport(Libraries.Ole32)]
internal static partial int CoGetStandardMarshal(ref Guid riid, IntPtr pv, int dwDestContext, IntPtr pvDestContext, int mshlflags, out IntPtr ppMarshal);
}
}
diff --git a/src/libraries/Common/src/System/Collections/Concurrent/MultiProducerMultiConsumerQueue.cs b/src/libraries/Common/src/System/Collections/Concurrent/MultiProducerMultiConsumerQueue.cs
index e9da29081a3180..4fac49a6dfb25f 100644
--- a/src/libraries/Common/src/System/Collections/Concurrent/MultiProducerMultiConsumerQueue.cs
+++ b/src/libraries/Common/src/System/Collections/Concurrent/MultiProducerMultiConsumerQueue.cs
@@ -15,21 +15,21 @@ internal sealed class MultiProducerMultiConsumerQueue : ConcurrentQueue, I
{
/// Enqueues an item into the queue.
/// The item to enqueue.
- void IProducerConsumerQueue.Enqueue(T item) { base.Enqueue(item); }
+ void IProducerConsumerQueue.Enqueue(T item) { Enqueue(item); }
/// Attempts to dequeue an item from the queue.
/// The dequeued item.
/// true if an item could be dequeued; otherwise, false.
- bool IProducerConsumerQueue.TryDequeue([MaybeNullWhen(false)] out T result) { return base.TryDequeue(out result); }
+ bool IProducerConsumerQueue.TryDequeue([MaybeNullWhen(false)] out T result) { return TryDequeue(out result); }
/// Gets whether the collection is currently empty.
- bool IProducerConsumerQueue.IsEmpty => base.IsEmpty;
+ bool IProducerConsumerQueue.IsEmpty => IsEmpty;
/// Gets the number of items in the collection.
- int IProducerConsumerQueue.Count => base.Count;
+ int IProducerConsumerQueue.Count => Count;
/// A thread-safe way to get the number of items in the collection. May synchronize access by locking the provided synchronization object.
/// ConcurrentQueue.Count is thread safe, no need to acquire the lock.
- int IProducerConsumerQueue.GetCountSafe(object syncObj) => base.Count;
+ int IProducerConsumerQueue.GetCountSafe(object syncObj) => Count;
}
}
diff --git a/src/libraries/Common/src/System/Diagnostics/DiagnosticsHelper.cs b/src/libraries/Common/src/System/Diagnostics/DiagnosticsHelper.cs
index 18f10c0d520d87..ce7f345b0ea9cd 100644
--- a/src/libraries/Common/src/System/Diagnostics/DiagnosticsHelper.cs
+++ b/src/libraries/Common/src/System/Diagnostics/DiagnosticsHelper.cs
@@ -32,7 +32,7 @@ internal static bool CompareTags(List>? sortedTags
}
int count = sortedTags.Count;
- int size = count / (sizeof(long) * 8) + 1;
+ int size = count / (sizeof(ulong) * 8) + 1;
BitMapper bitMapper = new BitMapper(size <= 100 ? stackalloc ulong[size] : new ulong[size]);
if (tags2 is ICollection> tagsCol)
@@ -124,15 +124,15 @@ public BitMapper(Span bitMap)
{
_bitMap = bitMap;
_bitMap.Clear();
- _maxIndex = bitMap.Length * sizeof(long) * 8;
+ _maxIndex = bitMap.Length * sizeof(ulong) * 8;
}
public int MaxIndex => _maxIndex;
private static void GetIndexAndMask(int index, out int bitIndex, out ulong mask)
{
- bitIndex = index >> 6;
- int bit = index & (sizeof(long) * 8 - 1);
+ bitIndex = index >> 6; // divide by 64 == (sizeof(ulong) * 8) bits
+ int bit = index & (sizeof(ulong) * 8 - 1);
mask = 1UL << bit;
}
diff --git a/src/libraries/Common/src/System/IO/PathInternal.cs b/src/libraries/Common/src/System/IO/PathInternal.cs
index c6ea2e2261805e..a07d5cafffde2a 100644
--- a/src/libraries/Common/src/System/IO/PathInternal.cs
+++ b/src/libraries/Common/src/System/IO/PathInternal.cs
@@ -128,7 +128,7 @@ internal static bool RemoveRelativeSegments(ReadOnlySpan path, int rootLen
// We treat "\.." , "\." and "\\" as a relative segment. We want to collapse the first separator past the root presuming
// the root actually ends in a separator. Otherwise the first segment for RemoveRelativeSegments
// in cases like "\\?\C:\.\" and "\\?\C:\..\", the first segment after the root will be ".\" and "..\" which is not considered as a relative segment and hence not be removed.
- if (PathInternal.IsDirectorySeparator(path[skip - 1]))
+ if (IsDirectorySeparator(path[skip - 1]))
skip--;
// Remove "//", "/./", and "/../" from the path by copying each character to the output,
@@ -143,18 +143,18 @@ internal static bool RemoveRelativeSegments(ReadOnlySpan path, int rootLen
{
char c = path[i];
- if (PathInternal.IsDirectorySeparator(c) && i + 1 < path.Length)
+ if (IsDirectorySeparator(c) && i + 1 < path.Length)
{
// Skip this character if it's a directory separator and if the next character is, too,
// e.g. "parent//child" => "parent/child"
- if (PathInternal.IsDirectorySeparator(path[i + 1]))
+ if (IsDirectorySeparator(path[i + 1]))
{
continue;
}
// Skip this character and the next if it's referring to the current directory,
// e.g. "parent/./child" => "parent/child"
- if ((i + 2 == path.Length || PathInternal.IsDirectorySeparator(path[i + 2])) &&
+ if ((i + 2 == path.Length || IsDirectorySeparator(path[i + 2])) &&
path[i + 1] == '.')
{
i++;
@@ -164,14 +164,14 @@ internal static bool RemoveRelativeSegments(ReadOnlySpan path, int rootLen
// Skip this character and the next two if it's referring to the parent directory,
// e.g. "parent/child/../grandchild" => "parent/grandchild"
if (i + 2 < path.Length &&
- (i + 3 == path.Length || PathInternal.IsDirectorySeparator(path[i + 3])) &&
+ (i + 3 == path.Length || IsDirectorySeparator(path[i + 3])) &&
path[i + 1] == '.' && path[i + 2] == '.')
{
// Unwind back to the last slash (and if there isn't one, clear out everything).
int s;
for (s = sb.Length - 1; s >= skip; s--)
{
- if (PathInternal.IsDirectorySeparator(sb[s]))
+ if (IsDirectorySeparator(sb[s]))
{
sb.Length = (i + 3 >= path.Length && s == skip) ? s + 1 : s; // to avoid removing the complete "\tmp\" segment in cases like \\?\C:\tmp\..\, C:\tmp\..
break;
@@ -188,9 +188,9 @@ internal static bool RemoveRelativeSegments(ReadOnlySpan path, int rootLen
}
// Normalize the directory separator if needed
- if (c != PathInternal.DirectorySeparatorChar && c == PathInternal.AltDirectorySeparatorChar)
+ if (c != DirectorySeparatorChar && c == AltDirectorySeparatorChar)
{
- c = PathInternal.DirectorySeparatorChar;
+ c = DirectorySeparatorChar;
flippedSeparator = true;
}
diff --git a/src/libraries/Common/src/System/Net/Http/X509ResourceClient.cs b/src/libraries/Common/src/System/Net/Http/X509ResourceClient.cs
index 5fe2b8ce402382..57538487800e1e 100644
--- a/src/libraries/Common/src/System/Net/Http/X509ResourceClient.cs
+++ b/src/libraries/Common/src/System/Net/Http/X509ResourceClient.cs
@@ -12,7 +12,7 @@ namespace System.Net.Http
{
internal static partial class X509ResourceClient
{
- private static readonly Func>? s_downloadBytes = CreateDownloadBytesFunc();
+ private static readonly Func>? s_downloadBytes = CreateDownloadBytesFunc();
static partial void ReportNoClient();
static partial void ReportNegativeTimeout();
@@ -24,18 +24,17 @@ internal static partial class X509ResourceClient
internal static byte[]? DownloadAsset(string uri, TimeSpan downloadTimeout)
{
- ValueTask task = DownloadAssetCore(uri, downloadTimeout, async: false);
+ Task task = DownloadAssetCore(uri, downloadTimeout, async: false);
Debug.Assert(task.IsCompletedSuccessfully);
return task.Result;
}
internal static Task DownloadAssetAsync(string uri, TimeSpan downloadTimeout)
{
- ValueTask task = DownloadAssetCore(uri, downloadTimeout, async: true);
- return task.AsTask();
+ return DownloadAssetCore(uri, downloadTimeout, async: true);
}
- private static async ValueTask DownloadAssetCore(string uri, TimeSpan downloadTimeout, bool async)
+ private static async Task DownloadAssetCore(string uri, TimeSpan downloadTimeout, bool async)
{
if (s_downloadBytes is null)
{
@@ -60,8 +59,12 @@ internal static partial class X509ResourceClient
try
{
- ret = await s_downloadBytes(uri, cts?.Token ?? default, async).ConfigureAwait(false);
- return ret;
+ Task task = s_downloadBytes(uri, cts?.Token ?? default, async);
+ await ((Task)task).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
+ if (task.IsCompletedSuccessfully)
+ {
+ return task.Result;
+ }
}
catch { }
finally
@@ -74,7 +77,7 @@ internal static partial class X509ResourceClient
return null;
}
- private static Func>? CreateDownloadBytesFunc()
+ private static Func>? CreateDownloadBytesFunc()
{
try
{
diff --git a/src/libraries/Common/src/System/Security/Cryptography/HashOneShotHelpers.cs b/src/libraries/Common/src/System/Security/Cryptography/HashOneShotHelpers.cs
index debc16c329b1e5..f80e69c8bf53a8 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/HashOneShotHelpers.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/HashOneShotHelpers.cs
@@ -12,28 +12,18 @@ internal static class HashOneShotHelpers
{
internal static byte[] HashData(HashAlgorithmName hashAlgorithm, ReadOnlySpan source)
{
- if (hashAlgorithm == HashAlgorithmName.SHA256)
- {
- return SHA256.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA1)
- {
- return SHA1.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA512)
- {
- return SHA512.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA384)
- {
- return SHA384.HashData(source);
- }
- else if (Helpers.HasMD5 && hashAlgorithm == HashAlgorithmName.MD5)
- {
- return MD5.HashData(source);
- }
-
- throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name));
+ return hashAlgorithm.Name switch
+ {
+ HashAlgorithmNames.SHA256 => SHA256.HashData(source),
+ HashAlgorithmNames.SHA1 => SHA1.HashData(source),
+ HashAlgorithmNames.SHA512 => SHA512.HashData(source),
+ HashAlgorithmNames.SHA384 => SHA384.HashData(source),
+ HashAlgorithmNames.SHA3_256 => SHA3_256.HashData(source),
+ HashAlgorithmNames.SHA3_384 => SHA3_384.HashData(source),
+ HashAlgorithmNames.SHA3_512 => SHA3_512.HashData(source),
+ HashAlgorithmNames.MD5 when Helpers.HasMD5 => MD5.HashData(source),
+ _ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name)),
+ };
}
internal static bool TryHashData(
@@ -42,54 +32,34 @@ internal static bool TryHashData(
Span destination,
out int bytesWritten)
{
- if (hashAlgorithm == HashAlgorithmName.SHA256)
- {
- return SHA256.TryHashData(source, destination, out bytesWritten);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA1)
- {
- return SHA1.TryHashData(source, destination, out bytesWritten);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA512)
- {
- return SHA512.TryHashData(source, destination, out bytesWritten);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA384)
- {
- return SHA384.TryHashData(source, destination, out bytesWritten);
- }
- else if (Helpers.HasMD5 && hashAlgorithm == HashAlgorithmName.MD5)
- {
- return MD5.TryHashData(source, destination, out bytesWritten);
- }
-
- throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name));
+ return hashAlgorithm.Name switch
+ {
+ HashAlgorithmNames.SHA256 => SHA256.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA1 => SHA1.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA512 => SHA512.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA384 => SHA384.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA3_256 => SHA3_256.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA3_384 => SHA3_384.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.SHA3_512 => SHA3_512.TryHashData(source, destination, out bytesWritten),
+ HashAlgorithmNames.MD5 when Helpers.HasMD5 => MD5.TryHashData(source, destination, out bytesWritten),
+ _ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name)),
+ };
}
internal static byte[] HashData(HashAlgorithmName hashAlgorithm, Stream source)
{
- if (hashAlgorithm == HashAlgorithmName.SHA256)
- {
- return SHA256.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA1)
- {
- return SHA1.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA512)
- {
- return SHA512.HashData(source);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA384)
- {
- return SHA384.HashData(source);
- }
- else if (Helpers.HasMD5 && hashAlgorithm == HashAlgorithmName.MD5)
- {
- return MD5.HashData(source);
- }
-
- throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name));
+ return hashAlgorithm.Name switch
+ {
+ HashAlgorithmNames.SHA256 => SHA256.HashData(source),
+ HashAlgorithmNames.SHA1 => SHA1.HashData(source),
+ HashAlgorithmNames.SHA512 => SHA512.HashData(source),
+ HashAlgorithmNames.SHA384 => SHA384.HashData(source),
+ HashAlgorithmNames.SHA3_256 => SHA3_256.HashData(source),
+ HashAlgorithmNames.SHA3_384 => SHA3_384.HashData(source),
+ HashAlgorithmNames.SHA3_512 => SHA3_512.HashData(source),
+ HashAlgorithmNames.MD5 when Helpers.HasMD5 => MD5.HashData(source),
+ _ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name)),
+ };
}
internal static int MacData(
@@ -98,28 +68,18 @@ internal static int MacData(
ReadOnlySpan source,
Span destination)
{
- if (hashAlgorithm == HashAlgorithmName.SHA256)
- {
- return HMACSHA256.HashData(key, source, destination);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA1)
- {
- return HMACSHA1.HashData(key, source, destination);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA512)
- {
- return HMACSHA512.HashData(key, source, destination);
- }
- else if (hashAlgorithm == HashAlgorithmName.SHA384)
- {
- return HMACSHA384.HashData(key, source, destination);
- }
- else if (Helpers.HasMD5 && hashAlgorithm == HashAlgorithmName.MD5)
- {
- return HMACMD5.HashData(key, source, destination);
- }
-
- throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name));
+ return hashAlgorithm.Name switch
+ {
+ HashAlgorithmNames.SHA256 => HMACSHA256.HashData(key, source, destination),
+ HashAlgorithmNames.SHA1 => HMACSHA1.HashData(key, source, destination),
+ HashAlgorithmNames.SHA512 => HMACSHA512.HashData(key, source, destination),
+ HashAlgorithmNames.SHA384 => HMACSHA384.HashData(key, source, destination),
+ HashAlgorithmNames.SHA3_256 => HMACSHA3_256.HashData(key, source, destination),
+ HashAlgorithmNames.SHA3_384 => HMACSHA3_384.HashData(key, source, destination),
+ HashAlgorithmNames.SHA3_512 => HMACSHA3_512.HashData(key, source, destination),
+ HashAlgorithmNames.MD5 when Helpers.HasMD5 => HMACMD5.HashData(key, source, destination),
+ _ => throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithm.Name)),
+ };
}
}
}
diff --git a/src/libraries/Common/src/System/Security/Cryptography/Oids.cs b/src/libraries/Common/src/System/Security/Cryptography/Oids.cs
index 40e761dab09d4c..b7364108222ad1 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/Oids.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/Oids.cs
@@ -24,6 +24,9 @@ internal static partial class Oids
internal const string RsaPkcs1Sha256 = "1.2.840.113549.1.1.11";
internal const string RsaPkcs1Sha384 = "1.2.840.113549.1.1.12";
internal const string RsaPkcs1Sha512 = "1.2.840.113549.1.1.13";
+ internal const string RsaPkcs1Sha3_256 = "2.16.840.1.101.3.4.3.14";
+ internal const string RsaPkcs1Sha3_384 = "2.16.840.1.101.3.4.3.15";
+ internal const string RsaPkcs1Sha3_512 = "2.16.840.1.101.3.4.3.16";
internal const string Esdh = "1.2.840.113549.1.9.16.3.5";
internal const string EcDiffieHellman = "1.3.132.1.12";
internal const string DiffieHellman = "1.2.840.10046.2.1";
@@ -65,6 +68,9 @@ internal static partial class Oids
internal const string Sha256 = "2.16.840.1.101.3.4.2.1";
internal const string Sha384 = "2.16.840.1.101.3.4.2.2";
internal const string Sha512 = "2.16.840.1.101.3.4.2.3";
+ internal const string Sha3_256 = "2.16.840.1.101.3.4.2.8";
+ internal const string Sha3_384 = "2.16.840.1.101.3.4.2.9";
+ internal const string Sha3_512 = "2.16.840.1.101.3.4.2.10";
// DSA CMS uses the combined signature+digest OID
internal const string DsaWithSha1 = "1.2.840.10040.4.3";
@@ -84,6 +90,10 @@ internal static partial class Oids
internal const string ECDsaWithSha384 = "1.2.840.10045.4.3.3";
internal const string ECDsaWithSha512 = "1.2.840.10045.4.3.4";
+ internal const string ECDsaWithSha3_256 = "2.16.840.1.101.3.4.3.10";
+ internal const string ECDsaWithSha3_384 = "2.16.840.1.101.3.4.3.11";
+ internal const string ECDsaWithSha3_512 = "2.16.840.1.101.3.4.3.12";
+
internal const string Mgf1 = "1.2.840.113549.1.1.8";
internal const string PSpecified = "1.2.840.113549.1.1.9";
diff --git a/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs b/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs
index dc002fdfd59732..34ad612323f2d7 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs
@@ -873,6 +873,18 @@ private static void ValidatePadding(RSAEncryptionPadding padding)
{
throw PaddingModeNotSupported();
}
+
+ // If the hash algorithm is not supported by the platform, such as SHA3, then we don't support it for
+ // RSAOpenSsl, even if OpenSSL itself might support OAEP-SHA3. We use the platform's hashing in some
+ // places for RSA, regardless of what is implementing RSA. If RSAOpenSsl were used on macOS, then
+ // there would be some incongruence between what hashes OpenSSL supports and what macOS support. Signing
+ // for example, always uses the platform's implementation of hashing.
+ if (padding.Mode == RSAEncryptionPaddingMode.Oaep &&
+ padding.OaepHashAlgorithm.Name is string name &&
+ !HashProviderDispenser.HashSupported(name))
+ {
+ throw new PlatformNotSupportedException();
+ }
}
private static void ValidatePadding(RSASignaturePadding padding)
diff --git a/src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs b/src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs
index 4db95b2ff4d242..66256440b7e708 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/RsaPaddingProcessor.cs
@@ -44,41 +44,62 @@ internal static class RsaPaddingProcessor
0x40,
};
+ private static ReadOnlySpan DigestInfoSha3_256 => new byte[]
+ {
+ 0x30, 0x31, 0x30, 0x0D, 0x06, 0x09, 0x60, 0x86, 0x48,
+ 0x01, 0x65, 0x03, 0x04, 0x02, 0x08, 0x05, 0x00, 0x04,
+ 0x20,
+ };
+
+ private static ReadOnlySpan DigestInfoSha3_384 => new byte[]
+ {
+ 0x30, 0x41, 0x30, 0x0D, 0x06, 0x09, 0x60, 0x86, 0x48,
+ 0x01, 0x65, 0x03, 0x04, 0x02, 0x09, 0x05, 0x00, 0x04,
+ 0x30,
+ };
+
+ private static ReadOnlySpan DigestInfoSha3_512 => new byte[]
+ {
+ 0x30, 0x51, 0x30, 0x0D, 0x06, 0x09, 0x60, 0x86, 0x48,
+ 0x01, 0x65, 0x03, 0x04, 0x02, 0x0A, 0x05, 0x00, 0x04,
+ 0x40,
+ };
+
private static ReadOnlySpan EightZeros => new byte[8];
private static ReadOnlySpan GetDigestInfoForAlgorithm(
HashAlgorithmName hashAlgorithmName,
out int digestLengthInBytes)
{
- if (hashAlgorithmName == HashAlgorithmName.MD5)
- {
- digestLengthInBytes = MD5.HashSizeInBytes;
- return DigestInfoMD5;
- }
- else if (hashAlgorithmName == HashAlgorithmName.SHA1)
- {
- digestLengthInBytes = SHA1.HashSizeInBytes;
- return DigestInfoSha1;
- }
- else if (hashAlgorithmName == HashAlgorithmName.SHA256)
- {
- digestLengthInBytes = SHA256.HashSizeInBytes;
- return DigestInfoSha256;
- }
- else if (hashAlgorithmName == HashAlgorithmName.SHA384)
+ switch (hashAlgorithmName.Name)
{
- digestLengthInBytes = SHA384.HashSizeInBytes;
- return DigestInfoSha384;
- }
- else if (hashAlgorithmName == HashAlgorithmName.SHA512)
- {
- digestLengthInBytes = SHA512.HashSizeInBytes;
- return DigestInfoSha512;
- }
- else
- {
- Debug.Fail("Unknown digest algorithm");
- throw new CryptographicException();
+ case HashAlgorithmNames.MD5:
+ digestLengthInBytes = MD5.HashSizeInBytes;
+ return DigestInfoMD5;
+ case HashAlgorithmNames.SHA1:
+ digestLengthInBytes = SHA1.HashSizeInBytes;
+ return DigestInfoSha1;
+ case HashAlgorithmNames.SHA256:
+ digestLengthInBytes = SHA256.HashSizeInBytes;
+ return DigestInfoSha256;
+ case HashAlgorithmNames.SHA384:
+ digestLengthInBytes = SHA384.HashSizeInBytes;
+ return DigestInfoSha384;
+ case HashAlgorithmNames.SHA512:
+ digestLengthInBytes = SHA512.HashSizeInBytes;
+ return DigestInfoSha512;
+ case HashAlgorithmNames.SHA3_256:
+ digestLengthInBytes = SHA3_256.HashSizeInBytes;
+ return DigestInfoSha3_256;
+ case HashAlgorithmNames.SHA3_384:
+ digestLengthInBytes = SHA3_384.HashSizeInBytes;
+ return DigestInfoSha3_384;
+ case HashAlgorithmNames.SHA3_512:
+ digestLengthInBytes = SHA3_512.HashSizeInBytes;
+ return DigestInfoSha3_512;
+ default:
+ Debug.Fail("Unknown digest algorithm");
+ throw new CryptographicException();
}
}
diff --git a/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdf.cs b/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdf.cs
index 6598136dfe7a77..a9f37b0fb11a4e 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdf.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdf.cs
@@ -559,6 +559,26 @@ private static void CheckHashAlgorithm(HashAlgorithmName hashAlgorithm)
case HashAlgorithmNames.SHA384:
case HashAlgorithmNames.SHA512:
break;
+#if NET8_0_OR_GREATER
+ case HashAlgorithmNames.SHA3_256:
+ if (!HMACSHA3_256.IsSupported)
+ {
+ throw new PlatformNotSupportedException();
+ }
+ break;
+ case HashAlgorithmNames.SHA3_384:
+ if (!HMACSHA3_384.IsSupported)
+ {
+ throw new PlatformNotSupportedException();
+ }
+ break;
+ case HashAlgorithmNames.SHA3_512:
+ if (!HMACSHA3_512.IsSupported)
+ {
+ throw new PlatformNotSupportedException();
+ }
+ break;
+#endif
default:
throw new CryptographicException(SR.Format(SR.Cryptography_UnknownHashAlgorithm, hashAlgorithmName));
}
diff --git a/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdfImplementationCng.cs b/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdfImplementationCng.cs
index ca8f72c61428aa..46143f15c86acc 100644
--- a/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdfImplementationCng.cs
+++ b/src/libraries/Common/src/System/Security/Cryptography/SP800108HmacCounterKdfImplementationCng.cs
@@ -193,7 +193,7 @@ private static unsafe SafeBCryptKeyHandle CreateSymmetricKey(byte* symmetricKey,
private static int GetHashBlockSize(string hashAlgorithmName)
{
- // Block sizes per NIST FIPS pub 180-4.
+ // Block sizes per NIST FIPS pub 180-4 and FIPS 202.
switch (hashAlgorithmName)
{
case HashAlgorithmNames.SHA1:
@@ -202,6 +202,12 @@ private static int GetHashBlockSize(string hashAlgorithmName)
case HashAlgorithmNames.SHA384:
case HashAlgorithmNames.SHA512:
return 1024 / 8;
+ case HashAlgorithmNames.SHA3_256:
+ return 1088 / 8;
+ case HashAlgorithmNames.SHA3_384:
+ return 832 / 8;
+ case HashAlgorithmNames.SHA3_512:
+ return 576 / 8;
default:
Debug.Fail($"Unexpected hash algorithm '{hashAlgorithmName}'");
throw new CryptographicException();
diff --git a/src/libraries/Common/src/System/Threading/AsyncOverSyncWithIoCancellation.cs b/src/libraries/Common/src/System/Threading/AsyncOverSyncWithIoCancellation.cs
index ac13cb07903f06..9285def9f15d83 100644
--- a/src/libraries/Common/src/System/Threading/AsyncOverSyncWithIoCancellation.cs
+++ b/src/libraries/Common/src/System/Threading/AsyncOverSyncWithIoCancellation.cs
@@ -14,7 +14,7 @@ namespace System.Threading
///
/// Helper for performing asynchronous I/O on Windows implemented as queueing a work item that performs synchronous I/O, complete with cancellation support.
///
- internal sealed class AsyncOverSyncWithIoCancellation : IThreadPoolWorkItem, ICriticalNotifyCompletion
+ internal sealed class AsyncOverSyncWithIoCancellation
{
/// A thread handle for the current OS thread.
/// This is lazily-initialized for the current OS thread. We rely on finalization to clean up after it when the thread goes away.
@@ -32,23 +32,9 @@ internal sealed class AsyncOverSyncWithIoCancellation : IThreadPoolWorkItem, ICr
/// the callback wasn't and will never be invoked. If it's non-null, its completion represents the completion of the asynchronous callback.
///
private volatile Task? CallbackCompleted;
- /// The continuation object handed to this instance when used as an awaiter to scheduler work to the thread pool.
- private Action? _continuation;
-
- // awaitable / awaiter implementation that enables this instance to be awaited in order to queue
- // execution to the thread pool. This is purely a cost-saving measure in order to reuse this
- // object we already need as the queued work item.
- public AsyncOverSyncWithIoCancellation GetAwaiter() => this;
- public bool IsCompleted => false;
- public void GetResult() { }
- public void OnCompleted(Action continuation) => throw new NotSupportedException();
- public void UnsafeOnCompleted(Action continuation)
- {
- Debug.Assert(_continuation is null);
- _continuation = continuation;
- ThreadPool.UnsafeQueueUserWorkItem(this, preferLocal: true);
- }
- void IThreadPoolWorkItem.Execute() => _continuation!();
+
+ /// Prevent external instantiation.
+ private AsyncOverSyncWithIoCancellation() { }
/// Queues the invocation of to the thread pool.
/// The type of the state passed to .
@@ -65,27 +51,24 @@ public void UnsafeOnCompleted(Action continuation)
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder))]
public static async ValueTask InvokeAsync(Action action, TState state, CancellationToken cancellationToken)
{
- // Create the work item state object. This is used to pass around state through various APIs,
- // while also serving double duty as the work item used to queue the operation to the thread pool.
- var workItem = new AsyncOverSyncWithIoCancellation();
-
- // Queue the work to the thread pool. This is implemented as a custom awaiter that queues the
- // awaiter itself to the thread pool.
- await workItem;
+ // Queue the work to complete asynchronously.
+ await Task.CompletedTask.ConfigureAwait(ConfigureAwaitOptions.ForceYielding);
// Register for cancellation, perform the work, and clean up. Even though we're in an async method, awaits _must not_ be used inside
// the using block, or else the I/O cancellation could both not work and negatively interact with I/O on another thread. The func
// _must_ be invoked on the same thread that invoked RegisterCancellation, with no intervening work.
- await using (workItem.RegisterCancellation(cancellationToken).ConfigureAwait(false))
+ SyncAsyncWorkItemRegistration reg = RegisterCancellation(cancellationToken);
+ try
{
- try
- {
- action(state);
- }
- catch (OperationCanceledException oce) when (cancellationToken.IsCancellationRequested && oce.CancellationToken != cancellationToken)
- {
- throw CreateAppropriateCancellationException(cancellationToken, oce);
- }
+ action(state);
+ }
+ catch (OperationCanceledException oce) when (cancellationToken.IsCancellationRequested && oce.CancellationToken != cancellationToken)
+ {
+ throw CreateAppropriateCancellationException(cancellationToken, oce);
+ }
+ finally
+ {
+ await reg.DisposeAsync().ConfigureAwait(false);
}
}
@@ -105,27 +88,24 @@ public static async ValueTask InvokeAsync(Action action, TState
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder<>))]
public static async ValueTask InvokeAsync(Func func, TState state, CancellationToken cancellationToken)
{
- // Create the work item state object. This is used to pass around state through various APIs,
- // while also serving double duty as the work item used to queue the operation to the thread pool.
- var workItem = new AsyncOverSyncWithIoCancellation();
-
- // Queue the work to the thread pool. This is implemented as a custom awaiter that queues the
- // awaiter itself to the thread pool.
- await workItem;
+ // Queue the work to complete asynchronously.
+ await Task.CompletedTask.ConfigureAwait(ConfigureAwaitOptions.ForceYielding);
// Register for cancellation, perform the work, and clean up. Even though we're in an async method, awaits _must not_ be used inside
// the using block, or else the I/O cancellation could both not work and negatively interact with I/O on another thread. The func
// _must_ be invoked on the same thread that invoked RegisterCancellation, with no intervening work.
- await using (workItem.RegisterCancellation(cancellationToken).ConfigureAwait(false))
+ SyncAsyncWorkItemRegistration reg = RegisterCancellation(cancellationToken);
+ try
{
- try
- {
- return func(state);
- }
- catch (OperationCanceledException oce) when (cancellationToken.IsCancellationRequested && oce.CancellationToken != cancellationToken)
- {
- throw CreateAppropriateCancellationException(cancellationToken, oce);
- }
+ return func(state);
+ }
+ catch (OperationCanceledException oce) when (cancellationToken.IsCancellationRequested && oce.CancellationToken != cancellationToken)
+ {
+ throw CreateAppropriateCancellationException(cancellationToken, oce);
+ }
+ finally
+ {
+ await reg.DisposeAsync().ConfigureAwait(false);
}
}
@@ -145,7 +125,7 @@ private static OperationCanceledException CreateAppropriateCancellationException
return newOce;
}
- /// The struct IDisposable returned from in order to clean up after the registration.
+ /// The struct IDisposable returned from RegisterCancellation in order to clean up after the registration.
private struct SyncAsyncWorkItemRegistration : IDisposable, IAsyncDisposable
{
public AsyncOverSyncWithIoCancellation WorkItem;
@@ -201,44 +181,44 @@ public async ValueTask DisposeAsync()
/// Registers for cancellation with the specified token.
/// Upon cancellation being requested, the implementation will attempt to CancelSynchronousIo for the thread calling RegisterCancellation.
- private SyncAsyncWorkItemRegistration RegisterCancellation(CancellationToken cancellationToken)
- {
- // If the token can't be canceled, there's nothing to register.
- if (!cancellationToken.CanBeCanceled)
- {
- return default;
- }
+ private static SyncAsyncWorkItemRegistration RegisterCancellation(CancellationToken cancellationToken) =>
+ cancellationToken.CanBeCanceled ? RegisterCancellation(new AsyncOverSyncWithIoCancellation(), cancellationToken) :
+ default; // If the token can't be canceled, there's nothing to register.
+ /// Registers for cancellation with the specified token.
+ /// Upon cancellation being requested, the implementation will attempt to CancelSynchronousIo for the thread calling RegisterCancellation.
+ private static SyncAsyncWorkItemRegistration RegisterCancellation(AsyncOverSyncWithIoCancellation instance, CancellationToken cancellationToken)
+ {
// Get a handle for the current thread. This is stored and used to cancel the I/O on this thread
// in response to the cancellation token having cancellation requested. If the handle is invalid,
// which could happen if OpenThread fails, skip attempts at cancellation. The handle needs to be
// opened with THREAD_TERMINATE in order to be able to call CancelSynchronousIo.
- ThreadHandle = t_currentThreadHandle;
- if (ThreadHandle is null)
+ instance.ThreadHandle = t_currentThreadHandle;
+ if (instance.ThreadHandle is null)
{
- ThreadHandle = Interop.Kernel32.OpenThread(Interop.Kernel32.THREAD_TERMINATE, bInheritHandle: false, Interop.Kernel32.GetCurrentThreadId());
- if (ThreadHandle.IsInvalid)
+ instance.ThreadHandle = Interop.Kernel32.OpenThread(Interop.Kernel32.THREAD_TERMINATE, bInheritHandle: false, Interop.Kernel32.GetCurrentThreadId());
+ if (instance.ThreadHandle.IsInvalid)
{
int lastError = Marshal.GetLastPInvokeError();
Debug.Fail($"{nameof(Interop.Kernel32.OpenThread)} unexpectedly failed with 0x{lastError:X8}: {Marshal.GetPInvokeErrorMessage(lastError)}");
return default;
}
- t_currentThreadHandle = ThreadHandle;
+ t_currentThreadHandle = instance.ThreadHandle;
}
// Register with the token.
SyncAsyncWorkItemRegistration reg = default;
- reg.WorkItem = this;
+ reg.WorkItem = instance;
reg.CancellationRegistration = cancellationToken.UnsafeRegister(static s =>
{
- var state = (AsyncOverSyncWithIoCancellation)s!;
+ var instance = (AsyncOverSyncWithIoCancellation)s!;
// If cancellation was already requested when UnsafeRegister was called, it'll invoke
// the callback immediately. If we allowed that to loop until cancellation was successful,
// we'd deadlock, as we'd never perform the very I/O it was waiting for. As such, if
// the callback is invoked prior to be ready for it, we ignore the callback.
- if (!state.FinishedCancellationRegistration)
+ if (!instance.FinishedCancellationRegistration)
{
return;
}
@@ -250,17 +230,17 @@ private SyncAsyncWorkItemRegistration RegisterCancellation(CancellationToken can
// this looping synchronously, we instead queue the invocation of the looping so that it
// runs asynchronously from the Cancel call. Then in order to be able to track its completion,
// we store the Task representing that asynchronous work, such that cleanup can wait for the Task.
- state.CallbackCompleted = Task.Factory.StartNew(static s =>
+ instance.CallbackCompleted = Task.Factory.StartNew(static s =>
{
- var state = (AsyncOverSyncWithIoCancellation)s!;
+ var instance = (AsyncOverSyncWithIoCancellation)s!;
// Cancel the I/O. If the cancellation happens too early and we haven't yet initiated
// the synchronous operation, CancelSynchronousIo will fail with ERROR_NOT_FOUND, and
// we'll loop to try again.
SpinWait sw = default;
- while (state.ContinueTryingToCancel)
+ while (instance.ContinueTryingToCancel)
{
- if (Interop.Kernel32.CancelSynchronousIo(state.ThreadHandle!))
+ if (Interop.Kernel32.CancelSynchronousIo(instance.ThreadHandle!))
{
// Successfully canceled I/O.
break;
@@ -276,12 +256,12 @@ private SyncAsyncWorkItemRegistration RegisterCancellation(CancellationToken can
sw.SpinOnce();
}
- }, s, CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskScheduler.Default);
- }, this);
+ }, instance, CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskScheduler.Default);
+ }, instance);
// Now that we've registered with the token, tell the callback it's safe to enter
// its cancellation loop if the callback is invoked.
- FinishedCancellationRegistration = true;
+ instance.FinishedCancellationRegistration = true;
// And now since cancellation may have been requested and we may have suppressed it
// until the previous line, check to see if cancellation has now been requested, and
diff --git a/src/libraries/Common/src/System/TimeProvider.cs b/src/libraries/Common/src/System/TimeProvider.cs
index d1489826eecc40..a8db64c6739047 100644
--- a/src/libraries/Common/src/System/TimeProvider.cs
+++ b/src/libraries/Common/src/System/TimeProvider.cs
@@ -16,7 +16,7 @@ public abstract class TimeProvider
/// and a timer based on .
///
///
- /// If the changes after the object is returned, the change will be reflected in any subsequent operations that retrieve .
+ /// If the changes after the object is returned, the change will be reflected in any subsequent operations that retrieve .
///
public static TimeProvider System { get; } = new SystemTimeProvider();
@@ -274,7 +274,7 @@ private static (uint duration, uint periodTime) CheckAndGetValues(TimeSpan dueTi
}
///
- /// Used to create a instance returned from and uses the default implementation
+ /// Used to create a instance returned from and uses the default implementation
/// provided by which uses , , , and .
///
private sealed class SystemTimeProvider : TimeProvider
diff --git a/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs b/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs
index 2c7627dd9ac07e..5a2c897571ccf9 100644
--- a/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs
+++ b/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs
@@ -603,15 +603,6 @@ protected sealed class CustomTaskScheduler : TaskScheduler
protected override IEnumerable GetScheduledTasks() => new Task[0];
}
- protected readonly struct JumpToThreadPoolAwaiter : ICriticalNotifyCompletion
- {
- public JumpToThreadPoolAwaiter GetAwaiter() => this;
- public bool IsCompleted => false;
- public void OnCompleted(Action continuation) => ThreadPool.QueueUserWorkItem(_ => continuation());
- public void UnsafeOnCompleted(Action continuation) => ThreadPool.UnsafeQueueUserWorkItem(_ => continuation(), null);
- public void GetResult() { }
- }
-
protected sealed unsafe class NativeMemoryManager : MemoryManager
{
private readonly int _length;
@@ -2046,9 +2037,7 @@ public static IEnumerable ReadAsync_ContinuesOnCurrentContextIfDesired
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "iOS/tvOS blocks binding to UNIX sockets")]
public virtual async Task ReadAsync_ContinuesOnCurrentSynchronizationContextIfDesired(bool flowExecutionContext, bool? continueOnCapturedContext)
{
- await default(JumpToThreadPoolAwaiter); // escape xunit sync ctx
-
- using StreamPair streams = await CreateConnectedStreamsAsync();
+ using StreamPair streams = await CreateConnectedStreamsAsync().ConfigureAwait(ConfigureAwaitOptions.ForceYielding /* escape xunit sync ctx */);
foreach ((Stream writeable, Stream readable) in GetReadWritePairs(streams))
{
Assert.Null(SynchronizationContext.Current);
@@ -2130,9 +2119,7 @@ public virtual async Task ReadAsync_ContinuesOnCurrentSynchronizationContextIfDe
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "iOS/tvOS blocks binding to UNIX sockets")]
public virtual async Task ReadAsync_ContinuesOnCurrentTaskSchedulerIfDesired(bool flowExecutionContext, bool? continueOnCapturedContext)
{
- await default(JumpToThreadPoolAwaiter); // escape xunit sync ctx
-
- using StreamPair streams = await CreateConnectedStreamsAsync();
+ using StreamPair streams = await CreateConnectedStreamsAsync().ConfigureAwait(ConfigureAwaitOptions.ForceYielding /* escape xunit sync ctx */);
foreach ((Stream writeable, Stream readable) in GetReadWritePairs(streams))
{
Assert.Null(SynchronizationContext.Current);
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanFactory.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanFactory.cs
index 6497640809f3ab..84bbea4587eef4 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanFactory.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanFactory.cs
@@ -14,6 +14,7 @@ public interface IECDiffieHellmanProvider
bool ExplicitCurvesSupported { get; }
bool CanDeriveNewPublicKey { get; }
bool SupportsRawDerivation { get; }
+ bool SupportsSha3 { get; }
}
public static partial class ECDiffieHellmanFactory
@@ -45,5 +46,7 @@ public static bool IsCurveValid(Oid oid)
public static bool CanDeriveNewPublicKey => s_provider.CanDeriveNewPublicKey;
public static bool SupportsRawDerivation => s_provider.SupportsRawDerivation;
+
+ public static bool SupportsSha3 => s_provider.SupportsSha3;
}
}
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hash.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hash.cs
index dfca203f4fa14c..0065092e52aad1 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hash.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hash.cs
@@ -207,56 +207,87 @@ public static void DeriveKeyMaterialEquivalentToDeriveKeyFromHash()
public static IEnumerable HashDerivationTestCases()
{
- return new object[][]
+ yield return new object[]
{
- new object[]
- {
- HashAlgorithmName.SHA256,
- null,
- null,
- "595B71C33D9D40ACD9CA847C47267DAEE7498EEF0B553482FAA45791418AC679",
- },
+ HashAlgorithmName.SHA256,
+ null,
+ null,
+ "595B71C33D9D40ACD9CA847C47267DAEE7498EEF0B553482FAA45791418AC679",
+ };
- new object[]
- {
- HashAlgorithmName.SHA1,
- null,
- null,
- "25E464FAC33F4A5F8786627FB3685F4C31B26327",
- },
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA1,
+ null,
+ null,
+ "25E464FAC33F4A5F8786627FB3685F4C31B26327",
+ };
- new object[]
- {
- HashAlgorithmName.SHA256,
- "02040608",
- null,
- "D0F4C42D61E794E508A079822F3069C9F89D9E3385C8E090425FF38927798017",
- },
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "02040608",
+ null,
+ "D0F4C42D61E794E508A079822F3069C9F89D9E3385C8E090425FF38927798017",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ null,
+ "010305",
+ "20DCB58E2AC4E70B1BF47362B0D1C8B728E27D6575EA9B85106CBE05E1F7D6DB",
+ };
- new object[]
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "02040608",
+ "010305",
+ "EFC758D39896E9DE96C120B0A74FB751F140BD7F3F4FC3777DC2A530145E01EC",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "010305",
+ "02040608",
+ "7DB5520A5D6351595FC286CD53509D964FBB152C289F072581CB5E16EBF319E8",
+ };
+
+ if (ECDiffieHellmanFactory.SupportsSha3)
+ {
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl dgst -sha3-256
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
- null,
+ HashAlgorithmName.SHA3_256,
"010305",
- "20DCB58E2AC4E70B1BF47362B0D1C8B728E27D6575EA9B85106CBE05E1F7D6DB",
- },
+ "02040608",
+ "2AF6DA738DADF26607513ECB56451B5A476C7D42CFEC89872791B7A6C136A4F9",
+ };
- new object[]
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl dgst -sha3-384
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
- "02040608",
+ HashAlgorithmName.SHA3_384,
"010305",
- "EFC758D39896E9DE96C120B0A74FB751F140BD7F3F4FC3777DC2A530145E01EC",
- },
+ "02040608",
+ "EE9C10A3E60EA06296699195B3E338575DA529BA167A7520CA8BF50C86C4A08AB153DD7B97ADEE58CE5A9CAAC2F52ED1",
+ };
- new object[]
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl dgst -sha3-512
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
+ HashAlgorithmName.SHA3_512,
"010305",
"02040608",
- "7DB5520A5D6351595FC286CD53509D964FBB152C289F072581CB5E16EBF319E8",
- },
- };
+ "3614398AFC4A09B0490E095A7901FEA3C9C217E742064A371F2E04D6363EF864" +
+ "FE52EEEEA976FEC3FF98DE55D1E78E864F009FD834E1D301FA069BD44F6ECD52",
+ };
+ }
}
#if NETCOREAPP
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hmac.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hmac.cs
index b8ba791ddd7844..f95fb64aafd3a4 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hmac.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDiffieHellman/ECDiffieHellmanTests.Hmac.cs
@@ -248,71 +248,105 @@ public static void SimpleHmacNullKeyForwardsNull()
public static IEnumerable HmacDerivationTestCases()
{
- return new object[][]
+ yield return new object[]
{
- new object[]
- {
- HashAlgorithmName.SHA256,
- null,
- null,
- null,
- "6D7D15C9A08FD47DFDABD3541BE3BBAF93B15FC65D30E6012CCC0B23ED5C43FF",
- },
-
- new object[]
- {
- HashAlgorithmName.SHA1,
- null,
- null,
- null,
- "39D4B035BC1A1E4108B965689E27BA98ACED8449",
- },
-
- new object[]
- {
- HashAlgorithmName.SHA256,
- "030609",
- null,
- null,
- "7A4F81BF065CC521AFB162DB4A45CEFC78227178A58632EA53D3E367AB7D1979",
- },
+ HashAlgorithmName.SHA256,
+ null,
+ null,
+ null,
+ "6D7D15C9A08FD47DFDABD3541BE3BBAF93B15FC65D30E6012CCC0B23ED5C43FF",
+ };
- new object[]
- {
- HashAlgorithmName.SHA256,
- null,
- "02040608",
- "010305",
- "DB39A6AC9334701D2DCD508C401C65BC69348F684C85EDDE506950F049668842",
- },
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA1,
+ null,
+ null,
+ null,
+ "39D4B035BC1A1E4108B965689E27BA98ACED8449",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "030609",
+ null,
+ null,
+ "7A4F81BF065CC521AFB162DB4A45CEFC78227178A58632EA53D3E367AB7D1979",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ null,
+ "02040608",
+ "010305",
+ "DB39A6AC9334701D2DCD508C401C65BC69348F684C85EDDE506950F049668842",
+ };
- new object[]
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ null,
+ "010305",
+ "02040608",
+ "66471DE2655DF9404636F9076F845F0B71A04DDA2BA6F1469EB0F2E9EF57DC33",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "030609",
+ "02040608",
+ "010305",
+ "2F7A31FF9118A6BBF92E268568C634A9F1E244CA8C1A74C864DECC50727B7DEE",
+ };
+
+ yield return new object[]
+ {
+ HashAlgorithmName.SHA256,
+ "030609",
+ "010305",
+ "02040608",
+ "AE3CD974F262B199B0859D9F933207D2F6E3E04434D60089FE0BE801ED38D370",
+ };
+
+ if (ECDiffieHellmanFactory.SupportsSha3)
+ {
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl sha3-256 -mac HMAC -macopt hexkey:030609
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
- null,
+ HashAlgorithmName.SHA3_256,
+ "030609",
"010305",
"02040608",
- "66471DE2655DF9404636F9076F845F0B71A04DDA2BA6F1469EB0F2E9EF57DC33",
- },
+ "23E7E5648EF46D537F4159F7F40E686279D89ADFD7EF6CFA110034F42EC8CEF7",
+ };
- new object[]
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl sha3-384 -mac HMAC -macopt hexkey:030609
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
+ HashAlgorithmName.SHA3_384,
"030609",
- "02040608",
"010305",
- "2F7A31FF9118A6BBF92E268568C634A9F1E244CA8C1A74C864DECC50727B7DEE",
- },
+ "02040608",
+ "FD0F7B11489F641DE0964F81D83EF90E33C46D1628C51FA79A85AD1034A9CAD36F8A38C3925704AC0E404BC6FE50ECA4",
+ };
- new object[]
+ // Created with:
+ // (echo -n -e '\x01\x03\0x05'; openssl pkeyutl -derive -inkey private.key -peerkey public.key; echo -n -e '\x02\x04\0x06\0x08') | openssl sha3-512 -mac HMAC -macopt hexkey:030609
+ yield return new object[]
{
- HashAlgorithmName.SHA256,
+ HashAlgorithmName.SHA3_512,
"030609",
"010305",
"02040608",
- "AE3CD974F262B199B0859D9F933207D2F6E3E04434D60089FE0BE801ED38D370",
- },
- };
+ "2C32BBFBF8A41118AAD3BAA94C8995B5B027246EA3D972937C1BFD9F460C6492" +
+ "44EEC68EF570B4BB74B0D3BBD463F18526400A77211B5CB39311CDE21104E209",
+ };
+ }
}
#if NETCOREAPP
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDsa/ECDsaTests.NistValidation.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDsa/ECDsaTests.NistValidation.cs
index aab3429ed745b7..d0c596935ef092 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDsa/ECDsaTests.NistValidation.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/ECDsa/ECDsaTests.NistValidation.cs
@@ -246,6 +246,112 @@ public void ValidateNistP521Sha512()
HashAlgorithmName.SHA512);
}
+ [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.SupportsSha3))]
+ public void ValidateNistP256Sha3_256()
+ {
+ // From https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Standards-and-Guidelines/documents/examples/P256_SHA3-256.pdf
+ byte[] msg = "4578616d706c65206f66204543445341207769746820502d323536".HexToByteArray();
+
+ ECParameters parameters = new ECParameters
+ {
+ Curve = ECCurve.NamedCurves.nistP256,
+ Q = new ECPoint
+ {
+ X = "b7e08afdfe94bad3f1dc8c734798ba1c62b3a0ad1e9ea2a38201cd0889bc7a19".HexToByteArray(),
+ Y = "3603f747959dbf7a4bb226e41928729063adc7ae43529e61b563bbc606cc5e09".HexToByteArray(),
+ },
+ D = "c477f9f65c22cce20657faa5b2d1d8122336f851a508a1ed04e479c34985bf96".HexToByteArray(),
+ };
+
+ byte[] signature = (
+ // r
+ "2b42f576d07f4165ff65d1f3b1500f81e44c316f1f0b3ef57325b69aca46104f" +
+ // s
+ "0a861c2526900245c73bacb9adaec1a5acb3ba1f7114a3c334fdcd5b7690dadd"
+ ).HexToByteArray();
+
+ Validate(
+ parameters,
+ EccTestData.GetNistP256ExplicitCurve(),
+ msg,
+ signature,
+ HashAlgorithmName.SHA3_256);
+ }
+
+ [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.SupportsSha3))]
+ public void ValidateNistP384Sha3_384()
+ {
+ // From https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Standards-and-Guidelines/documents/examples/P384_SHA3-384.pdf
+ byte[] msg = "4578616d706c65206f66204543445341207769746820502d333834".HexToByteArray();
+
+ ECParameters parameters = new ECParameters
+ {
+ Curve = ECCurve.NamedCurves.nistP384,
+ Q = new ECPoint
+ {
+ X = ("3bf701bc9e9d36b4d5f1455343f09126f2564390f2b48736" +
+ "5071243c61e6471fb9d2ab74657b82f9086489d9ef0f5cb5").HexToByteArray(),
+ Y = ("d1a358eafbf952e68d533855ccbdaa6ff75b137a51014431" +
+ "99325583552a6295ffe5382d00cfcda30344a9b5b68db855").HexToByteArray(),
+ },
+ D = ("f92c02ed629e4b48c0584b1c6ce3a3e3b4faae4afc6acb04" +
+ "55e73dfc392e6a0ae393a8565e6b9714d1224b57d83f8a08").HexToByteArray(),
+ };
+
+ byte[] signature = (
+ // r
+ "30ea514fc0d38d8208756f068113c7cada9f66a3b40ea3b3" +
+ "13d040d9b57dd41a332795d02cc7d507fcef9faf01a27088" +
+ // s
+ "691b9d4969451a98036d53aa725458602125de74881bbc33" +
+ "3012ca4fa55bde39d1bf16a6aae3fe4992c567c6e7892337"
+ ).HexToByteArray();
+
+ Validate(
+ parameters,
+ EccTestData.GetNistP384ExplicitCurve(),
+ msg,
+ signature,
+ HashAlgorithmName.SHA3_384);
+ }
+
+ [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.SupportsSha3))]
+ public void ValidateNistP521Sha3_512()
+ {
+ // From https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Standards-and-Guidelines/documents/examples/P521_SHA3-512.pdf
+ byte[] msg = "4578616d706c65206f66204543445341207769746820502d353231".HexToByteArray();
+
+ ECParameters parameters = new ECParameters
+ {
+ Curve = ECCurve.NamedCurves.nistP521,
+ Q = new ECPoint
+ {
+ X = ("0098e91eef9a68452822309c52fab453f5f117c1da8ed796b255e9ab8f6410cca1" +
+ "6e59df403a6bdc6ca467a37056b1e54b3005d8ac030decfeb68df18b171885d5c4").HexToByteArray(),
+ Y = ("0164350c321aecfc1cca1ba4364c9b15656150b4b78d6a48d7d28e7f31985ef17b" +
+ "e8554376b72900712c4b83ad668327231526e313f5f092999a4632fd50d946bc2e").HexToByteArray(),
+ },
+ D = ("0100085f47b8e1b8b11b7eb33028c0b2888e304bfc98501955b45bba1478dc184e" +
+ "eedf09b86a5f7c21994406072787205e69a63709fe35aa93ba333514b24f961722").HexToByteArray(),
+ };
+
+ byte[] signature = (
+ // r
+ "0140c8edca57108ce3f7e7a240ddd3ad74d81e2de62451fc1d558fdc79269adacd" +
+ "1c2526eeeef32f8c0432a9d56e2b4a8a732891c37c9b96641a9254ccfe5dc3e2ba" +
+ // s
+ "00b25188492d58e808edebd7bf440ed20db771ca7c618595d5398e1b1c0098e300" +
+ "d8c803ec69ec5f46c84fc61967a302d366c627fcfa56f87f241ef921b6e627adbf"
+ ).HexToByteArray();
+
+ Validate(
+ parameters,
+ EccTestData.GetNistP521ExplicitCurve(),
+ msg,
+ signature,
+ HashAlgorithmName.SHA3_512);
+ }
+
private static void Validate(
ECParameters parameters,
ECCurve explicitCurve,
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs
index c74a993e54a7b6..5b97f468a4271e 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs
@@ -315,7 +315,19 @@ public void RsaCryptRoundtrip_OaepSHA384() =>
public void RsaCryptRoundtrip_OaepSHA512() =>
RsaCryptRoundtrip(RSAEncryptionPadding.OaepSHA512, RSAFactory.SupportsSha2Oaep);
- private void RsaCryptRoundtrip(RSAEncryptionPadding paddingMode, bool expectSuccess=true)
+ [Fact]
+ public void RsaCryptRoundtrip_OaepSHA3_256() =>
+ RsaCryptRoundtrip(RSAEncryptionPadding.OaepSHA3_256, RSAFactory.SupportsSha3);
+
+ [Fact]
+ public void RsaCryptRoundtrip_OaepSHA3_384() =>
+ RsaCryptRoundtrip(RSAEncryptionPadding.OaepSHA3_384, RSAFactory.SupportsSha3);
+
+ [Fact]
+ public void RsaCryptRoundtrip_OaepSHA3_512() =>
+ RsaCryptRoundtrip(RSAEncryptionPadding.OaepSHA3_512, RSAFactory.SupportsSha3);
+
+ private void RsaCryptRoundtrip(RSAEncryptionPadding paddingMode, bool expectSuccess = true)
{
byte[] crypt;
byte[] output;
@@ -324,9 +336,12 @@ private void RsaCryptRoundtrip(RSAEncryptionPadding paddingMode, bool expectSucc
{
if (!expectSuccess)
{
- Assert.ThrowsAny(
+ Exception ex = Assert.ThrowsAny(
() => Encrypt(rsa, TestData.HelloBytes, paddingMode));
+ Assert.True(ex is CryptographicException or PlatformNotSupportedException,
+ "exception is CryptographicException or PlatformNotSupportedException");
+
return;
}
@@ -370,6 +385,13 @@ void RoundtripEmpty(RSAEncryptionPadding paddingMode)
RoundtripEmpty(RSAEncryptionPadding.OaepSHA384);
RoundtripEmpty(RSAEncryptionPadding.OaepSHA512);
}
+
+ if (RSAFactory.SupportsSha3)
+ {
+ RoundtripEmpty(RSAEncryptionPadding.OaepSHA3_256);
+ RoundtripEmpty(RSAEncryptionPadding.OaepSHA3_384);
+ RoundtripEmpty(RSAEncryptionPadding.OaepSHA3_512);
+ }
}
}
@@ -715,6 +737,13 @@ public static IEnumerable OaepPaddingModes
yield return new object[] { RSAEncryptionPadding.OaepSHA384 };
yield return new object[] { RSAEncryptionPadding.OaepSHA512 };
}
+
+ if (RSAFactory.SupportsSha3)
+ {
+ yield return new object[] { RSAEncryptionPadding.OaepSHA3_256 };
+ yield return new object[] { RSAEncryptionPadding.OaepSHA3_384 };
+ yield return new object[] { RSAEncryptionPadding.OaepSHA3_512 };
+ }
}
}
}
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAFactory.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAFactory.cs
index 9a4b27348ab4a0..b13a6ee01d8e41 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAFactory.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAFactory.cs
@@ -12,6 +12,7 @@ public interface IRSAProvider
bool SupportsSha2Oaep { get; }
bool SupportsPss { get; }
bool SupportsSha1Signatures { get; }
+ bool SupportsSha3 { get; }
}
public static partial class RSAFactory
@@ -42,5 +43,8 @@ public static RSA Create(RSAParameters rsaParameters)
public static bool SupportsPss => s_provider.SupportsPss;
public static bool SupportsSha1Signatures => s_provider.SupportsSha1Signatures;
+
+ public static bool SupportsSha3 => s_provider.SupportsSha3;
+ public static bool NoSupportsSha3 => !SupportsSha3;
}
}
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/SignVerify.cs b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/SignVerify.cs
index 0bbdd7db7372da..6b0fd33bf89e7b 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/SignVerify.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/SignVerify.cs
@@ -504,6 +504,84 @@ public void VerifySignature_SHA256_2048()
VerifySignature(signature, TestData.HelloBytes, "SHA256", TestData.RSA2048Params);
}
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifySignature_SHA3_256_RSA2048()
+ {
+ byte[] signature = new byte[]
+ {
+ 0x37, 0xfc, 0x5b, 0xa7, 0xcd, 0x72, 0x93, 0x55, 0x43, 0x6f, 0x2c, 0x5c, 0xce, 0x10, 0x20, 0x67,
+ 0x38, 0xbc, 0x82, 0x65, 0xad, 0x64, 0x41, 0x3d, 0x7b, 0xe1, 0x13, 0x6e, 0xbf, 0x7c, 0x0a, 0xa7,
+ 0xc7, 0x42, 0x57, 0xf3, 0x75, 0xec, 0xda, 0xf1, 0xa0, 0xeb, 0x6d, 0x61, 0x04, 0x07, 0x4b, 0x3c,
+ 0xf8, 0x0d, 0x1d, 0x7a, 0xb2, 0x1f, 0xb1, 0xa4, 0xae, 0x35, 0x85, 0x18, 0xe4, 0xe1, 0xa5, 0x66,
+ 0x3d, 0x0c, 0xcd, 0x60, 0xf6, 0x6c, 0x89, 0xf6, 0x49, 0x87, 0x19, 0x3a, 0xa2, 0x24, 0x8a, 0xd9,
+ 0x17, 0x58, 0x2e, 0x50, 0x69, 0xd4, 0xae, 0x92, 0x17, 0xc1, 0xf8, 0x8e, 0xae, 0xca, 0x05, 0xc0,
+ 0x9c, 0x9d, 0xda, 0x0c, 0x82, 0x16, 0xae, 0xa4, 0x73, 0xe7, 0x65, 0x90, 0xb9, 0x25, 0x89, 0xe8,
+ 0xae, 0x60, 0x74, 0x86, 0x2e, 0xd6, 0xa8, 0x8a, 0x05, 0xda, 0x6c, 0x2e, 0x3f, 0x32, 0x24, 0x84,
+ 0x23, 0x87, 0x10, 0x5b, 0x42, 0x79, 0xdd, 0x64, 0x2c, 0xe4, 0x60, 0x75, 0x38, 0x2b, 0x37, 0x43,
+ 0x1c, 0x9b, 0x75, 0x6a, 0xf2, 0xfa, 0xbf, 0x95, 0x7f, 0xdb, 0x64, 0xe0, 0xa6, 0xc0, 0xb1, 0x24,
+ 0xcb, 0xf9, 0xdc, 0x30, 0xcd, 0xba, 0x54, 0x9a, 0xd4, 0xea, 0x3d, 0xcf, 0x3b, 0x08, 0x97, 0x19,
+ 0xdd, 0xc1, 0xf0, 0x20, 0xf6, 0x02, 0x7c, 0x0e, 0x0c, 0x84, 0x70, 0x89, 0x78, 0x5c, 0x50, 0xc5,
+ 0xce, 0xe5, 0xdd, 0x03, 0xf4, 0xc5, 0x86, 0x0e, 0xb7, 0x98, 0xd5, 0x96, 0x98, 0x2e, 0x02, 0xc9,
+ 0xb0, 0x41, 0x84, 0xa9, 0xa1, 0xda, 0x75, 0x81, 0x1d, 0x5a, 0x6a, 0x00, 0x33, 0x0f, 0x94, 0xa4,
+ 0x20, 0x0c, 0x91, 0xae, 0xae, 0xa4, 0xd3, 0xa8, 0x95, 0x20, 0x5c, 0x0c, 0x3b, 0x6e, 0x32, 0x7e,
+ 0xd3, 0x5b, 0xf4, 0x9b, 0x0c, 0xe2, 0x97, 0xd4, 0x1c, 0xf7, 0x98, 0x96, 0x77, 0xda, 0x05, 0xf5,
+ };
+
+ VerifySignature(signature, TestData.HelloBytes, HashAlgorithmName.SHA3_256.Name, TestData.RSA2048Params);
+ }
+
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifySignature_SHA3_384_RSA2048()
+ {
+ byte[] signature = new byte[]
+ {
+ 0x96, 0x76, 0xdf, 0xa3, 0xd3, 0xac, 0x74, 0x08, 0x18, 0x41, 0x0a, 0xa1, 0x03, 0x6b, 0x28, 0x5d,
+ 0xf8, 0xf7, 0xe5, 0x72, 0xe1, 0x7e, 0xb0, 0x11, 0x2f, 0x91, 0xbf, 0xce, 0xba, 0xa1, 0x4e, 0xfa,
+ 0x6f, 0x20, 0xaa, 0xc7, 0x8d, 0xf4, 0x8a, 0xba, 0xd3, 0xa3, 0x07, 0xa0, 0x0f, 0x09, 0x44, 0x7d,
+ 0xcc, 0xa1, 0x7b, 0xe4, 0x17, 0xcb, 0xe9, 0xe0, 0x78, 0xff, 0xd0, 0xf2, 0xdf, 0xd1, 0xdb, 0xd3,
+ 0x4a, 0xfb, 0xa8, 0x11, 0xac, 0xd3, 0x27, 0xcc, 0x05, 0x59, 0xc8, 0x16, 0x5d, 0x15, 0x68, 0x4d,
+ 0x20, 0x17, 0xd2, 0x52, 0xa1, 0xee, 0x7f, 0xce, 0xdb, 0xd4, 0x1a, 0x18, 0x6a, 0xc1, 0x2e, 0x42,
+ 0x74, 0xcd, 0xc8, 0x31, 0x92, 0x24, 0x33, 0x7e, 0x41, 0x0b, 0x16, 0xf3, 0xd4, 0xa0, 0x30, 0x78,
+ 0x8e, 0x3a, 0x05, 0xfa, 0xcb, 0x5b, 0x9f, 0x71, 0xd4, 0xfb, 0x27, 0x17, 0x19, 0x66, 0xe1, 0x05,
+ 0x6d, 0x3a, 0xd8, 0x89, 0x27, 0x19, 0x4f, 0x28, 0x3f, 0x6c, 0xb9, 0xfe, 0xca, 0xfb, 0x77, 0xcc,
+ 0xf1, 0x46, 0x2a, 0x3a, 0x74, 0x17, 0x8a, 0xb1, 0x76, 0x3a, 0x1b, 0xc3, 0x2e, 0x29, 0x29, 0xa6,
+ 0xeb, 0x1e, 0xf5, 0x3d, 0x4a, 0x41, 0x39, 0x49, 0xc3, 0x87, 0xe6, 0xcb, 0x49, 0x81, 0x8c, 0xa9,
+ 0x8a, 0xd8, 0xf5, 0x9e, 0x1c, 0x96, 0x8f, 0xac, 0xc4, 0xac, 0xd3, 0x1b, 0x7a, 0x09, 0x33, 0x48,
+ 0x62, 0xcc, 0x13, 0x3b, 0xb6, 0x9c, 0x20, 0x55, 0xb3, 0x3c, 0x0b, 0x77, 0xc5, 0x0e, 0x3d, 0x26,
+ 0x41, 0x15, 0xbf, 0xf7, 0xea, 0xdf, 0x44, 0x9c, 0x35, 0x0b, 0xca, 0x2e, 0x6e, 0xac, 0x04, 0x34,
+ 0x52, 0x18, 0x68, 0xa0, 0xc8, 0x3b, 0x51, 0xfb, 0x67, 0xed, 0x89, 0x29, 0x1e, 0xe4, 0xfe, 0xb3,
+ 0x78, 0xb1, 0x40, 0xe3, 0xd3, 0x53, 0xf4, 0x3e, 0xa5, 0x07, 0x33, 0x83, 0xf7, 0x02, 0x39, 0x1b,
+ };
+
+ VerifySignature(signature, TestData.HelloBytes, HashAlgorithmName.SHA3_384.Name, TestData.RSA2048Params);
+ }
+
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifySignature_SHA3_512_RSA2048()
+ {
+ byte[] signature = new byte[]
+ {
+ 0x18, 0x83, 0xab, 0x01, 0xc1, 0xaf, 0x98, 0xa9, 0xc2, 0xde, 0x2d, 0x63, 0x4f, 0x8a, 0xf6, 0x47,
+ 0xd9, 0xd2, 0xd1, 0xa0, 0x5b, 0xf4, 0x14, 0x6a, 0x49, 0x14, 0x56, 0x90, 0x63, 0xf9, 0x2c, 0xf6,
+ 0xed, 0xc2, 0x19, 0x65, 0x80, 0x0b, 0xa9, 0x7a, 0xa9, 0xcc, 0x00, 0x93, 0xdc, 0x8c, 0xcd, 0xef,
+ 0xa1, 0x76, 0x4c, 0x56, 0x8c, 0x36, 0x25, 0xcb, 0xbe, 0xd2, 0xbf, 0x49, 0xe6, 0xac, 0xbe, 0x50,
+ 0x52, 0xf8, 0xd0, 0xf9, 0xb0, 0xcb, 0xfb, 0xae, 0x41, 0x8b, 0x94, 0xbc, 0x47, 0xee, 0x71, 0x1d,
+ 0x9e, 0x4b, 0xcc, 0x6b, 0xe2, 0x66, 0x01, 0x92, 0x16, 0x06, 0x43, 0x64, 0x65, 0x19, 0x3a, 0xed,
+ 0x5f, 0x60, 0x85, 0xb1, 0xa8, 0x29, 0x76, 0x17, 0x2d, 0xb3, 0xc5, 0xee, 0x6d, 0x76, 0xef, 0x14,
+ 0xf8, 0x47, 0x8b, 0x35, 0x32, 0x4e, 0x8f, 0x79, 0xa8, 0x02, 0xe1, 0xe6, 0x49, 0xda, 0x50, 0x98,
+ 0xd7, 0xb9, 0x71, 0xf6, 0x9d, 0x9b, 0xcd, 0xb6, 0x0d, 0xd3, 0xc8, 0x9d, 0xf2, 0xac, 0xfa, 0xad,
+ 0xd4, 0xb6, 0xc2, 0x72, 0x9f, 0x14, 0xce, 0x24, 0x87, 0x68, 0x47, 0x6d, 0x7a, 0x73, 0xce, 0x67,
+ 0xb2, 0xd5, 0x07, 0x69, 0x0d, 0xc3, 0xdd, 0xb4, 0xc2, 0x56, 0xbc, 0x10, 0xc2, 0xe5, 0x9d, 0xb3,
+ 0xf9, 0x4d, 0xcd, 0xef, 0x9f, 0xe3, 0x3b, 0x8c, 0x49, 0xdf, 0x4b, 0x1f, 0xc0, 0x0c, 0x2d, 0xf6,
+ 0xad, 0x6c, 0x0f, 0x83, 0x5c, 0x74, 0xb0, 0xff, 0x95, 0x7f, 0x37, 0x34, 0x7b, 0x07, 0x50, 0x91,
+ 0x7f, 0xd2, 0x26, 0x39, 0x1f, 0xad, 0x72, 0x8a, 0x47, 0x2e, 0x6c, 0xb4, 0xf2, 0xcd, 0x09, 0xfd,
+ 0x1b, 0xca, 0xf6, 0xbe, 0x70, 0x14, 0xe8, 0x51, 0x1e, 0xf4, 0x5b, 0xd9, 0x57, 0xa6, 0x73, 0xe2,
+ 0xa0, 0xde, 0xc4, 0x67, 0x7a, 0x4a, 0xe2, 0x24, 0x64, 0x79, 0xd5, 0xe7, 0x39, 0xfc, 0xb2, 0xd9,
+ };
+
+ VerifySignature(signature, TestData.HelloBytes, HashAlgorithmName.SHA3_512.Name, TestData.RSA2048Params);
+ }
+
[Theory]
[MemberData(nameof(RoundTripTheories))]
public void SignAndVerify_Roundtrip(string hashAlgorithm, RSAParameters rsaParameters)
@@ -877,19 +955,10 @@ public void VerifyHashSignature_SHA256_2048()
VerifyHashSignature(hashSignature, dataHash, "SHA256", TestData.RSA2048Params);
}
- [ConditionalTheory]
- [InlineData("SHA256")]
- [InlineData("SHA384")]
- [InlineData("SHA512")]
- [InlineData("MD5")]
- [InlineData("SHA1")]
+ [Theory]
+ [MemberData(nameof(HashAlgorithmNames))]
public void PssRoundtrip(string hashAlgorithmName)
{
- if (!RSAFactory.SupportsSha1Signatures && hashAlgorithmName == "SHA1")
- {
- throw new SkipTestException("Platform does not support RSA with SHA1 signatures.");
- }
-
RSAParameters privateParameters = TestData.RSA2048Params;
RSAParameters publicParameters = new RSAParameters
{
@@ -1062,6 +1131,150 @@ public void VerifyExpectedSignature_PssSha512()
helloSignature);
}
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifyExpectedSignature_PssSha3_256()
+ {
+ // Signature independently created with
+ // echo -n Hello | openssl dgst -sha3-256 -binary | openssl pkeyutl -sign -inkey key.pem -pkeyopt rsa_padding_mode:pss -pkeyopt digest:sha3-256 | xxd -i -c 16
+ byte[] helloSignature =
+ {
+ 0x0b, 0x02, 0x36, 0x0c, 0x56, 0x6b, 0x93, 0xa8, 0x4e, 0x4d, 0x0e, 0xa3, 0x67, 0x28, 0xc6, 0xea,
+ 0xdb, 0x05, 0xc8, 0x63, 0x64, 0xe5, 0xec, 0xf9, 0xc1, 0x25, 0x5a, 0x34, 0xe6, 0x3f, 0xe1, 0x38,
+ 0x49, 0x45, 0x0b, 0x34, 0x6b, 0xa7, 0x91, 0xe1, 0x22, 0xe0, 0x2c, 0x8d, 0x46, 0x23, 0x38, 0xb1,
+ 0x9e, 0x4c, 0xf3, 0xa0, 0xff, 0x57, 0x96, 0x01, 0xbb, 0xa4, 0x0f, 0x03, 0x27, 0xaf, 0x8e, 0x99,
+ 0x5e, 0x0e, 0x99, 0xfd, 0xc6, 0x55, 0x81, 0x65, 0x56, 0x83, 0x2d, 0x42, 0xa3, 0x60, 0xc3, 0x36,
+ 0x69, 0xb4, 0x39, 0x04, 0x70, 0xfe, 0x26, 0x9b, 0x65, 0x6a, 0x15, 0xaa, 0xfc, 0xc5, 0x70, 0x71,
+ 0xba, 0x01, 0x49, 0xae, 0x27, 0xd4, 0x52, 0x28, 0xe4, 0xce, 0x37, 0x8a, 0x7d, 0x16, 0x2f, 0x59,
+ 0x3a, 0x1d, 0x36, 0x44, 0x82, 0xba, 0x51, 0xa3, 0x2c, 0xf9, 0x7e, 0xad, 0x36, 0xd4, 0x24, 0xfb,
+ 0x77, 0xae, 0x55, 0x10, 0x6e, 0xbc, 0x9d, 0xc8, 0xd7, 0xf4, 0x2b, 0x07, 0x70, 0x20, 0x8e, 0xd4,
+ 0xc6, 0xb3, 0x7f, 0x72, 0x9e, 0xa7, 0x0e, 0xd5, 0x31, 0x5d, 0x8c, 0x0c, 0xca, 0xae, 0x5a, 0x5a,
+ 0x38, 0x10, 0x42, 0x9f, 0x4f, 0x64, 0xc8, 0xe3, 0xdb, 0x4c, 0x17, 0x0b, 0x7b, 0x97, 0x82, 0x1f,
+ 0x9c, 0x88, 0xcf, 0x5e, 0xe7, 0x19, 0x8f, 0xa9, 0x5f, 0xd0, 0x1b, 0x19, 0x21, 0x2a, 0x2d, 0x36,
+ 0x45, 0x4a, 0xb1, 0x4d, 0xc8, 0xe1, 0x5e, 0x53, 0x24, 0xe1, 0x0b, 0x03, 0x01, 0x5d, 0xd9, 0x8f,
+ 0xe3, 0x3a, 0xdb, 0x3f, 0xbf, 0xa4, 0x24, 0x16, 0x44, 0x3e, 0x52, 0x5c, 0x6c, 0xfd, 0x2d, 0x0e,
+ 0x22, 0x69, 0xcf, 0x2a, 0x58, 0x8f, 0xe7, 0x31, 0x88, 0x22, 0xb8, 0x21, 0x68, 0x56, 0x6b, 0xb6,
+ 0xab, 0x76, 0x81, 0x41, 0x59, 0xb9, 0xe5, 0xc2, 0x61, 0x06, 0xea, 0x9b, 0x5b, 0x57, 0xf7, 0xb0,
+ };
+
+ VerifyExpectedSignature_Pss(
+ TestData.RSA2048Params,
+ HashAlgorithmName.SHA3_256,
+ TestData.HelloBytes,
+ helloSignature);
+ }
+
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifyExpectedSignature_PssSha3_384()
+ {
+ // Signature independently created with
+ // echo -n Hello | openssl dgst -sha3-384 -binary | openssl pkeyutl -sign -inkey key.pem -pkeyopt rsa_padding_mode:pss -pkeyopt digest:sha3-384 | xxd -i -c 16
+ byte[] helloSignature =
+ {
+ 0x75, 0x00, 0x1d, 0xa9, 0x3b, 0xfa, 0x79, 0x57, 0x4a, 0x3c, 0x9f, 0xec, 0x7b, 0xa4, 0x1f, 0x17,
+ 0x3a, 0x32, 0x20, 0x90, 0x62, 0x16, 0x25, 0xae, 0x80, 0x54, 0x71, 0xb0, 0xf5, 0x28, 0x5c, 0x97,
+ 0xfc, 0x4c, 0x21, 0x18, 0xb0, 0xfb, 0x90, 0xae, 0xca, 0xa6, 0x76, 0xf5, 0x14, 0x7a, 0x23, 0x47,
+ 0x39, 0x1c, 0xf4, 0x62, 0x23, 0x82, 0x18, 0x72, 0x58, 0x64, 0x9a, 0xe2, 0x3b, 0x4e, 0x5a, 0xe5,
+ 0xf1, 0x16, 0x77, 0xe4, 0xbe, 0xb5, 0x90, 0x54, 0x34, 0x38, 0x24, 0xe4, 0xe1, 0x59, 0xd2, 0x58,
+ 0x82, 0xc3, 0x68, 0x95, 0xb0, 0x1a, 0x0e, 0x1a, 0x67, 0x4a, 0x28, 0x1e, 0x7e, 0x5c, 0x67, 0xd2,
+ 0x2e, 0x9c, 0xb6, 0x12, 0x29, 0xf4, 0x57, 0x62, 0xaa, 0xf6, 0x46, 0x00, 0x90, 0xee, 0xc1, 0xb8,
+ 0x06, 0x40, 0x41, 0x8d, 0xc4, 0x69, 0x68, 0x75, 0xc5, 0xfc, 0xbc, 0x73, 0x40, 0x7c, 0xd6, 0xef,
+ 0xd0, 0x28, 0xe7, 0x32, 0x52, 0x34, 0x2d, 0x43, 0xca, 0xb8, 0x1e, 0x80, 0x07, 0x1c, 0xe6, 0x54,
+ 0xd9, 0x03, 0x06, 0xc0, 0xc7, 0x1a, 0x5a, 0xaf, 0x1a, 0xf3, 0xaa, 0xce, 0xa4, 0x02, 0xff, 0xf5,
+ 0xed, 0x7e, 0x0b, 0x2a, 0x68, 0x7d, 0x65, 0x98, 0xf4, 0xf4, 0x82, 0x3f, 0x54, 0x2a, 0x18, 0x02,
+ 0xf9, 0x3e, 0x3d, 0xed, 0x18, 0x98, 0xd5, 0xb2, 0x5f, 0x7f, 0xee, 0x26, 0xa1, 0x8f, 0xf5, 0x39,
+ 0x97, 0xeb, 0x71, 0x18, 0xf1, 0x3c, 0x10, 0x49, 0x34, 0x2e, 0xa8, 0x92, 0x50, 0x17, 0xb7, 0x91,
+ 0x6b, 0x66, 0x67, 0x2b, 0xd1, 0x73, 0x0b, 0x73, 0xa3, 0xc5, 0xef, 0x03, 0x70, 0xbc, 0x38, 0x59,
+ 0x1d, 0x23, 0xa2, 0xf8, 0x2d, 0xae, 0x43, 0xc8, 0x6d, 0x10, 0x2c, 0x56, 0x48, 0x2a, 0xc7, 0xef,
+ 0xf2, 0x4e, 0x5c, 0x30, 0x75, 0xbc, 0xd3, 0xd6, 0x4c, 0xfd, 0x45, 0xc6, 0x43, 0xca, 0xe8, 0x73,
+ };
+
+ VerifyExpectedSignature_Pss(
+ TestData.RSA2048Params,
+ HashAlgorithmName.SHA3_384,
+ TestData.HelloBytes,
+ helloSignature);
+ }
+
+ [ConditionalFact(typeof(RSAFactory), nameof(RSAFactory.SupportsSha3))]
+ public void VerifyExpectedSignature_PssSha3_512()
+ {
+ // Signature independently created with
+ // echo -n Hello | openssl dgst -sha3-512 -binary | openssl pkeyutl -sign -inkey key.pem -pkeyopt rsa_padding_mode:pss -pkeyopt digest:sha3-512 | xxd -i -c 16
+ byte[] helloSignature =
+ {
+ 0x2b, 0x02, 0x93, 0xd1, 0x56, 0x1d, 0x03, 0xce, 0xc7, 0xe0, 0xd0, 0x76, 0xf7, 0xce, 0x76, 0xd7,
+ 0xd6, 0xf1, 0xeb, 0xc2, 0xc4, 0x97, 0xee, 0x16, 0xdb, 0x04, 0x9c, 0x51, 0xd3, 0xc4, 0x76, 0xcd,
+ 0xdb, 0x77, 0x3f, 0x82, 0xf1, 0x31, 0x56, 0x77, 0xc8, 0x44, 0xfd, 0x29, 0x42, 0xf0, 0x68, 0x5d,
+ 0x27, 0x11, 0x74, 0xc1, 0x44, 0x4e, 0x30, 0x6e, 0xfc, 0x3c, 0xa8, 0x61, 0x14, 0x1d, 0x84, 0xec,
+ 0x92, 0x2c, 0x80, 0xeb, 0x57, 0xd0, 0x81, 0xe7, 0xae, 0xf7, 0x83, 0xe1, 0xe6, 0xfb, 0xc6, 0xed,
+ 0x5e, 0x0a, 0xd8, 0x67, 0x01, 0xf4, 0xc8, 0x8d, 0x27, 0x7c, 0x5f, 0xff, 0xb7, 0x9d, 0xb4, 0xd4,
+ 0x70, 0x18, 0xd4, 0x94, 0x7d, 0x36, 0xa7, 0x63, 0xe1, 0xa2, 0x30, 0x53, 0x6a, 0xef, 0x6e, 0xfa,
+ 0xad, 0xf4, 0xfd, 0xea, 0xba, 0x81, 0xd9, 0x73, 0x75, 0x90, 0xff, 0x98, 0x4c, 0xdb, 0x7d, 0xc6,
+ 0x91, 0xd9, 0x83, 0xdf, 0x31, 0xa3, 0xb4, 0x05, 0x91, 0xf9, 0xbe, 0x96, 0xfc, 0xed, 0xfd, 0x8b,
+ 0x8e, 0x9a, 0xee, 0xe5, 0xaf, 0xfd, 0x7e, 0x7e, 0xe0, 0xe1, 0x65, 0xa4, 0xf1, 0xdb, 0x4f, 0x01,
+ 0x47, 0x00, 0xee, 0x03, 0xb7, 0x96, 0x4c, 0x73, 0xdc, 0x6f, 0xf8, 0xf9, 0xac, 0xce, 0x1e, 0x4b,
+ 0xb8, 0x3d, 0x5b, 0xd5, 0xea, 0xf1, 0x4b, 0xd9, 0x63, 0x45, 0x2a, 0x49, 0x22, 0x2d, 0x6c, 0xc5,
+ 0x48, 0xe3, 0x1d, 0xbd, 0x92, 0xed, 0x42, 0x5a, 0x0c, 0xde, 0xb3, 0x2b, 0x91, 0xbd, 0x8e, 0x74,
+ 0x59, 0x6c, 0x9c, 0x39, 0x24, 0x43, 0xe4, 0x5c, 0xf6, 0x26, 0xd6, 0x30, 0xa0, 0x53, 0xb5, 0x2a,
+ 0x0d, 0x0d, 0xfc, 0x68, 0xb5, 0xae, 0x8d, 0x7e, 0xdf, 0x45, 0xfc, 0x66, 0x43, 0x93, 0x8d, 0x40,
+ 0x42, 0x29, 0xe1, 0xe9, 0xe3, 0xff, 0xda, 0xd8, 0x7d, 0x64, 0x9a, 0x26, 0xea, 0xb3, 0xba, 0x7d,
+ };
+
+ VerifyExpectedSignature_Pss(
+ TestData.RSA2048Params,
+ HashAlgorithmName.SHA3_512,
+ TestData.HelloBytes,
+ helloSignature);
+ }
+
+ [ConditionalTheory(typeof(RSAFactory), nameof(RSAFactory.NoSupportsSha3))]
+ [InlineData("SHA3-256")]
+ [InlineData("SHA3-384")]
+ [InlineData("SHA3-512")]
+ public void Pkcs1UnsupportedHashAlgorithm(string hashAlgorithm)
+ {
+ using (RSA rsa = RSAFactory.Create())
+ {
+ Exception ex = Assert.ThrowsAny(() =>
+ SignData(rsa, new byte[] { 1 }, new HashAlgorithmName(hashAlgorithm), RSASignaturePadding.Pkcs1));
+
+ Assert.True(
+ ex is PlatformNotSupportedException or CryptographicException,
+ "ex is PlatformNotSupportedException or CryptographicException");
+
+ ex = Assert.ThrowsAny(() =>
+ VerifyData(rsa, new byte[] { 1 }, new byte[] { 1 }, new HashAlgorithmName(hashAlgorithm), RSASignaturePadding.Pkcs1));
+
+ Assert.True(
+ ex is PlatformNotSupportedException or CryptographicException,
+ "ex is PlatformNotSupportedException or CryptographicException");
+ }
+ }
+
+ [ConditionalTheory(typeof(RSAFactory), nameof(RSAFactory.NoSupportsSha3))]
+ [InlineData("SHA3-256")]
+ [InlineData("SHA3-384")]
+ [InlineData("SHA3-512")]
+ public void PssUnsupportedHashAlgorithm(string hashAlgorithm)
+ {
+ using (RSA rsa = RSAFactory.Create())
+ {
+ Exception ex = Assert.ThrowsAny(() =>
+ SignData(rsa, new byte[] { 1 }, new HashAlgorithmName(hashAlgorithm), RSASignaturePadding.Pss));
+
+ Assert.True(
+ ex is CryptographicException or PlatformNotSupportedException,
+ "ex is CryptographicException or PlatformNotSupportedException");
+
+ ex = Assert.ThrowsAny(() =>
+ VerifyData(rsa, new byte[] { 1 }, new byte[] { 1 }, new HashAlgorithmName(hashAlgorithm), RSASignaturePadding.Pss));
+
+ Assert.True(
+ ex is CryptographicException or PlatformNotSupportedException,
+ "ex is CryptographicException or PlatformNotSupportedException");
+ }
+ }
+
private void VerifyExpectedSignature_Pss(
RSAParameters keyParameters,
HashAlgorithmName hashAlgorithm,
@@ -1314,5 +1527,28 @@ private void SignAndVerify(byte[] data, string hashAlgorithmName, RSAParameters
Assert.True(signatureMatched);
}
}
+
+ public static IEnumerable HashAlgorithmNames
+ {
+ get
+ {
+ yield return new object[] { HashAlgorithmName.SHA256.Name };
+ yield return new object[] { HashAlgorithmName.SHA384.Name };
+ yield return new object[] { HashAlgorithmName.SHA512.Name };
+ yield return new object[] { HashAlgorithmName.MD5.Name };
+
+ if (RSAFactory.SupportsSha1Signatures)
+ {
+ yield return new object[] { HashAlgorithmName.SHA1.Name };
+ }
+
+ if (RSAFactory.SupportsSha3)
+ {
+ yield return new object[] { HashAlgorithmName.SHA3_256.Name };
+ yield return new object[] { HashAlgorithmName.SHA3_384.Name };
+ yield return new object[] { HashAlgorithmName.SHA3_512.Name };
+ }
+ }
+ }
}
}
diff --git a/src/libraries/Common/tests/System/Security/Cryptography/SP800108HmacCounterKdfTests.Functional.cs b/src/libraries/Common/tests/System/Security/Cryptography/SP800108HmacCounterKdfTests.Functional.cs
index 338ffc7a2a6527..5eb6d097026c65 100644
--- a/src/libraries/Common/tests/System/Security/Cryptography/SP800108HmacCounterKdfTests.Functional.cs
+++ b/src/libraries/Common/tests/System/Security/Cryptography/SP800108HmacCounterKdfTests.Functional.cs
@@ -3,6 +3,7 @@
using System.Collections.Generic;
using System.Globalization;
+using Microsoft.DotNet.XUnitExtensions;
using Xunit;
namespace System.Security.Cryptography.Tests
@@ -81,7 +82,7 @@ public static void EmptyTests(byte[] key, string label, string context, byte[] e
VerifyKbkdf(expected, key, HashAlgorithmName.SHA256, label.ToCharArray(), context.ToCharArray());
}
- [Theory]
+ [ConditionalTheory]
[InlineData(nameof(HashAlgorithmName.SHA1), 512 / 8 - 1, new byte[] { 0xc9, 0x0f, 0x9d, 0x91, 0x85, 0xe5, 0xeb, 0x9b })]
[InlineData(nameof(HashAlgorithmName.SHA1), 512 / 8, new byte[] { 0x7b, 0xdb, 0x38, 0x28, 0xc0, 0x9f, 0x49, 0x05 })]
[InlineData(nameof(HashAlgorithmName.SHA1), 512 / 8 + 1, new byte[] { 0x6c, 0x3a, 0xba, 0x28, 0x38, 0xad, 0x51, 0x2c })]
@@ -94,8 +95,28 @@ public static void EmptyTests(byte[] key, string label, string context, byte[] e
[InlineData(nameof(HashAlgorithmName.SHA512), 1024 / 8 - 1, new byte[] { 0xa4, 0xe5, 0x24, 0xe8, 0x56, 0x2b, 0x48, 0xa4 })]
[InlineData(nameof(HashAlgorithmName.SHA512), 1024 / 8, new byte[] { 0xba, 0xf6, 0xed, 0xa7, 0x3a, 0xf7, 0x12, 0x27 })]
[InlineData(nameof(HashAlgorithmName.SHA512), 1024 / 8 + 1, new byte[] { 0x34, 0xdf, 0x2d, 0x21, 0xfd, 0xf1, 0x0e, 0x13 })]
+#if NET8_0_OR_GREATER
+ [InlineData("SHA3-256", 1088 / 8 - 1, new byte[] { 0xa1, 0x96, 0xae, 0x83, 0x56, 0xf4, 0x2a, 0x4b })]
+ [InlineData("SHA3-256", 1088 / 8, new byte[] { 0xe7, 0xe9, 0xe0, 0x98, 0x09, 0x54, 0x54, 0x2d })]
+ [InlineData("SHA3-256", 1088 / 8 + 1, new byte[] { 0x7d, 0x7a, 0x71, 0xdf, 0x1f, 0x5d, 0x5b, 0x44 })]
+ [InlineData("SHA3-384", 832 / 8 - 1, new byte[] { 0xd6, 0x08, 0x69, 0xd0, 0x99, 0x98, 0x6d, 0xcc })]
+ [InlineData("SHA3-384", 832 / 8, new byte[] { 0x49, 0x83, 0x06, 0x4e, 0x08, 0xf8, 0x93, 0x62 })]
+ [InlineData("SHA3-384", 832 / 8 + 1, new byte[] { 0xcc, 0x03, 0x1f, 0x57, 0x5e, 0x0c, 0xe1, 0xe8 })]
+ [InlineData("SHA3-512", 576 / 8 - 1, new byte[] { 0x47, 0xd2, 0x7e, 0x61, 0x01, 0x61, 0x9a, 0xd0 })]
+ [InlineData("SHA3-512", 576 / 8, new byte[] { 0xda, 0x56, 0x5b, 0x08, 0x73, 0xbc, 0x4d, 0x33 })]
+ [InlineData("SHA3-512", 576 / 8 + 1, new byte[] { 0xd3, 0xa1, 0xfd, 0x76, 0xc4, 0xf9, 0x62, 0xc3 })]
+#endif
public static void Kdk_HmacBlockBoundarySizes(string hashAlgorithmName, int kdkSize, byte[] expected)
{
+#if NET8_0_OR_GREATER
+ if ((hashAlgorithmName == "SHA3-256" && !SHA3_256.IsSupported) ||
+ (hashAlgorithmName == "SHA3-384" && !SHA3_384.IsSupported) ||
+ (hashAlgorithmName == "SHA3-512" && !SHA3_512.IsSupported))
+ {
+ throw new SkipTestException($"Algorithm '{hashAlgorithmName}' is not supported on the current platform.");
+ }
+#endif
+
// We do HMAC key adjust for the CNG implementation when the kdk exceeds the block size of the HMAC algorithm.
// This tests one byte below, at, and above the block size for each HMAC algorithm.
// Verified against OpenSSL 3. Example command used below. Adjust the digest and the seq upper boundary as needed.
@@ -306,6 +327,149 @@ public static IEnumerable GetOutputLengthBoundaries()
0x61,
}
};
+
+#if NET8_0_OR_GREATER
+ if (HMACSHA3_256.IsSupported)
+ {
+ // HMACSHA3_256 output size is 32 bytes
+ yield return new object[]
+ {
+ "SHA3-256",
+ new byte[63]
+ {
+ 0x25, 0xb6, 0xc5, 0xba, 0x60, 0x47, 0x95, 0xb6, 0x5c, 0x92, 0xcb, 0x8f, 0xd2, 0x4d, 0x40, 0xfa,
+ 0x68, 0xa9, 0xa6, 0xd3, 0x2c, 0x7f, 0x3c, 0xcd, 0x88, 0x89, 0xdc, 0x45, 0x10, 0xeb, 0x16, 0x38,
+ 0xf3, 0xe2, 0xe0, 0x26, 0xb5, 0xbc, 0x61, 0x9f, 0x56, 0x66, 0x7d, 0xbd, 0x55, 0x4e, 0x6b, 0x70,
+ 0x84, 0xc1, 0x9c, 0x95, 0x1e, 0x8f, 0x1b, 0xdc, 0xc2, 0x68, 0xc3, 0x10, 0x9c, 0x13, 0xf8,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-256",
+ new byte[64]
+ {
+ 0xc8, 0x8a, 0xfe, 0xd4, 0x62, 0xa6, 0x9e, 0xa6, 0x90, 0xa7, 0xb1, 0xe2, 0x01, 0xff, 0x67, 0x52,
+ 0x5d, 0xca, 0xa9, 0x13, 0xa2, 0x92, 0x3c, 0xc0, 0x42, 0xd1, 0x1b, 0xe2, 0xff, 0xe0, 0xa2, 0xb9,
+ 0x48, 0x3d, 0xb1, 0x17, 0x73, 0x76, 0x3d, 0x37, 0x0f, 0xc6, 0x6f, 0x65, 0xa5, 0xb9, 0x23, 0xcf,
+ 0xc0, 0xd4, 0x5f, 0xdf, 0x46, 0x0e, 0xab, 0xce, 0x8e, 0x7d, 0x2c, 0x9d, 0x71, 0x40, 0xf0, 0x89,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-256",
+ new byte[65]
+ {
+ 0xcc, 0x3a, 0x45, 0x55, 0x26, 0x1e, 0x56, 0xf4, 0x3c, 0x18, 0x66, 0x1d, 0xa8, 0xe3, 0x91, 0xe7,
+ 0xbc, 0xc3, 0x93, 0x1f, 0x85, 0xbe, 0x0c, 0xfb, 0xab, 0x10, 0xa1, 0x52, 0x93, 0xc7, 0x3c, 0xac,
+ 0x20, 0x09, 0xfb, 0x40, 0x95, 0xb9, 0x0f, 0x0a, 0x35, 0x50, 0xe2, 0x05, 0x33, 0x20, 0xdd, 0xb0,
+ 0xb7, 0xca, 0xda, 0x15, 0xa8, 0xc8, 0x73, 0x5d, 0xb7, 0x22, 0x5f, 0x23, 0xef, 0xcd, 0x33, 0x41,
+ 0xa2,
+ }
+ };
+ }
+
+ if (HMACSHA3_384.IsSupported)
+ {
+ // HMACSHA3_384 output size is 48 bytes
+ yield return new object[]
+ {
+ "SHA3-384",
+ new byte[95]
+ {
+ 0x80, 0x3b, 0x0a, 0x83, 0xe5, 0xae, 0xab, 0xff, 0x16, 0x7a, 0x04, 0x60, 0x97, 0x74, 0x39, 0xcf,
+ 0x8b, 0xeb, 0xbb, 0x57, 0x89, 0x62, 0x24, 0xab, 0x01, 0x61, 0xce, 0x23, 0xbc, 0xc1, 0xe0, 0xc4,
+ 0x9e, 0x73, 0xd1, 0x21, 0xdc, 0x21, 0xee, 0x30, 0xdb, 0xec, 0xb0, 0xbf, 0xec, 0xaa, 0x24, 0x18,
+ 0xf9, 0x93, 0xda, 0xd1, 0xaf, 0xfe, 0xbb, 0x1c, 0x7f, 0xcd, 0xea, 0x8f, 0x57, 0x41, 0x3c, 0x01,
+ 0xa8, 0x94, 0x06, 0x27, 0x8d, 0x95, 0x5d, 0x62, 0x61, 0xe7, 0x67, 0x60, 0xc4, 0xa2, 0xc4, 0xf5,
+ 0x7e, 0x38, 0x6e, 0x30, 0x38, 0x83, 0x7d, 0x5e, 0xf4, 0x0f, 0xdf, 0xd3, 0xac, 0x65, 0x48,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-384",
+ new byte[96]
+ {
+ 0xb8, 0xea, 0xbd, 0xcb, 0x67, 0xcb, 0xd7, 0xf5, 0x12, 0xdf, 0xc2, 0x35, 0x89, 0x66, 0x80, 0x6f,
+ 0xc4, 0x7c, 0x28, 0xf2, 0x4b, 0xf5, 0xd1, 0x4d, 0x54, 0x4c, 0x78, 0xae, 0xde, 0xc2, 0xae, 0xb0,
+ 0x7d, 0xfc, 0x02, 0x6e, 0x0e, 0x08, 0x2b, 0xfc, 0xd2, 0x2b, 0xae, 0xfc, 0xa1, 0x55, 0xf7, 0xc7,
+ 0xd5, 0x77, 0xe4, 0x84, 0xbb, 0x53, 0x1d, 0x3d, 0xd2, 0x7a, 0x61, 0x97, 0x8a, 0xd3, 0xae, 0x1b,
+ 0x47, 0xbe, 0xa9, 0x40, 0x02, 0xac, 0x55, 0x1b, 0x74, 0x72, 0x02, 0x5b, 0xa8, 0xa8, 0xd4, 0xfe,
+ 0xcf, 0xd1, 0xb2, 0x44, 0x01, 0xec, 0xb7, 0xef, 0x63, 0x0f, 0x16, 0xae, 0xa9, 0x63, 0xb1, 0x56,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-384",
+ new byte[97]
+ {
+ 0x62, 0x93, 0xa5, 0x45, 0x03, 0x7a, 0x2f, 0x50, 0xfe, 0x6b, 0xe6, 0x85, 0x56, 0x88, 0x8c, 0x78,
+ 0xbd, 0x4f, 0x08, 0x68, 0xdd, 0x33, 0xab, 0x18, 0x05, 0x81, 0x79, 0x70, 0x24, 0xe0, 0x3f, 0x94,
+ 0x40, 0x9b, 0xc3, 0x41, 0xa2, 0xe0, 0xfd, 0x02, 0xe8, 0xb8, 0x41, 0x53, 0x2e, 0x95, 0x59, 0xc5,
+ 0xf7, 0xab, 0x5b, 0xe9, 0x2d, 0x15, 0x0a, 0xcc, 0xb8, 0xc0, 0x87, 0xbd, 0x78, 0x39, 0xb9, 0xb8,
+ 0x83, 0x2c, 0x49, 0x7b, 0x45, 0x9b, 0x81, 0xed, 0x33, 0xdc, 0x75, 0xf1, 0x7d, 0xbd, 0x44, 0xce,
+ 0xa5, 0xf3, 0xbe, 0x43, 0xa5, 0x15, 0x7c, 0x7e, 0xcd, 0x06, 0x09, 0xb0, 0x4a, 0x17, 0x48, 0xf3,
+ 0x51,
+ }
+ };
+ }
+
+ if (HMACSHA3_512.IsSupported)
+ {
+ // HMACSHA3_512 output size is 64 bytes
+ yield return new object[]
+ {
+ "SHA3-512",
+ new byte[127]
+ {
+ 0x41, 0xe4, 0xb5, 0xe6, 0x5b, 0x8c, 0xce, 0x14, 0xa8, 0x39, 0xd3, 0xdf, 0xb4, 0x27, 0x0f, 0xff,
+ 0x1f, 0x9a, 0xca, 0x24, 0x03, 0x59, 0x4b, 0x86, 0x40, 0x66, 0xf0, 0xae, 0xdc, 0x59, 0x44, 0x0e,
+ 0x6a, 0x87, 0xa5, 0x38, 0x5a, 0x79, 0x27, 0xa5, 0x61, 0x91, 0x1a, 0x9d, 0x39, 0xca, 0x80, 0xe0,
+ 0x11, 0x3b, 0xd0, 0xfa, 0x2e, 0x48, 0x70, 0x83, 0x31, 0xa7, 0xa6, 0x6c, 0x5f, 0xf2, 0x2f, 0x09,
+ 0x5b, 0x1e, 0xad, 0x8b, 0x4f, 0xbd, 0x63, 0x26, 0x6f, 0xcd, 0xc6, 0xce, 0xff, 0xa6, 0x86, 0xbe,
+ 0xf6, 0x4b, 0x81, 0x63, 0x65, 0x24, 0x48, 0xbc, 0x59, 0x6a, 0x4f, 0x00, 0xb5, 0x9e, 0x9b, 0xbd,
+ 0x85, 0xbc, 0xf6, 0xf1, 0xf1, 0xaa, 0xd3, 0x91, 0x9b, 0x7b, 0x26, 0xfe, 0xd0, 0xd7, 0x4c, 0x34,
+ 0xdd, 0x2c, 0x57, 0x2e, 0x10, 0x09, 0x76, 0x32, 0xed, 0x3a, 0x95, 0x7d, 0x88, 0x2b, 0xdc,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-512",
+ new byte[128]
+ {
+ 0xbe, 0x9f, 0x29, 0xd7, 0x9d, 0x11, 0x7e, 0xd7, 0x33, 0xd4, 0xcf, 0x66, 0xed, 0x1a, 0x61, 0xfd,
+ 0xb0, 0x68, 0x3b, 0x03, 0xd1, 0x7c, 0x1a, 0x69, 0xd0, 0x33, 0x1c, 0xf4, 0x7a, 0xf0, 0x4a, 0x46,
+ 0xed, 0x4a, 0x33, 0xbf, 0x2b, 0x18, 0x16, 0x4b, 0xda, 0x30, 0x96, 0x11, 0xee, 0x52, 0x4b, 0x0e,
+ 0x22, 0xa0, 0x8f, 0xb0, 0xdc, 0xe6, 0x8c, 0x57, 0xd0, 0x6e, 0x25, 0xd8, 0x29, 0xf7, 0xf9, 0x94,
+ 0xe3, 0x73, 0x20, 0x4e, 0x11, 0xc0, 0xa9, 0x3b, 0x00, 0xf0, 0x8e, 0xff, 0xb3, 0x07, 0x87, 0x4c,
+ 0x3b, 0x96, 0x09, 0xff, 0x14, 0x3b, 0x65, 0x17, 0x5f, 0x54, 0x16, 0xaf, 0x15, 0xe5, 0x59, 0x5b,
+ 0x75, 0xf5, 0x03, 0x61, 0x5b, 0xd9, 0xf3, 0x7c, 0x71, 0x96, 0xaa, 0x67, 0xef, 0xdd, 0x40, 0x4c,
+ 0xd2, 0x9e, 0xf6, 0x05, 0x02, 0x93, 0x23, 0x83, 0x33, 0xcc, 0x0f, 0xf2, 0xdd, 0x9c, 0xd6, 0x46,
+ }
+ };
+
+ yield return new object[]
+ {
+ "SHA3-512",
+ new byte[129]
+ {
+ 0x5f, 0xf7, 0xd9, 0x1e, 0x98, 0xb3, 0xa4, 0xab, 0x4d, 0xc8, 0x50, 0x9a, 0xd7, 0x50, 0x37, 0xba,
+ 0x10, 0xeb, 0x1b, 0x10, 0x5f, 0x6c, 0xfe, 0x81, 0x75, 0x5d, 0x6c, 0x63, 0x8a, 0xfe, 0x60, 0x4f,
+ 0xb3, 0x9e, 0xf4, 0x84, 0x5d, 0xa3, 0x88, 0xf7, 0x29, 0x5c, 0x30, 0xbb, 0xf7, 0x30, 0x69, 0x84,
+ 0xab, 0x9e, 0xc6, 0xb0, 0xc3, 0xce, 0x45, 0x7a, 0xf2, 0xac, 0x18, 0x0b, 0x09, 0x8c, 0xbb, 0xae,
+ 0x22, 0x12, 0xef, 0xd8, 0xec, 0x6f, 0x9f, 0xbf, 0x09, 0x7f, 0xe2, 0xb6, 0xc8, 0x6a, 0xf7, 0x5e,
+ 0x86, 0x27, 0x22, 0xeb, 0xa8, 0x7b, 0x1f, 0x54, 0x97, 0x16, 0x0f, 0x21, 0xad, 0xcd, 0xd9, 0x9d,
+ 0xd6, 0x01, 0x0b, 0x79, 0xe9, 0x16, 0xd8, 0xcd, 0xa6, 0x93, 0x25, 0xd1, 0xeb, 0xac, 0x9b, 0x5f,
+ 0x26, 0xf4, 0xad, 0x0a, 0xf2, 0xc6, 0x4d, 0xad, 0x69, 0x96, 0xb7, 0xc8, 0xcd, 0xe0, 0x13, 0xfb,
+ 0xf6,
+ }
+ };
+ }
+#endif
}
public static IEnumerable GetRfc8009TestVectors()
diff --git a/src/libraries/Common/tests/System/TimeProviderTests.cs b/src/libraries/Common/tests/System/TimeProviderTests.cs
index b3fe700165e2fd..93a92dde712907 100644
--- a/src/libraries/Common/tests/System/TimeProviderTests.cs
+++ b/src/libraries/Common/tests/System/TimeProviderTests.cs
@@ -321,7 +321,7 @@ public static void RunDelayTests(TimeProvider provider, ITestTaskFactory taskFac
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
[MemberData(nameof(TimersProvidersWithTaskFactorData))]
- public static async void RunWaitAsyncTests(TimeProvider provider, ITestTaskFactory taskFactory)
+ public static async Task RunWaitAsyncTests(TimeProvider provider, ITestTaskFactory taskFactory)
{
CancellationTokenSource cts = new CancellationTokenSource();
@@ -377,7 +377,7 @@ public static async void RunWaitAsyncTests(TimeProvider provider, ITestTaskFacto
#if !NETFRAMEWORK
[ConditionalTheory(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
[MemberData(nameof(TimersProvidersListData))]
- public static async void PeriodicTimerTests(TimeProvider provider)
+ public static async Task PeriodicTimerTests(TimeProvider provider)
{
var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(1), provider);
Assert.True(await timer.WaitForNextTickAsync());
diff --git a/src/libraries/Common/tests/TestUtilities/System/AssertExtensions.cs b/src/libraries/Common/tests/TestUtilities/System/AssertExtensions.cs
index 479d2aef83c6e2..bf676b7b6f26be 100644
--- a/src/libraries/Common/tests/TestUtilities/System/AssertExtensions.cs
+++ b/src/libraries/Common/tests/TestUtilities/System/AssertExtensions.cs
@@ -405,6 +405,15 @@ public static void Equal(HashSet expected, HashSet actual)
}
}
+ /// Validates that the two sets contains the same elements. XUnit doesn't display the full collections.
+ public static void Equal(ISet expected, ISet actual)
+ {
+ if (!actual.SetEquals(expected))
+ {
+ throw new XunitException($"Expected: {string.Join(", ", expected)}{Environment.NewLine}Actual: {string.Join(", ", actual)}");
+ }
+ }
+
///
/// Validates that the actual collection contains same items as expected collection. If the test fails, this will display:
/// 1. Count if two collection count are different;
diff --git a/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs b/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs
index 68ca7995303e7e..88ab461e49dcea 100644
--- a/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs
+++ b/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs
@@ -303,6 +303,7 @@ private static bool GetAlpnSupport()
private static readonly Lazy s_supportsTls12 = new Lazy(GetTls12Support);
private static readonly Lazy s_supportsTls13 = new Lazy(GetTls13Support);
private static readonly Lazy s_sendsCAListByDefault = new Lazy(GetSendsCAListByDefault);
+ private static readonly Lazy s_supportsSha3 = new Lazy(GetSupportsSha3);
public static bool SupportsTls10 => s_supportsTls10.Value;
public static bool SupportsTls11 => s_supportsTls11.Value;
@@ -310,6 +311,8 @@ private static bool GetAlpnSupport()
public static bool SupportsTls13 => s_supportsTls13.Value;
public static bool SendsCAListByDefault => s_sendsCAListByDefault.Value;
public static bool SupportsSendingCustomCANamesInTls => UsesAppleCrypto || IsOpenSslSupported || (PlatformDetection.IsWindows8xOrLater && SendsCAListByDefault);
+ public static bool SupportsSha3 => s_supportsSha3.Value;
+ public static bool DoesNotSupportSha3 => !s_supportsSha3.Value;
private static readonly Lazy s_largeArrayIsNotSupported = new Lazy(IsLargeArrayNotSupported);
@@ -651,5 +654,25 @@ private static bool AssemblyConfigurationEquals(string configuration)
return assemblyConfigurationAttribute != null &&
string.Equals(assemblyConfigurationAttribute.Configuration, configuration, StringComparison.InvariantCulture);
}
+
+ private static bool GetSupportsSha3()
+ {
+ if (IsOpenSslSupported)
+ {
+ if (OpenSslVersion.Major >= 3)
+ {
+ return true;
+ }
+
+ return OpenSslVersion.Major == 1 && OpenSslVersion.Minor >= 1 && OpenSslVersion.Build >= 1;
+ }
+
+ if (IsWindowsVersionOrLater(10, 0, 25324))
+ {
+ return true;
+ }
+
+ return false;
+ }
}
}
diff --git a/src/libraries/Directory.Build.targets b/src/libraries/Directory.Build.targets
index bc1e24d670a421..8e000b5f4d7604 100644
--- a/src/libraries/Directory.Build.targets
+++ b/src/libraries/Directory.Build.targets
@@ -47,14 +47,8 @@
-->
$(WarningsAsErrors.Replace('NU1605', ''))
- true
-
- truetrue
- true
+ truetrue
diff --git a/src/libraries/Microsoft.Bcl.Cryptography/src/System/Security/Cryptography/HashAlgorithmNames.cs b/src/libraries/Microsoft.Bcl.Cryptography/src/System/Security/Cryptography/HashAlgorithmNames.cs
index 8f10590302e4bb..feb43d2caa487b 100644
--- a/src/libraries/Microsoft.Bcl.Cryptography/src/System/Security/Cryptography/HashAlgorithmNames.cs
+++ b/src/libraries/Microsoft.Bcl.Cryptography/src/System/Security/Cryptography/HashAlgorithmNames.cs
@@ -3,11 +3,15 @@
namespace System.Security.Cryptography
{
+ // Strings need to match CNG identifiers.
internal static class HashAlgorithmNames
{
internal const string SHA1 = nameof(SHA1);
internal const string SHA256 = nameof(SHA256);
internal const string SHA384 = nameof(SHA384);
internal const string SHA512 = nameof(SHA512);
+ internal const string SHA3_256 = "SHA3-256";
+ internal const string SHA3_384 = "SHA3-384";
+ internal const string SHA3_512 = "SHA3-512";
}
}
diff --git a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/IDispatchMetaObject.cs b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/IDispatchMetaObject.cs
index d1c80b62c0a580..c6aed3cca13b6e 100644
--- a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/IDispatchMetaObject.cs
+++ b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/IDispatchMetaObject.cs
@@ -11,6 +11,7 @@ namespace Microsoft.CSharp.RuntimeBinder.ComInterop
internal sealed class IDispatchMetaObject : ComFallbackMetaObject
{
private readonly IDispatchComObject _self;
+ private static readonly bool[] s_false = new bool[] { false };
[RequiresUnreferencedCode(Binder.TrimmerWarning)]
internal IDispatchMetaObject(Expression expression, IDispatchComObject self)
@@ -218,7 +219,7 @@ private DynamicMetaObject TryPropertyPut(SetMemberBinder binder, DynamicMetaObje
DynamicMetaObject result = new ComInvokeBinder(
new CallInfo(1),
new[] { value },
- new bool[] { false },
+ s_false,
restrictions,
Expression.Constant(method),
dispatch,
diff --git a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/VariantArray.cs b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/VariantArray.cs
index 71e466deb0acf0..37e34303feadff 100644
--- a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/VariantArray.cs
+++ b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/ComInterop/VariantArray.cs
@@ -49,6 +49,7 @@ internal static class VariantArray
// Don't need a dictionary for this, it will have very few elements
// (guaranteed less than 28, in practice 0-2)
private static readonly List s_generatedTypes = new List(0);
+ private static readonly string[] s_genericTName = new string[] { "T" };
[DynamicDependency(DynamicallyAccessedMemberTypes.PublicFields, typeof(VariantArray1))]
[DynamicDependency(DynamicallyAccessedMemberTypes.PublicFields, typeof(VariantArray2))]
@@ -100,7 +101,7 @@ private static Type CreateCustomType(int size)
{
TypeAttributes attrs = TypeAttributes.NotPublic | TypeAttributes.SequentialLayout;
TypeBuilder type = UnsafeMethods.DynamicModule.DefineType("VariantArray" + size, attrs, typeof(ValueType));
- GenericTypeParameterBuilder T = type.DefineGenericParameters(new string[] { "T" })[0];
+ GenericTypeParameterBuilder T = type.DefineGenericParameters(s_genericTName)[0];
for (int i = 0; i < size; i++)
{
type.DefineField("Element" + i, T, FieldAttributes.Public);
diff --git a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/Semantics/ExpressionBinder.cs b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/Semantics/ExpressionBinder.cs
index 8deb86a491c978..f2d34b30cb9a79 100644
--- a/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/Semantics/ExpressionBinder.cs
+++ b/src/libraries/Microsoft.CSharp/src/Microsoft/CSharp/RuntimeBinder/Semantics/ExpressionBinder.cs
@@ -1151,7 +1151,7 @@ private void AdjustCallArgumentsForParams(CType callingObjectType, CType type, M
CType arrayType = (ArrayType)TypeManager.SubstType(mp.Params[mp.Params.Count - 1], type, pTypeArgs);
// Use an EK_ARRINIT even in the empty case so empty param arrays in attributes work.
- ExprArrayInit arrayInit = ExprFactory.CreateArrayInit(arrayType, null, null, new[] { 0 });
+ ExprArrayInit arrayInit = ExprFactory.CreateArrayInit(arrayType, null, null, s_zero);
arrayInit.GeneratedForParamArray = true;
arrayInit.OptionalArguments = named.Value;
@@ -1229,7 +1229,7 @@ private void AdjustCallArgumentsForParams(CType callingObjectType, CType type, M
CType elementType = subArr.ElementType;
// Use an EK_ARRINIT even in the empty case so empty param arrays in attributes work.
- ExprArrayInit exprArrayInit = ExprFactory.CreateArrayInit(substitutedArrayType, null, null, new[] { 0 });
+ ExprArrayInit exprArrayInit = ExprFactory.CreateArrayInit(substitutedArrayType, null, null, s_zero);
exprArrayInit.GeneratedForParamArray = true;
if (it.AtEnd())
@@ -1618,6 +1618,8 @@ private static void CheckUnsafe(CType type)
private AggregateSymbol ContextForMemberLookup => Context.ContextForMemberLookup;
+ private static readonly int[] s_zero = new[] { 0 };
+
private static ExprWrap WrapShortLivedExpression(Expr expr) => ExprFactory.CreateWrap(expr);
private static ExprAssignment GenerateOptimizedAssignment(Expr op1, Expr op2) => ExprFactory.CreateAssignment(op1, op2);
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Emitter.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Emitter.cs
index f309f203486bf7..911a00d91fdbd5 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Emitter.cs
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Emitter.cs
@@ -176,7 +176,7 @@ private void EmitGetValueMethods()
if (_generationSpec.ShouldEmitMethods(BinderMethodSpecifier.GetValue_TypeOf_key_defaultValue))
{
EmitBlankLineIfRequired();
- _writer.WriteLine($"public static object? {Identifier.GetValue}(this {FullyQualifiedDisplayName.IConfiguration} {Identifier.configuration}, {FullyQualifiedDisplayName.Type} {Identifier.type}, string {Identifier.key}, object? {Identifier.defaultValue}) =>" +
+ _writer.WriteLine($"public static object? {Identifier.GetValue}(this {FullyQualifiedDisplayName.IConfiguration} {Identifier.configuration}, {FullyQualifiedDisplayName.Type} {Identifier.type}, string {Identifier.key}, object? {Identifier.defaultValue}) => " +
$"{expressionForGetValueCore}({Identifier.configuration}, {Identifier.type}, {Identifier.key}) ?? {Identifier.defaultValue};");
}
}
@@ -527,7 +527,7 @@ private void EmitHelperMethods()
_precedingBlockExists = true;
}
- if (_generationSpec.ShouldEmitMethods(BinderMethodSpecifier.RootMethodsWithConfigOptions))
+ if (_generationSpec.ShouldEmitMethods(BinderMethodSpecifier.MethodsThatAssessBinderOptions))
{
_writer.WriteBlankLine();
EmitGetBinderOptionsHelper();
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Parser.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Parser.cs
index 020c1104018669..4064232c5a7153 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Parser.cs
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Parser.cs
@@ -7,7 +7,6 @@
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
-using System.Runtime.CompilerServices;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Operations;
@@ -15,7 +14,7 @@ namespace Microsoft.Extensions.Configuration.Binder.SourceGeneration
{
public sealed partial class ConfigurationBindingGenerator
{
- private sealed class Parser
+ private sealed partial class Parser
{
private readonly SourceProductionContext _context;
private readonly KnownTypeSymbols _typeSymbols;
@@ -140,18 +139,19 @@ private void ProcessBindCall(BinderInvocationOperation binderOperation)
}
ITypeSymbol? type = ResolveType(objectArg.Value)?.WithNullableAnnotation(NullableAnnotation.None);
- INamedTypeSymbol? namedType;
- if ((namedType = type as INamedTypeSymbol) is null ||
- namedType.SpecialType == SpecialType.System_Object ||
- namedType.SpecialType == SpecialType.System_Void ||
- // Binding to root-level struct is a no-op.
- namedType.IsValueType)
+ if (!IsValidRootConfigType(overload, type, binderOperation.Location))
{
return;
}
- AddRootConfigType(methodGroup: BinderMethodSpecifier.Bind, overload, namedType, binderOperation.Location);
+ if (type.IsValueType)
+ {
+ _context.ReportDiagnostic(Diagnostic.Create(Diagnostics.ValueTypesInvalidForBind, binderOperation.Location, type));
+ return;
+ }
+
+ AddRootConfigType(methodGroup: BinderMethodSpecifier.Bind, overload, type, binderOperation.Location);
static ITypeSymbol? ResolveType(IOperation conversionOperation) =>
conversionOperation switch
@@ -180,7 +180,7 @@ private void ProcessGetCall(BinderInvocationOperation binderOperation)
}
BinderMethodSpecifier overload = BinderMethodSpecifier.None;
- INamedTypeSymbol? namedType;
+ ITypeSymbol? type;
if (targetMethod.IsGenericMethod)
{
@@ -189,7 +189,7 @@ private void ProcessGetCall(BinderInvocationOperation binderOperation)
return;
}
- namedType = targetMethod.TypeArguments[0].WithNullableAnnotation(NullableAnnotation.None) as INamedTypeSymbol;
+ type = targetMethod.TypeArguments[0].WithNullableAnnotation(NullableAnnotation.None);
if (paramLength is 1)
{
@@ -207,7 +207,7 @@ private void ProcessGetCall(BinderInvocationOperation binderOperation)
else
{
ITypeOfOperation? typeOfOperation = operation.Arguments[1].ChildOperations.FirstOrDefault() as ITypeOfOperation;
- namedType = typeOfOperation?.TypeOperand as INamedTypeSymbol;
+ type = typeOfOperation?.TypeOperand;
if (paramLength is 2)
{
@@ -219,15 +219,12 @@ private void ProcessGetCall(BinderInvocationOperation binderOperation)
}
}
- if (overload is BinderMethodSpecifier.None ||
- namedType is null ||
- namedType.SpecialType == SpecialType.System_Object ||
- namedType.SpecialType == SpecialType.System_Void)
+ if (!IsValidRootConfigType(overload, type, binderOperation.Location))
{
return;
}
- AddRootConfigType(methodGroup: BinderMethodSpecifier.Get, overload, namedType, binderOperation.Location);
+ AddRootConfigType(methodGroup: BinderMethodSpecifier.Get, overload, type, binderOperation.Location);
}
private void ProcessGetValueCall(BinderInvocationOperation binderOperation)
@@ -282,15 +279,12 @@ private void ProcessGetValueCall(BinderInvocationOperation binderOperation)
}
}
- if (overload is BinderMethodSpecifier.None ||
- type is null ||
- type.SpecialType == SpecialType.System_Object ||
- type.SpecialType == SpecialType.System_Void)
+ if (!IsValidRootConfigType(overload, type, binderOperation.Location))
{
return;
}
- ITypeSymbol effectiveType = IsNullable(type, out ITypeSymbol? underlyingType) ? underlyingType : type;
+ ITypeSymbol effectiveType = (IsNullable(type, out ITypeSymbol? underlyingType) ? underlyingType : type)!;
if (IsParsableFromString(effectiveType, out _))
{
AddRootConfigType(methodGroup: BinderMethodSpecifier.GetValue, overload, type, binderOperation.Location);
@@ -310,23 +304,38 @@ private void ProcessConfigureCall(BinderInvocationOperation binderOperation)
SymbolEqualityComparer.Default.Equals(_typeSymbols.IConfiguration, @params[1].Type))
{
ITypeSymbol? type = targetMethod.TypeArguments[0].WithNullableAnnotation(NullableAnnotation.None);
- if (type is not INamedTypeSymbol namedType ||
- namedType.SpecialType == SpecialType.System_Object)
+
+ if (!IsValidRootConfigType(BinderMethodSpecifier.Configure, type, binderOperation.Location))
{
return;
}
- AddRootConfigType(methodGroup: BinderMethodSpecifier.Configure, overload: BinderMethodSpecifier.Configure, namedType, binderOperation.Location);
+ AddRootConfigType(methodGroup: BinderMethodSpecifier.Configure, overload: BinderMethodSpecifier.Configure, type, binderOperation.Location);
}
}
- private void AddRootConfigType(BinderMethodSpecifier methodGroup, BinderMethodSpecifier overload, ITypeSymbol type, Location? location)
+ private bool IsValidRootConfigType(BinderMethodSpecifier overload, ITypeSymbol? type, Location? location)
{
- if (type is INamedTypeSymbol namedType && ContainsGenericParameters(namedType))
+ if (overload is BinderMethodSpecifier.None)
{
- return;
+ return false;
+ }
+
+ if (type is null ||
+ type.SpecialType is SpecialType.System_Object or SpecialType.System_Void ||
+ type.TypeKind is TypeKind.TypeParameter or TypeKind.Pointer or TypeKind.Error ||
+ type.IsRefLikeType ||
+ ContainsGenericParameters(type))
+ {
+ _context.ReportDiagnostic(Diagnostic.Create(Diagnostics.CouldNotDetermineTypeInfo, location));
+ return false;
}
+ return true;
+ }
+
+ private void AddRootConfigType(BinderMethodSpecifier methodGroup, BinderMethodSpecifier overload, ITypeSymbol type, Location? location)
+ {
if (GetOrCreateTypeSpec(type, location) is TypeSpec spec)
{
RegisterConfigType(spec, overload);
@@ -345,7 +354,7 @@ private void AddRootConfigType(BinderMethodSpecifier methodGroup, BinderMethodSp
if (IsNullable(type, out ITypeSymbol? underlyingType))
{
- spec = TryGetTypeSpec(underlyingType, ParserDiagnostics.NullableUnderlyingTypeNotSupported, out TypeSpec? underlyingTypeSpec)
+ spec = TryGetTypeSpec(underlyingType, Diagnostics.NullableUnderlyingTypeNotSupported, out TypeSpec? underlyingTypeSpec)
? new NullableSpec(type) { Location = location, UnderlyingType = underlyingTypeSpec }
: null;
}
@@ -386,7 +395,7 @@ private void AddRootConfigType(BinderMethodSpecifier methodGroup, BinderMethodSp
if (spec is null)
{
- ReportUnsupportedType(type, ParserDiagnostics.TypeNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.TypeNotSupported, location);
return null;
}
@@ -562,7 +571,7 @@ private bool TryGetTypeSpec(ITypeSymbol type, DiagnosticDescriptor descriptor, o
private EnumerableSpec? CreateArraySpec(IArrayTypeSymbol arrayType, Location? location)
{
- if (!TryGetTypeSpec(arrayType.ElementType, ParserDiagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
+ if (!TryGetTypeSpec(arrayType.ElementType, Diagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
{
return null;
}
@@ -594,7 +603,7 @@ private bool IsSupportedArrayType(ITypeSymbol type, Location? location)
if (arrayType.Rank > 1)
{
- ReportUnsupportedType(arrayType, ParserDiagnostics.MultiDimArraysNotSupported, location);
+ ReportUnsupportedType(arrayType, Diagnostics.MultiDimArraysNotSupported, location);
return false;
}
@@ -624,15 +633,15 @@ private bool IsSupportedArrayType(ITypeSymbol type, Location? location)
private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? location, ITypeSymbol keyType, ITypeSymbol elementType)
{
- if (!TryGetTypeSpec(keyType, ParserDiagnostics.DictionaryKeyNotSupported, out TypeSpec keySpec) ||
- !TryGetTypeSpec(elementType, ParserDiagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
+ if (!TryGetTypeSpec(keyType, Diagnostics.DictionaryKeyNotSupported, out TypeSpec keySpec) ||
+ !TryGetTypeSpec(elementType, Diagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
{
return null;
}
if (keySpec.SpecKind != TypeSpecKind.ParsableFromString)
{
- ReportUnsupportedType(type, ParserDiagnostics.DictionaryKeyNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.DictionaryKeyNotSupported, location);
return null;
}
@@ -657,7 +666,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
}
else
{
- ReportUnsupportedType(type, ParserDiagnostics.CollectionNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.CollectionNotSupported, location);
return null;
}
}
@@ -678,7 +687,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
}
else
{
- ReportUnsupportedType(type, ParserDiagnostics.CollectionNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.CollectionNotSupported, location);
return null;
}
@@ -702,7 +711,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
private EnumerableSpec? CreateEnumerableSpec(INamedTypeSymbol type, Location? location)
{
if (!TryGetElementType(type, out ITypeSymbol? elementType) ||
- !TryGetTypeSpec(elementType, ParserDiagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
+ !TryGetTypeSpec(elementType, Diagnostics.ElementTypeNotSupported, out TypeSpec elementSpec))
{
return null;
}
@@ -727,7 +736,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
}
else
{
- ReportUnsupportedType(type, ParserDiagnostics.CollectionNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.CollectionNotSupported, location);
return null;
}
}
@@ -767,7 +776,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
}
else
{
- ReportUnsupportedType(type, ParserDiagnostics.CollectionNotSupported, location);
+ ReportUnsupportedType(type, Diagnostics.CollectionNotSupported, location);
return null;
}
@@ -829,8 +838,8 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
bool hasPublicParameterlessCtor = type.IsValueType || parameterlessCtor is not null;
if (!hasPublicParameterlessCtor && hasMultipleParameterizedCtors)
{
- diagnosticDescriptor = ParserDiagnostics.MultipleParameterizedConstructors;
- objectSpec.InitExceptionMessage = string.Format(ExceptionMessages.MultipleParameterizedConstructors, typeName);
+ diagnosticDescriptor = Diagnostics.MultipleParameterizedConstructors;
+ objectSpec.InitExceptionMessage = string.Format(Emitter.ExceptionMessages.MultipleParameterizedConstructors, typeName);
}
ctor = type.IsValueType
@@ -843,8 +852,8 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
if (ctor is null)
{
- diagnosticDescriptor = ParserDiagnostics.MissingPublicInstanceConstructor;
- objectSpec.InitExceptionMessage = string.Format(ExceptionMessages.MissingPublicInstanceConstructor, typeName);
+ diagnosticDescriptor = Diagnostics.MissingPublicInstanceConstructor;
+ objectSpec.InitExceptionMessage = string.Format(Emitter.ExceptionMessages.MissingPublicInstanceConstructor, typeName);
}
if (diagnosticDescriptor is not null)
@@ -869,7 +878,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
TypeSpec? propertyTypeSpec = GetOrCreateTypeSpec(property.Type);
if (propertyTypeSpec is null)
{
- _context.ReportDiagnostic(Diagnostic.Create(ParserDiagnostics.PropertyNotSupported, location, new string[] { propertyName, type.ToDisplayString() }));
+ _context.ReportDiagnostic(Diagnostic.Create(Diagnostics.PropertyNotSupported, location, new string[] { propertyName, type.ToDisplayString() }));
}
else
{
@@ -914,7 +923,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
if (invalidParameters.Count > 0)
{
- objectSpec.InitExceptionMessage = string.Format(ExceptionMessages.CannotBindToConstructorParameter, typeName, FormatParams(invalidParameters));
+ objectSpec.InitExceptionMessage = string.Format(Emitter.ExceptionMessages.CannotBindToConstructorParameter, typeName, FormatParams(invalidParameters));
}
else if (missingParameters.Count > 0)
{
@@ -924,7 +933,7 @@ private DictionarySpec CreateDictionarySpec(INamedTypeSymbol type, Location? loc
}
else
{
- objectSpec.InitExceptionMessage = string.Format(ExceptionMessages.ConstructorParametersDoNotMatchProperties, typeName, FormatParams(missingParameters));
+ objectSpec.InitExceptionMessage = string.Format(Emitter.ExceptionMessages.ConstructorParametersDoNotMatchProperties, typeName, FormatParams(missingParameters));
}
}
@@ -1021,16 +1030,17 @@ private static bool IsInterfaceMatch(INamedTypeSymbol type, INamedTypeSymbol @in
return SymbolEqualityComparer.Default.Equals(type, @interface);
}
- public static bool ContainsGenericParameters(INamedTypeSymbol type)
+ public static bool ContainsGenericParameters(ITypeSymbol type)
{
- if (!type.IsGenericType)
+ if (type is not INamedTypeSymbol { IsGenericType: true } genericType)
{
return false;
}
- foreach (ITypeSymbol typeArg in type.TypeArguments)
+ foreach (ITypeSymbol typeArg in genericType.TypeArguments)
{
- if (typeArg.TypeKind == TypeKind.TypeParameter)
+ if (typeArg.TypeKind is TypeKind.TypeParameter or TypeKind.Error ||
+ ContainsGenericParameters(typeArg))
{
return true;
}
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.cs
index 37b228343d7a63..cc0787ebf56d4e 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.cs
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.cs
@@ -54,7 +54,7 @@ private static void Execute(CompilationData compilationData, ImmutableArray
+ new DiagnosticDescriptor(
+ id: "SYSLIB1100",
+ title: new LocalizableResourceString(nameof(SR.TypeNotSupportedTitle), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
+ messageFormat: new LocalizableResourceString(nameofLocalizableMessageFormat, SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
+ category: ProjectName,
+ defaultSeverity: DiagnosticSeverity.Warning,
+ isEnabledByDefault: true);
+ }
+ }
+ }
+}
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Helpers/ParserDiagnostics.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Helpers/ParserDiagnostics.cs
deleted file mode 100644
index be2631422fe391..00000000000000
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Helpers/ParserDiagnostics.cs
+++ /dev/null
@@ -1,45 +0,0 @@
-// Licensed to the .NET Foundation under one or more agreements.
-// The .NET Foundation licenses this file to you under the MIT license.
-
-using System;
-using Microsoft.CodeAnalysis;
-
-namespace Microsoft.Extensions.Configuration.Binder.SourceGeneration
-{
- internal static class ParserDiagnostics
- {
- public static DiagnosticDescriptor TypeNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.TypeNotSupported));
- public static DiagnosticDescriptor MissingPublicInstanceConstructor { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.MissingPublicInstanceConstructor));
- public static DiagnosticDescriptor CollectionNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.CollectionNotSupported));
- public static DiagnosticDescriptor DictionaryKeyNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.DictionaryKeyNotSupported));
- public static DiagnosticDescriptor ElementTypeNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.ElementTypeNotSupported));
- public static DiagnosticDescriptor MultipleParameterizedConstructors { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.MultipleParameterizedConstructors));
- public static DiagnosticDescriptor MultiDimArraysNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.MultiDimArraysNotSupported));
- public static DiagnosticDescriptor NullableUnderlyingTypeNotSupported { get; } = CreateTypeNotSupportedDescriptor(nameof(SR.NullableUnderlyingTypeNotSupported));
-
- public static DiagnosticDescriptor PropertyNotSupported { get; } = new DiagnosticDescriptor(
- id: "SYSLIB1101",
- title: new LocalizableResourceString(nameof(SR.PropertyNotSupportedTitle), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- messageFormat: new LocalizableResourceString(nameof(SR.PropertyNotSupportedMessageFormat), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- category: ConfigurationBindingGenerator.ProjectName,
- defaultSeverity: DiagnosticSeverity.Warning,
- isEnabledByDefault: true);
-
- public static DiagnosticDescriptor LanguageVersionNotSupported { get; } = new DiagnosticDescriptor(
- id: "SYSLIB1102",
- title: new LocalizableResourceString(nameof(SR.LanguageVersionIsNotSupportedTitle), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- messageFormat: new LocalizableResourceString(nameof(SR.Language_VersionIsNotSupportedMessageFormat), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- category: ConfigurationBindingGenerator.ProjectName,
- defaultSeverity: DiagnosticSeverity.Error,
- isEnabledByDefault: true);
-
- private static DiagnosticDescriptor CreateTypeNotSupportedDescriptor(string nameofLocalizableMessageFormat) =>
- new DiagnosticDescriptor(
- id: "SYSLIB1100",
- title: new LocalizableResourceString(nameof(SR.TypeNotSupportedTitle), SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- messageFormat: new LocalizableResourceString(nameofLocalizableMessageFormat, SR.ResourceManager, typeof(FxResources.Microsoft.Extensions.Configuration.Binder.SourceGeneration.SR)),
- category: ConfigurationBindingGenerator.ProjectName,
- defaultSeverity: DiagnosticSeverity.Warning,
- isEnabledByDefault: true);
- }
-}
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Microsoft.Extensions.Configuration.Binder.SourceGeneration.csproj b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Microsoft.Extensions.Configuration.Binder.SourceGeneration.csproj
index c3929ec6ba6662..65150ea38a4111 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Microsoft.Extensions.Configuration.Binder.SourceGeneration.csproj
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Microsoft.Extensions.Configuration.Binder.SourceGeneration.csproj
@@ -24,16 +24,16 @@
+
-
-
-
+
+
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/BinderMethodSpecifier.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/BinderMethodSpecifier.cs
index 33b2859a867bfa..dafbd0bee3c019 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/BinderMethodSpecifier.cs
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/BinderMethodSpecifier.cs
@@ -80,6 +80,7 @@ internal enum BinderMethodSpecifier
Bind = Bind_instance | Bind_instance_BinderOptions | Bind_key_instance,
Get = Get_T | Get_T_BinderOptions | Get_TypeOf | Get_TypeOf_BinderOptions,
GetValue = GetValue_T_key | GetValue_T_key_defaultValue | GetValue_TypeOf_key | GetValue_TypeOf_key_defaultValue,
- RootMethodsWithConfigOptions = Bind_instance_BinderOptions | Get_T_BinderOptions | Get_TypeOf_BinderOptions,
+
+ MethodsThatAssessBinderOptions = Bind_instance_BinderOptions | Get
}
}
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Helpers/KnownTypeSymbols.cs b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/KnownTypeSymbols.cs
similarity index 100%
rename from src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Helpers/KnownTypeSymbols.cs
rename to src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Model/KnownTypeSymbols.cs
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/Strings.resx b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/Strings.resx
index 9bf38777bde79a..301913987d7c7e 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/Strings.resx
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/Strings.resx
@@ -120,6 +120,12 @@
The collection type is not supported: '{0}'.
+
+ Binding logic was not generated for a binder call. Unsupported input patterns include generic calls and passing boxed objects.
+
+
+ The target type for a binder call could not be determined
+
The dictionary key type is not supported: '{0}'.
@@ -156,4 +162,10 @@
Did not generate binding logic for a type
+
+ Binding logic was not generated for a binder call with target type '{0}'. Value types are invalid inputs to configuration 'Bind' methods.
+
+
+ Value types are invalid inputs to configuration 'Bind' methods
+
\ No newline at end of file
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.cs.xlf b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.cs.xlf
index 67cc2e5f23ea48..e248c54626865f 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.cs.xlf
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.cs.xlf
@@ -7,6 +7,16 @@
Typ kolekce se nepodporuje: „{0}“.
+
+ Binding logic was not generated for a binder call. Unsupported input patterns include generic calls and passing boxed objects.
+ Pro volání vazače se nevygenerovala logika vazby. Nepodporované vstupní vzory zahrnují obecná volání a předávání zabalených objektů.
+
+
+
+ The target type for a binder call could not be determined
+ Cílový typ volání vazače nelze určit
+
+ The dictionary key type is not supported: '{0}'.Typ klíče slovníku se nepodporuje:„{0}“.
@@ -29,7 +39,7 @@
Cannot create instance of type '{0}' because it is missing a public instance constructor.
- Cannot create instance of type '{0}' because it is missing a public instance constructor.
+ Nelze vytvořit instanci typu „{0}“, protože chybí konstruktor veřejné instance.
@@ -39,7 +49,7 @@
Cannot create instance of type '{0}' because it has multiple public parameterized constructors.
- Cannot create instance of type '{0}' because it has multiple public parameterized constructors.
+ Nelze vytvořit instanci typu „{0}“, protože má více veřejných parametrizovaných konstruktorů.
@@ -67,6 +77,16 @@
Negenerovala se logika vazby pro typ
+
+ Binding logic was not generated for a binder call with target type '{0}'. Value types are invalid inputs to configuration 'Bind' methods.
+ Logika vazby nebyla vygenerována pro volání pořadače s cílovým typem „{0}“. Typy hodnot jsou neplatné vstupy pro konfigurační metody Bind.
+
+
+
+ Value types are invalid inputs to configuration 'Bind' methods
+ Typy hodnot nejsou platné vstupy pro konfigurační metody „Bind“.
+
+
\ No newline at end of file
diff --git a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.de.xlf b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.de.xlf
index 9c6839a3fd2fac..1fa847592bd02e 100644
--- a/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.de.xlf
+++ b/src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Resources/xlf/Strings.de.xlf
@@ -7,6 +7,16 @@