adding documentation for query null semantics

entity-framework/core/querying/null-semantics.md

@@ -0,0 +1,131 @@
+---
+title: Query null semantics in EF Core
+description: Information on how Entity Framework Core handles null comparisons in queries 
+author: maumar
+ms.date: 11/11/2020
+uid: core/querying/null-semantics
+---
+# Query null semantics
+
+## Introduction
+
+SQL databases operate on 3-valued logic (`true`, `false`, `null`) when performing comparisons, as opposed to boolean logic of C#. When translating LINQ queries to SQL, EF Core tries to compensate for the difference by introducing additional null checks for some elements of the query.
+To illustrate this lets define the following entity:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/NullSemanticsEntity.cs#Entity)]
+
+and issue several queries:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/Program.cs#BasicExamples)]
+
+First two queries produce simple comparison. In case of the first query both columns are non-nullable so null checks are not needed. In case of the second query `NullableInt` could contain null, however `Id` is non-nullable. Comparing `null` to non-null yields `null` as a result, which would be filtered out by `WHERE` operation. No additional terms need to be added either.
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [e].[Id] = [e].[Int]
+
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [e].[Id] = [e].[NullableInt]
+```
+
+Third query introduces the null check. When `NullableInt` is `null` the comparison `Id <> NullableInt` yields `null`, which would be filtered out by `WHERE` operation. However from the boolean logic perspective this case should be returned as part of the result. Hence EF Core adds the necessary check to ensure that.
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE ([e].[Id] <> [e].[NullableInt]) OR [e].[NullableInt] IS NULL
+```
+
+Queries four and five show the pattern when both columns are nullable. It's worth noting that `<>` operation produces significantly more complicated (and slower) query than equality operation.
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE ([e].[String1] = [e].[String2]) OR ([e].[String1] IS NULL AND [e].[String2] IS NULL)
+
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE (([e].[String1] <> [e].[String2]) OR ([e].[String1] IS NULL OR [e].[String2] IS NULL)) AND ([e].[String1] IS NOT NULL OR [e].[String2] IS NOT NULL)
+```
+
+## Null semantics in functions
+
+Many functions in SQL can only return null result if some of their arguments are null. EF Core takes advantage of this to produce more efficient queries.
+The query below illustrates the optimization:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/Program.cs#Functions)]
+
+The generated SQL is as follows (we don't need to evaluate the `SUBSTRING` function):
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [e].[String1] IS NULL OR [e].[String2] IS NULL
+```
+
+This optimization can also be used for user defined functions. This can be done by adding `PropagatesNullability()` call to relevant function parameters model configuration.
+To illustrate this, define two user functions inside the `DbContext`:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/NullSemanticsContext.cs#UdfBody)]
+
+The model configuration (inside `OnModelCreating` method) is as follows:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/NullSemanticsContext.cs#UdfModelConfiguration)]
+
+First function is configured in a standard way and the second function is configured to take advantage of the nullability propagation optimization.
+
+When issuing the following queries:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/Program.cs#UdfExamples)]
+
+We get this SQL:
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [dbo].[ConcatStrings]([e].[String1], [e].[String2]) IS NOT NULL
+
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [e].[String1] IS NOT NULL AND [e].[String2] IS NOT NULL
+```
+
+Just like with built-in `Substring` function, the second query doesn't need to evaluate the function itself to test it's nullability.
+
+> [!NOTE]
+> This optimization should only be used if the function can only return `null` becuase it's parameters are `null`.
+
+## Optimizations and considerations
+
+- Comparing non-nullable columns is simpler and faster than comparing nullable columns. Consider marking columns as non-nullable wherever it is possible.
+
+- `Equal` comparison is simpler and faster than `NotEqual`, because query doesn't need to distinguish between `null` and `false` result. Consider using equality comparison whenever it is possible.
+
+> [!NOTE]
+> Wrapping `Equal` comparison around `Not` is effectively `NotEqual`. `Equal` comparison is only faster/simpler if it is not negated.
+
+- In some cases it is possible to simplify a complex comparison by filtering out `null` values from a column explicitly - e.g. when no `null` values are present or these values are not relevant in the result. Consider the following example:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/Program.cs#ManualOptimization)]
+
+These queries produce the following SQL:
+
+```sql
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE ((([e].[String1] <> [e].[String1]) OR [e].[String1] IS NULL) AND [e].[String1] IS NOT NULL) OR ((CAST(LEN([e].[String1]) AS int) = CAST(LEN([e].[String1]) AS int)) OR [e].[String1] IS NULL)
+
+SELECT [e].[Id], [e].[Int], [e].[NullableInt], [e].[String1], [e].[String2]
+FROM [Entities] AS [e]
+WHERE [e].[String1] IS NOT NULL AND (CAST(LEN([e].[String1]) AS int) = CAST(LEN([e].[String1]) AS int))
+```
+
+In the second query, `null` results are filtered out from `String1` column explicitly. EF Core can safely treat the `String1` column as non-nullable when performing comparison, which results is simpler query.
+
+## Using relational null semantics
+
+It is possible to disable the null comparison compensation and use relational null semantics directly. This can be done by calling `UseRelationalNulls(true)` method on the options builder inside `OnConfiguring` method:
+
+[!code-csharp[Main](../../../samples/core/Querying/NullSemantics/NullSemanticsContext.cs#UseRelationalNulls)]

entity-framework/core/querying/single-split-queries.md

@@ -5,7 +5,7 @@ author: smitpatel
 ms.date: 10/03/2019
 uid: core/querying/single-split-queries
 ---
-# Single vs. split queries
+# Split queries
 
 ## Single queries
 

entity-framework/toc.yml

@@ -193,6 +193,8 @@
         href: core/querying/filters.md
       - name: Query tags
         href: core/querying/tags.md
+      - name: Query null semantics
+        href: core/querying/null-semantics.md
       - name: How queries work
         href: core/querying/how-query-works.md
 

samples/core/Querying/NullSemantics/NullSemantics.csproj

@@ -0,0 +1,12 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>netcoreapp3.1</TargetFramework>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="5.0.0" />
+  </ItemGroup>
+
+</Project>

samples/core/Querying/NullSemantics/NullSemanticsContext.cs

@@ -0,0 +1,56 @@
+using Microsoft.EntityFrameworkCore;
+using Microsoft.EntityFrameworkCore.Infrastructure;
+
+namespace NullSemantics
+{
+    public class NullSemanticsContext : DbContext
+    {
+		public DbSet<NullSemanticsEntity> Entities { get; set; }
+
+		protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
+		{
+            var relationalNulls = false;
+            if (relationalNulls)
+            {
+                #region UseRelationalNulls
+                new SqlServerDbContextOptionsBuilder(optionsBuilder).UseRelationalNulls(true);
+                #endregion
+            }
+
+            optionsBuilder.UseSqlServer(@"Server=(localdb)\mssqllocaldb;Database=NullSemanticsSample;Trusted_Connection=True;MultipleActiveResultSets=true");
+		}
+
+        #region UdfBody
+        public string ConcatStrings(string prm1, string prm2)
+            => throw new System.InvalidOperationException();
+
+        public string ConcatStringsOptimized(string prm1, string prm2)
+            => throw new System.InvalidOperationException();
+        #endregion
+
+        protected override void OnModelCreating(ModelBuilder modelBuilder)
+        {
+            #region UdfModelConfiguration
+            modelBuilder
+                .HasDbFunction(typeof(NullSemanticsContext).GetMethod(nameof(ConcatStrings), new[] { typeof(string), typeof(string) }))
+                .HasName("ConcatStrings");
+
+            modelBuilder.HasDbFunction(
+                typeof(NullSemanticsContext).GetMethod(nameof(ConcatStringsOptimized), new[] { typeof(string), typeof(string) }),
+                b =>
+                {
+                    b.HasName("ConcatStrings");
+                    b.HasParameter("prm1").PropagatesNullability();
+                    b.HasParameter("prm2").PropagatesNullability();
+                });
+            #endregion
+
+            modelBuilder.Entity<NullSemanticsEntity>().HasData(
+                new NullSemanticsEntity { Id = 1, Int = 1, NullableInt = 1, String1 = "A", String2 = "A" },
+                new NullSemanticsEntity { Id = 2, Int = 2, NullableInt = 2, String1 = "A", String2 = "B" },
+                new NullSemanticsEntity { Id = 3, Int = 2, NullableInt = null, String1 = null, String2 = "A" },
+                new NullSemanticsEntity { Id = 4, Int = 2, NullableInt = null, String1 = "B", String2 = null },
+                new NullSemanticsEntity { Id = 5, Int = 1, NullableInt = 3, String1 = null, String2 = null });
+        }
+    }
+}

samples/core/Querying/NullSemantics/NullSemanticsEntity.cs

@@ -0,0 +1,13 @@
+namespace NullSemantics
+{
+    #region Entity
+    public class NullSemanticsEntity
+    {
+        public int Id { get; set; }
+        public int Int { get; set; }
+        public int? NullableInt { get; set; }
+        public string String1 { get; set; }
+        public string String2 { get; set; }
+    }
+    #endregion
+}

samples/core/Querying/NullSemantics/Program.cs

@@ -0,0 +1,80 @@
+using Microsoft.EntityFrameworkCore;
+using System;
+using System.Linq;
+using System.Linq.Expressions;
+
+namespace NullSemantics
+{
+    class Program
+    {
+        static void Main(string[] args)
+        {
+            using var context = new NullSemanticsContext();
+            context.Database.EnsureDeleted();
+            context.Database.EnsureCreated();
+
+            #region FunctionSqlRaw
+            context.Database.ExecuteSqlRaw(
+                @"create function [dbo].[ConcatStrings] (@prm1 nvarchar(max), @prm2 nvarchar(max))
+                    returns nvarchar(max)
+                    as
+                    begin
+                        return @prm1 + @prm2;
+                    end");
+            #endregion
+
+            BasicExamples();
+            Functions();
+            ManulaOptimization();
+        }
+
+        static void BasicExamples()
+        {
+            using var context = new NullSemanticsContext();
+            #region BasicExamples
+            var query1 = context.Entities.Where(e => e.Id == e.Int);
+            var query2 = context.Entities.Where(e => e.Id == e.NullableInt);
+            var query3 = context.Entities.Where(e => e.Id != e.NullableInt);
+            var query4 = context.Entities.Where(e => e.String1 == e.String2);
+            var query5 = context.Entities.Where(e => e.String1 != e.String2);
+            #endregion
+
+            var result1 = query1.ToList();
+            var result2 = query2.ToList();
+            var result3 = query3.ToList();
+            var result4 = query4.ToList();
+            var result5 = query5.ToList();
+        }
+
+        static void Functions()
+        {
+            using var context = new NullSemanticsContext();
+
+            #region Functions
+            var query = context.Entities.Where(e => e.String1.Substring(0, e.String2.Length) == null);
+            #endregion
+
+            #region UdfExamples
+            var query1 = context.Entities.Where(e => context.ConcatStrings(e.String1, e.String2) != null);
+            var query2 = context.Entities.Where(e => context.ConcatStringsOptimized(e.String1, e.String2) != null);
+            #endregion
+
+            var result = query.ToList();
+            var result1 = query1.ToList();
+            var result2 = query2.ToList();
+        }
+
+        static void ManulaOptimization()
+        {
+            using var context = new NullSemanticsContext();
+
+            #region ManualOptimization
+            var query1 = context.Entities.Where(e => e.String1 != e.String1 || e.String1.Length == e.String1.Length);
+            var query2 = context.Entities.Where(e => e.String1 != null && (e.String1 != e.String1 || e.String1.Length == e.String1.Length));
+            #endregion
+
+            var result1 = query1.ToList();
+            var result2 = query2.ToList();
+        }
+    }
+}

samples/core/Samples.sln

@@ -113,20 +113,22 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebApplication1.Migrations"
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebApplication1", "Schemas\ThreeProjectMigrations\WebApplication1\WebApplication1.csproj", "{A15F08F8-966D-4A38-A9FF-F0B3FB3335BE}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ConnectionInterception", "Miscellaneous\ConnectionInterception\ConnectionInterception.csproj", "{1CE06110-E69A-44BD-ACBE-D7F96B02288B}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ConnectionInterception", "Miscellaneous\ConnectionInterception\ConnectionInterception.csproj", "{1CE06110-E69A-44BD-ACBE-D7F96B02288B}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SaveChangesInterception", "Miscellaneous\SaveChangesInterception\SaveChangesInterception.csproj", "{8F5C3F05-BD4B-4ED3-BCE4-45CA738C2AD3}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SaveChangesInterception", "Miscellaneous\SaveChangesInterception\SaveChangesInterception.csproj", "{8F5C3F05-BD4B-4ED3-BCE4-45CA738C2AD3}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CachingInterception", "Miscellaneous\CachingInterception\CachingInterception.csproj", "{DCF56A96-123F-48A6-A848-2BAE19B58C79}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CachingInterception", "Miscellaneous\CachingInterception\CachingInterception.csproj", "{DCF56A96-123F-48A6-A848-2BAE19B58C79}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CommandInterception", "Miscellaneous\CommandInterception\CommandInterception.csproj", "{97BECA9A-A72B-4C77-ADDB-DCC84966570F}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CommandInterception", "Miscellaneous\CommandInterception\CommandInterception.csproj", "{97BECA9A-A72B-4C77-ADDB-DCC84966570F}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Events", "Miscellaneous\Events\Events.csproj", "{8138D0F5-D1A7-4908-8A52-08196FF46B69}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Events", "Miscellaneous\Events\Events.csproj", "{8138D0F5-D1A7-4908-8A52-08196FF46B69}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DiagnosticListeners", "Miscellaneous\DiagnosticListeners\DiagnosticListeners.csproj", "{AF719729-AED8-4DEB-B895-61D8EBB50A01}"
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DiagnosticListeners", "Miscellaneous\DiagnosticListeners\DiagnosticListeners.csproj", "{AF719729-AED8-4DEB-B895-61D8EBB50A01}"
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ConfiguringDbContext", "Miscellaneous\ConfiguringDbContext\ConfiguringDbContext.csproj", "{73503DF2-CD85-4710-BE94-B83B87054709}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "NullSemantics", "Querying\NullSemantics\NullSemantics.csproj", "{8300FAFD-F840-4A40-BA96-4F857F1F6BF2}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -345,6 +347,10 @@ Global
 		{73503DF2-CD85-4710-BE94-B83B87054709}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{73503DF2-CD85-4710-BE94-B83B87054709}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{73503DF2-CD85-4710-BE94-B83B87054709}.Release|Any CPU.Build.0 = Release|Any CPU
+		{8300FAFD-F840-4A40-BA96-4F857F1F6BF2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{8300FAFD-F840-4A40-BA96-4F857F1F6BF2}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{8300FAFD-F840-4A40-BA96-4F857F1F6BF2}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{8300FAFD-F840-4A40-BA96-4F857F1F6BF2}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -399,6 +405,7 @@ Global
 		{8138D0F5-D1A7-4908-8A52-08196FF46B69} = {85AFD7F1-6943-40FE-B8EC-AA9DBB42CCA6}
 		{AF719729-AED8-4DEB-B895-61D8EBB50A01} = {85AFD7F1-6943-40FE-B8EC-AA9DBB42CCA6}
 		{73503DF2-CD85-4710-BE94-B83B87054709} = {85AFD7F1-6943-40FE-B8EC-AA9DBB42CCA6}
+		{8300FAFD-F840-4A40-BA96-4F857F1F6BF2} = {1AD64707-0BE0-48B0-A803-916FF96DCB4F}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {20C98D35-54EF-46A6-8F3B-1855C1AE4F70}