Source generator dependency unable to be resolved

[Turnerj]

Version Used:
MSBuild: 16.9.0.11203
.NET: 5.0.201
CSC: 3.900.21.12328 (comes with .NET 5.0.201 and used by dotnet build), 3.900.21.16010 (comes with Visual Studio and used at least by msbuild but probably VS too) - All the commits between these two versions
Visual Studio: 16.9.2

Steps to Reproduce / Actual Behaviour:
Sample project to recreate issue: https://github.com/Turnerj/SourceGeneratorDependencyTest

Using .NET CLI (like we use on a CI)

1. Clean solution
2. Build solution with command dotnet build
3. Solution builds 🎉

Using MSBuild

1. Clean solution
2. Build solution with command msbuild
3. Solution fails (see generator warning) 😕

Using Visual Studio

1. Clean solution
2. Build solution via Visual Studio
3. Solution fails (see generator warning) 😞

Generator Warning

CSC : warning CS8784: Generator 'CustomSourceGenerator' failed to initialize. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'FileNotFoundException' with message 'Could not load file or assembly
'System.Text.Encodings.Web, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.'

Expected Behavior:

Each method builds the project.

Notes:

The sample project is a minimum example of what we use in a full project: https://github.com/RehanSaeed/Schema.NET/
I'm following the guidance listed in this discussion about how to mark the package reference: #47517 (comment)
This is also seen on the Roslyn SDK C# samples: https://github.com/dotnet/roslyn-sdk/blob/0313c80ed950ac4f4eef11bb2e1c6d1009b328c4/samples/CSharp/SourceGenerators/SourceGeneratorSamples/SourceGeneratorSamples.csproj#L13-L30
I've added the transient dependency (in my case System.Text.Encodings.Web) as per this comment in the same discussion: #47517 (reply in thread)

If I use the version 5.0.0 of System.Text.Encodings.Web, it does build with all methods. I've tried to debug binlogs of this but I haven't found anything explaining why it resolves correctly via dotnet build and not via Visual Studio (or msbuild).

I don't see why I can't specifically use version 5.0.1 of System.Text.Encodings.Web, hopefully you can work out what is going on!

[jmarolf]

@Turnerj you must package all dependencies into your nuget package. System.Text.Encodings.Web for netstandard2.0 has a dependency on both System.Buffers and System.Memory (which in turn has a dependency on System.Numerics.Vectors). All of these types will be found implicitly on .NET Core and .NET 5 runtimes (i.e. dotnet build) but none of them exist on .NET Framework (both msbuild and Visual Studio).

The full set up dependencies you need to deploy are:

• System.Text.Encodings.Web
  • System.Buffers 4.5.1
  • System.Memory 4.5.4
    • System.Numerics.Vectors, 4.4.0
• System.Text.Json
  • Microsoft.Bcl.AsyncInterfaces, 5.0.0
  • System.Buffers, 4.5.1
  • System.Memory, 4.5.4
    • System.Numerics.Vectors, 4.4.0
  • System.Numerics.Vectors, 4.5.0
  • System.Runtime.CompilerServices.Unsafe, 5.0.0
  • System.Text.Encodings.Web, 5.0.0
  • System.Threading.Tasks.Extensions, 4.5.4

[Turnerj]

Thanks for the information @jmarolf - it sucks but is good to know why this is happening.

Is there any plans to changing this behaviour for dependencies used inside source generators? While source generators allow a lot of flexibility, some advanced functionality (like what we do in Schema.NET) is difficult to implement without having dependencies.

[Turnerj]

Edit: Ignore below, I re-read the actual warning message and missed the " or one of its dependencies" which would explain the message specifically.

---

Actually I have an additional question, while what you mentioned makes sense, why would the error message be specifically that it couldn't find System.Text.Encodings.Web instead of not finding one of those dependencies? Additionally how does that apply when going from version 5.0.0 for System.Text.Encodings.Web to 5.0.1?

Like with what you described, I would expect to see an warning saying it couldn't find one of those dependencies. Saying that it couldn't find System.Text.Encodings.Web because I've moved from 5.0.0 to 5.0.1, would seem like that its dependencies then did exist in .NET Framework (MSBuild/Visual Studio) - if that makes sense.

[jaredpar]

Is there any plans to changing this behaviour for dependencies used inside source generators? While source generators allow a lot of flexibility, some advanced functionality (like what we do in Schema.NET) is difficult to implement without having dependencies.

There isn't a restriction here on having dependencies just a requirement that source generators deploy their dependencies with them.

Additionally how does that apply when going from version 5.0.0 for System.Text.Encodings.Web to 5.0.1?

That minor release added a lot more dependencies and that is reflecting in this issue you are hitting.

Actually I have an additional question, while what you mentioned makes sense, why would the error message be specifically that it couldn't find System.Text.Encodings.Web instead of not finding one of those dependencies?

This is a case where we are reflecting the error message that the .NET runtime provides when it attempts to load your DLL. Essentially we ask the runtime to load your generator and the runtime is providing the exception and the diagnostic here about the missing dependencies. It's not an error that we are directly providing.

Closing as this isn't an issue in the compiler. Happy to continue chatting here to help you work through this though and make sure your deployment is successful.

@chsienki

[Turnerj]

There isn't a restriction here on having dependencies just a requirement that source generators deploy their dependencies with them.

I guess my main issue here is that where one would normally just install the specific dependency they need, for a source generator I need to specify the entire tree of dependencies in my csproj file manually. While I'm sure a mountain of work has gone into making source generators work, this (and debugging source generators) make it feel somewhat incomplete.

While @jmarolf went out of their way to list all of the dependencies in my case (thank you!), basically I'm wondering why some combination of NuGet restore/compilation process isn't doing this automatically. NuGet restore already has to work out all the dependencies anyway however this current process of manually adding everything is quite intensive, especially for when package updates happen as I would need to re-check to see if any new dependencies are added.

So would there be any plans on changing this behaviour and having the compiler (or some other part of the build process) automatically handle this?

[Turnerj]

I've been looking into ways that a user can avoid manually specifying transient dependencies for a source generator and have something working with the following:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>
  
  <ItemGroup Label="Package References">
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" />
    <PackageReference Include="System.Text.Json" Version="5.0.1" PrivateAssets="all" />
  </ItemGroup>

  <PropertyGroup>
    <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
  </PropertyGroup>

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="@(ResolvedCompileFileDefinitions)" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>
</Project>

This effectively adds:

~\.nuget\packages\microsoft.bcl.asyncinterfaces\5.0.0\lib\netstandard2.0\Microsoft.Bcl.AsyncInterfaces.dll
~\.nuget\packages\microsoft.codeanalysis.common\3.9.0\lib\netstandard2.0\Microsoft.CodeAnalysis.dll
~\.nuget\packages\microsoft.codeanalysis.csharp\3.9.0\lib\netstandard2.0\Microsoft.CodeAnalysis.CSharp.dll
~\.nuget\packages\system.buffers\4.5.1\ref\netstandard2.0\System.Buffers.dll
~\.nuget\packages\system.collections.immutable\5.0.0\lib\netstandard2.0\System.Collections.Immutable.dll
~\.nuget\packages\system.memory\4.5.4\lib\netstandard2.0\System.Memory.dll
~\.nuget\packages\system.numerics.vectors\4.5.0\ref\netstandard2.0\System.Numerics.Vectors.dll
~\.nuget\packages\system.reflection.metadata\5.0.0\lib\netstandard2.0\System.Reflection.Metadata.dll
~\.nuget\packages\system.runtime.compilerservices.unsafe\5.0.0\ref\netstandard2.0\System.Runtime.CompilerServices.Unsafe.dll
~\.nuget\packages\system.text.encoding.codepages\4.5.1\lib\netstandard2.0\System.Text.Encoding.CodePages.dll
~\.nuget\packages\system.text.encodings.web\5.0.0\lib\netstandard2.0\System.Text.Encodings.Web.dll
~\.nuget\packages\system.text.json\5.0.1\lib\netstandard2.0\System.Text.Json.dll
~\.nuget\packages\system.threading.tasks.extensions\4.5.4\lib\netstandard2.0\System.Threading.Tasks.Extensions.dll

This isn't perfect as it adds a few other libraries against TargetPathWithTargetPlatformMoniker that shouldn't be there but with some more time processing the available data in the custom target, it would be possible to have this configured without any issues.

My question though goes back to: Why isn't something like this done automatically for projects with source generators anyway?

Honestly, I don't know really if this is much a Roslyn issue now or a default targets issue with MSBuild or .NET itself. I want to push for having this automatically handled by some part of the build process so users don't need to bother with manually configuring it.

Any advice on where this would be best to further discuss would be appreciated!

[Lonli-Lokli]

@jaredpar can you comment on that?

[kamronbatman]

What @Turnerj recommended did not work for me:

Exception was of type 'FileLoadException' with message 'Could not load file or assembly 'System.Text.Json, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. Could not find or load a specific file. (0x80131621)'

This is a source generator used against a .net 5 project.

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>netstandard2.0</TargetFramework>
        <LangVersion>preview</LangVersion>
        <BuildOutputTargetFolder>analyzers</BuildOutputTargetFolder>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.2" PrivateAssets="all" />
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="3.9.0" />
        <PackageReference Include="Humanizer.Core" Version="2.10.1" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="5.0.0" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Buffers" Version="4.5.1" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Memory" Version="4.5.4" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Numerics.Vectors" Version="4.5.0" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="5.0.0" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Text.Encodings.Web" Version="5.0.1" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Threading.Tasks.Extensions" Version="4.5.4" GeneratePathProperty="true" PrivateAssets="all" />
        <PackageReference Include="System.Text.Json" Version="5.0.2" GeneratePathProperty="true" PrivateAssets="all" />
    </ItemGroup>
    
    <PropertyGroup>
        <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
    </PropertyGroup>
    
    <Target Name="GetDependencyTargetPaths">
        <ItemGroup>
            <TargetPathWithTargetPlatformMoniker Include="$(PKGHumanizer_Core)\lib\netstandard2.0\Humanizer.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Text_Json)\lib\netstandard2.0\System.Text.Json.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Text_Encodings_Web)\lib\netstandard2.0\System.Text.Encodings.Web.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Buffers)\lib\netstandard2.0\System.Buffers.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Memory)\lib\netstandard2.0\System.Memory.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Numerics_Vectors)\lib\netstandard2.0\System.Numerics.Vectors.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Runtime_CompilerServices_Unsafe)\lib\netstandard2.0\System.Runtime.CompilerServices.Unsafe.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Threading_Tasks_Extensions)\lib\netstandard2.0\System.Threading.Tasks.Extensions.dll" IncludeRuntimeDependency="false" />
            <TargetPathWithTargetPlatformMoniker Include="$(PKGMicrosoft_Bcl_AsyncInterfaces)\lib\netstandard2.0\Microsoft.Bcl.AsyncInterfaces.dll" IncludeRuntimeDependency="false" />
        </ItemGroup>
    </Target>
</Project>