Skip to content

Update to Microsoft.CodeAnalysis.Testing #2562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 19, 2019
Merged

Update to Microsoft.CodeAnalysis.Testing #2562

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Directory.Build.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
https://api.nuget.org/v3/index.json;
https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json;
https://dotnet.myget.org/F/dotnet-core/api/v3/index.json;
https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json;
</RestoreSources>
</PropertyGroup>

Expand Down
1 change: 1 addition & 0 deletions build/Dependencies.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
<!-- Test-only Dependencies -->
<PropertyGroup>
<BenchmarkDotNetVersion>0.11.3</BenchmarkDotNetVersion>
<MicrosoftCodeAnalysisTestingVersion>1.0.0-beta1-63812-02</MicrosoftCodeAnalysisTestingVersion>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know anything regarding analyzers. Is it normal to use beta package? Is where are stable version available?

Copy link
Contributor Author

@sharwell sharwell Feb 15, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a dependency we control. We're still stabilizing the API for the testing package (and determining whether breaking changes are allowed in updates after stable) so this is what's currently available. The package performs much more rigorous testing than previous harnesses, which will show up in some of the comments. Several other analyzers internally (dotnet/roslyn-analyzers, Microsoft/vs-threading, Microsoft/VSSDK-Analyzers) and externally (DotNetAnalyzers/StyleCopAnalyzers) are already using this.

<MicrosoftMLTestModelsPackageVersion>0.0.3-test</MicrosoftMLTestModelsPackageVersion>
<MicrosoftMLTensorFlowTestModelsVersion>0.0.10-test</MicrosoftMLTensorFlowTestModelsVersion>
<MicrosoftMLOnnxTestModelsVersion>0.0.4-test</MicrosoftMLOnnxTestModelsVersion>
Expand Down
7 changes: 7 additions & 0 deletions src/Microsoft.ML.Analyzer/Properties/AssemblyInfo.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Runtime.CompilerServices;

[assembly: InternalsVisibleTo("Microsoft.ML.CodeAnalyzer.Tests, PublicKey=002400000480000094000000060200000024000052534131000400000100010015c01ae1f50e8cc09ba9eac9147cf8fd9fce2cfe9f8dce4f7301c4132ca9fb50ce8cbf1df4dc18dd4d210e4345c744ecb3365ed327efdbc52603faa5e21daa11234c8c4a73e51f03bf192544581ebe107adee3a34928e39d04e524a9ce729d5090bfd7dad9d10c722c0def9ccc08ff0a03790e48bcd1f9b6c476063e1966a1c4")]
70 changes: 30 additions & 40 deletions test/Microsoft.ML.CodeAnalyzer.Tests/Code/BestFriendOnPublicDeclarationTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,61 +3,51 @@
// See the LICENSE file in the project root for more information.

using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Reflection;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Diagnostics;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis.Testing;
using Microsoft.ML.CodeAnalyzer.Tests.Helpers;
using Xunit;
using VerifyCS = Microsoft.ML.CodeAnalyzer.Tests.Helpers.CSharpCodeFixVerifier<
Microsoft.ML.InternalCodeAnalyzer.BestFriendOnPublicDeclarationsAnalyzer,
Microsoft.CodeAnalysis.Testing.EmptyCodeFixProvider>;

namespace Microsoft.ML.InternalCodeAnalyzer.Tests
{
public sealed class BestFriendOnPublicDeclarationTest : DiagnosticVerifier<BestFriendOnPublicDeclarationsAnalyzer>
public sealed class BestFriendOnPublicDeclarationTest
{
private readonly Lazy<string> SourceAttribute = TestUtils.LazySource("BestFriendAttribute.cs");
private readonly Lazy<string> SourceDeclaration = TestUtils.LazySource("BestFriendOnPublicDeclaration.cs");

[Fact]
public void BestFriendOnPublicDeclaration()
public async Task BestFriendOnPublicDeclaration()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 I kept the test names the same, which means you'll keep your pass/fail history on ADO. If you prefer the Async suffix convention even in test code, I'm happy to add it.

{
Solution solution = null;
var projA = CreateProject("ProjectA", ref solution, SourceDeclaration.Value, SourceAttribute.Value);

var analyzer = new BestFriendOnPublicDeclarationsAnalyzer();

var refs = new List<MetadataReference> {
RefFromType<object>(), RefFromType<Attribute>(),
MetadataReference.CreateFromFile(Assembly.Load("netstandard, Version=2.0.0.0").Location),
MetadataReference.CreateFromFile(Assembly.Load("System.Runtime, Version=0.0.0.0").Location)
var expected = new DiagnosticResult[] {
VerifyCS.Diagnostic().WithLocation(8, 6).WithArguments("PublicClass"),
VerifyCS.Diagnostic().WithLocation(11, 10).WithArguments("PublicField"),
VerifyCS.Diagnostic().WithLocation(14, 10).WithArguments("PublicProperty"),
VerifyCS.Diagnostic().WithLocation(20, 10).WithArguments("PublicMethod"),
VerifyCS.Diagnostic().WithLocation(26, 10).WithArguments("PublicDelegate"),
VerifyCS.Diagnostic().WithLocation(29, 10).WithArguments("PublicClass"),
VerifyCS.Diagnostic().WithLocation(35, 6).WithArguments("PublicStruct"),
VerifyCS.Diagnostic().WithLocation(40, 6).WithArguments("PublicEnum"),
VerifyCS.Diagnostic().WithLocation(47, 6).WithArguments("PublicInterface"),
VerifyCS.Diagnostic().WithLocation(102, 10).WithArguments("PublicMethod"),
};

var comp = projA.GetCompilationAsync().Result.WithReferences(refs.ToArray());
var compilationWithAnalyzers = comp.WithAnalyzers(ImmutableArray.Create((DiagnosticAnalyzer)analyzer));
var allDiags = compilationWithAnalyzers.GetAnalyzerDiagnosticsAsync().Result;

var projectTrees = new HashSet<SyntaxTree>(projA.Documents.Select(r => r.GetSyntaxTreeAsync().Result));
var diags = allDiags
.Where(d => d.Location == Location.None || d.Location.IsInMetadata || projectTrees.Contains(d.Location.SourceTree))
.OrderBy(d => d.Location.SourceSpan.Start).ToArray();

var diag = analyzer.SupportedDiagnostics[0];
var expected = new DiagnosticResult[] {
diag.CreateDiagnosticResult(8, 6, "PublicClass"),
diag.CreateDiagnosticResult(11, 10, "PublicField"),
diag.CreateDiagnosticResult(14, 10, "PublicProperty"),
diag.CreateDiagnosticResult(20, 10, "PublicMethod"),
diag.CreateDiagnosticResult(26, 10, "PublicDelegate"),
diag.CreateDiagnosticResult(29, 10, "PublicClass"),
diag.CreateDiagnosticResult(35, 6, "PublicStruct"),
diag.CreateDiagnosticResult(40, 6, "PublicEnum"),
diag.CreateDiagnosticResult(47, 6, "PublicInterface"),
diag.CreateDiagnosticResult(102, 10, "PublicMethod")
var test = new VerifyCS.Test
{
TestState =
{
Sources =
{
SourceDeclaration.Value,
("BestFriendAttribute.cs", SourceAttribute.Value),
},
},
};

VerifyDiagnosticResults(diags, analyzer, expected);
test.ExpectedDiagnostics.AddRange(expected);
await test.RunAsync();
}
}
}

111 changes: 54 additions & 57 deletions test/Microsoft.ML.CodeAnalyzer.Tests/Code/BestFriendTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,20 @@
// See the LICENSE file in the project root for more information.

using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Diagnostics;
using Microsoft.CodeAnalysis.Testing;
using Microsoft.ML.CodeAnalyzer.Tests.Helpers;
using Xunit;
using VerifyCS = Microsoft.ML.CodeAnalyzer.Tests.Helpers.CSharpCodeFixVerifier<
Microsoft.ML.InternalCodeAnalyzer.BestFriendAnalyzer,
Microsoft.CodeAnalysis.Testing.EmptyCodeFixProvider>;

namespace Microsoft.ML.InternalCodeAnalyzer.Tests
{
public sealed class BestFriendTest : DiagnosticVerifier<BestFriendAnalyzer>
public sealed class BestFriendTest
{
// We do things in this somewhat odd way rather than just referencing the Core assembly directly,
// because we certainly want the best friend attribute itself to be internal, but the assembly
Expand All @@ -29,70 +29,67 @@ public sealed class BestFriendTest : DiagnosticVerifier<BestFriendAnalyzer>
private readonly Lazy<string> SourceUser = TestUtils.LazySource("BestFriendUser.cs");

[Fact]
public void BestFriend()
public async Task BestFriend()
{
// The setup to this one is a bit more involved than many of the analyzer tests,
// because in this case we have to actually set up *two* assemblies, where the
// first considers the second a friend. But, setting up this dependency structure
// so that things actually compile to the point where the analyzer can actually do
// its work is rather involved.
Solution solution = null;
var projA = CreateProject("ProjectA", ref solution, SourceDeclaration.Value);
var projB = CreateProject("ProjectB", ref solution, SourceUser.Value);
solution = solution.AddProjectReference(projB.Id, new ProjectReference(projA.Id));

var analyzer = new BestFriendAnalyzer();
var expected = new DiagnosticResult[] {
VerifyCS.Diagnostic().WithLocation(10, 31).WithArguments("A"),
VerifyCS.Diagnostic().WithLocation(11, 31).WithArguments("A"),
VerifyCS.Diagnostic().WithLocation(11, 33).WithArguments("My"),
VerifyCS.Diagnostic().WithLocation(14, 33).WithArguments("Awhile"),
VerifyCS.Diagnostic().WithLocation(15, 33).WithArguments("And"),
VerifyCS.Diagnostic().WithLocation(18, 13).WithArguments("A"),
VerifyCS.Diagnostic().WithLocation(18, 25).WithArguments("A"),
new DiagnosticResult("CS0122", DiagnosticSeverity.Error).WithLocation(23, 21).WithMessage("'D.D(float)' is inaccessible due to its protection level"),
VerifyCS.Diagnostic().WithLocation(25, 13).WithArguments("IA"),
VerifyCS.Diagnostic().WithLocation(25, 23).WithArguments("IA"),
VerifyCS.Diagnostic().WithLocation(32, 38).WithArguments(".ctor"),
VerifyCS.Diagnostic().WithLocation(38, 38).WithArguments(".ctor"),
};

MetadataReference peRef;
var refs = new List<MetadataReference> {
RefFromType<object>(), RefFromType<Attribute>(),
MetadataReference.CreateFromFile(Assembly.Load("netstandard, Version=2.0.0.0").Location),
MetadataReference.CreateFromFile(Assembly.Load("System.Runtime, Version=0.0.0.0").Location)
};
using (var ms = new MemoryStream())
VerifyCS.Test test = null;
test = new VerifyCS.Test
{
// We also test whether private protected can be accessed, so we need C# 7.2 at least.
var parseOpts = new CSharpParseOptions(LanguageVersion.CSharp7_3);
var tree = CSharpSyntaxTree.ParseText(SourceDeclaration.Value, parseOpts);
var treeAttr = CSharpSyntaxTree.ParseText(SourceAttribute.Value, parseOpts);

var compOpts = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
var innerComp = CSharpCompilation.Create(projA.Name, new[] { tree, treeAttr }, refs, compOpts);

var emitResult = innerComp.Emit(ms);
Assert.True(emitResult.Success, $"Compilation of {projA.Name} did not work. Diagnostics: {string.Join(" || ", emitResult.Diagnostics)}");
LanguageVersion = LanguageVersion.CSharp7_2,
TestState =
{
Sources = { SourceUser.Value },
AdditionalReferences = { MetadataReference.CreateFromFile(typeof(Console).Assembly.Location) },
},
SolutionTransforms =
{
(solution, projectId) =>
{
var projectA = solution.AddProject("ProjectA", "ProjectA", LanguageNames.CSharp);
projectA = projectA.AddDocument("BestFriendAttribute.cs", SourceAttribute.Value).Project;
projectA = projectA.AddDocument("BestFriendDeclaration.cs", SourceDeclaration.Value).Project;
projectA = projectA.WithParseOptions(((CSharpParseOptions)projectA.ParseOptions).WithLanguageVersion(LanguageVersion.CSharp7_2));
projectA = projectA.WithCompilationOptions(projectA.CompilationOptions.WithOutputKind(OutputKind.DynamicallyLinkedLibrary));
projectA = projectA.WithMetadataReferences(solution.GetProject(projectId).MetadataReferences.Concat(test.TestState.AdditionalReferences));
solution = projectA.Solution;

var peImage = ms.ToArray().ToImmutableArray();
peRef = MetadataReference.CreateFromImage(peImage);
}
solution = solution.AddProjectReference(projectId, new ProjectReference(projectA.Id));
solution = solution.WithProjectAssemblyName(projectId, "ProjectB");

refs.Add(peRef);
var comp = projB.GetCompilationAsync().Result
.WithReferences(refs.ToArray());
var compilationWithAnalyzers = comp.WithAnalyzers(ImmutableArray.Create((DiagnosticAnalyzer)analyzer));
var allDiags = compilationWithAnalyzers.GetAnalyzerDiagnosticsAsync().Result;

var projectTrees = new HashSet<SyntaxTree>(projB.Documents.Select(r => r.GetSyntaxTreeAsync().Result));
var diags = allDiags
.Where(d => d.Location == Location.None || d.Location.IsInMetadata || projectTrees.Contains(d.Location.SourceTree))
.OrderBy(d => d.Location.SourceSpan.Start).ToArray();

var diag = analyzer.SupportedDiagnostics[0];
var expected = new DiagnosticResult[] {
diag.CreateDiagnosticResult(10, 31, "A"),
diag.CreateDiagnosticResult(11, 31, "A"),
diag.CreateDiagnosticResult(11, 33, "My"),
diag.CreateDiagnosticResult(14, 33, "Awhile"),
diag.CreateDiagnosticResult(15, 33, "And"),
diag.CreateDiagnosticResult(18, 13, "A"),
diag.CreateDiagnosticResult(18, 25, "A"),
diag.CreateDiagnosticResult(25, 13, "IA"),
diag.CreateDiagnosticResult(25, 23, "IA"),
diag.CreateDiagnosticResult(32, 38, ".ctor"),
diag.CreateDiagnosticResult(38, 38, ".ctor"),
return solution;
},
},
};

VerifyDiagnosticResults(diags, analyzer, expected);
// Remove these assemblies, or an additional definition of BestFriendAttribute could exist and the
// compilation will not be able to locate a single definition for use in the analyzer.
test.TestState.AdditionalReferences.Remove(AdditionalMetadataReferences.MLNetCoreReference);
test.TestState.AdditionalReferences.Remove(AdditionalMetadataReferences.MLNetDataReference);
test.TestState.AdditionalReferences.Remove(AdditionalMetadataReferences.MLNetStaticPipeReference);

test.Exclusions &= ~AnalysisExclusions.GeneratedCode;
test.ExpectedDiagnostics.AddRange(expected);
await test.RunAsync();
}
}
}
Loading