Run All fails with NotImplementedException

[JesseOlmer]

Using the Run All button in Test Explorer fails to execute tests with the following stack in Test Output. Highlighting all tests in Test Explorer, right-clicking and using "Run Selected Tests" successfully executes all tests.

Visual Studio 2015 Community Edition, Update 2.

ERROR: Exception while running tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.TestAdapter.TestExecutor.GetAllTestCasesInExecutables(IEnumerable`1 executables)
   at GoogleTestAdapter.TestAdapter.TestExecutor.RunTests(IEnumerable`1 executables, IRunContext runContext, IFrameworkHandle frameworkHandle)

[csoltenborn]

I've just updated my VS 2015 Community Edition to Update 2, and I cannot reproduce the issue. What's weird is that the code to resolve test case locations is the same, no matter whether all or a selection of tests is executed...

Can you please let us know the version of your msdia140.dll (to be found at C:\Program Files (x86)\Microsoft Visual Studio 14.0\DIA SDK\bin)?

As a workaround, you can disable symbol information parsing at the Google Test Adapter General options (resulting in no source locations of your tests, but you will still get links to failed assertions etc.).

[JesseOlmer]

msdia140.dll is 14.0.23918.0

For full disclosure, the other issues I'm hitting with GoogleTestAdapter (and not the built-in test adapters) are:

1. My test assembly pdb files remain locked after a test run (by TE.ProcessHost.Managed.exe)
2. After building I have the following error:

------ Discover test started ------
Test run will use DLL(s) built for framework Framework45 and platform X64. Following DLL(s) will not be part of run: 
test.exe is built for Framework None and Platform X86.
 Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
GoogleTest: Does D:\path\to\test.exe match [Tt]est[s]{0,1}.*.exe: true
Found 7 tests, resolving symbols
Loading symbols from D:\path\to\test.exe
From D:\path\to\test.exe, found 0 symbols in 42 ms
Couldn't find 7 symbols in D:\path\to\test.exe, looking from DllImports in module directory D:\path\to
D:\path\to\test.exe matches regex '[Tt]est[s]?\.exe'
ERROR: Exception while discovering tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.GoogleTestDiscoverer.DiscoverTests(IEnumerable`1 executables, ITestFrameworkReporter reporter)
   at GoogleTestAdapter.TestAdapter.TestDiscoverer.DiscoverTests(IEnumerable`1 executables, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
========== Discover test finished: 7 found (0:00:01.9906745) ==========

[csoltenborn]

Ok, the Dia Dll is the same as on my machine.

We haven't seen any issue like this afaik, and some of the log output is not generated by the test adapter itself. Any chance that you can share the solution containing your 7 tests?

[csoltenborn]

For the record: replacing some of the implemented method bodies in DiaMemoryStream with throw new NotImplementedException(); results in different exceptions (x86 as well as x64):

ERROR: Exception while discovering tests: System.Runtime.InteropServices.COMException (0x806D0004): Ausnahme von HRESULT: 0x806D0004
bei Dia.IDiaDataSource.loadDataFromIStream(IStream pIStream)...

[jgefele]

Hi @JesseOlmer,

the debug output ("GoogleTest: Does D:\path\to\test.exe match", "Found 7 tests, resolving symbols") doesn't look like our debug output, but rather those of the test runner we forked / ported to C#: github, VS gallery

Are you sure you're reporting to the correct project?

Cheers,
Jonas

[JesseOlmer]

Hi @jgefele,

I'm running "Google Test Adapter" from the VS Extension Gallery. You and Christian are listed as the authors, version is 0.5.1.345.
I'm using Google Test from https://github.com/google/googletest which I compiled locally on the machine experiencing this issue.

@csoltenborn I'll put together a stripped-down project tonight that will hopefully repro the issue.

Also, a slight correction to an earlier comment of mine: My test PDB files are locked after test discovery after a build, not only after an actual test run.

Thanks!

[csoltenborn]

@JesseOlmer Please make sure that you do not have both test adapters installed for some reason! If you search for resolving within our repository here at GitHub, you don't get any hit; if you do the same at the original repository, you get this hit:

logger.SendMessage(TestMessageLevel.Informational, sprintf "Found %d tests, resolving symbols" (testCases.Length))

The according line in our code looks like this:

TestEnvironment.LogInfo("Found " + testCases.Count + " tests in executable " + executable);

Same for the search term loading. That makes me think that Jonas might be right...

[jgefele]

In this commit the logging output has changed and the lines mentioned above have been removed.

[jgefele]

I checked again. These lines of your output seem to be from GoogleTestRunner (the other GoogleTest extension)

GoogleTest: Does D:\path\to\test.exe match [Tt]est[s]{0,1}.*.exe: true
Found 7 tests, resolving symbols
Loading symbols from D:\path\to\test.exe
From D:\path\to\test.exe, found 0 symbols in 42 ms
Couldn't find 7 symbols in D:\path\to\test.exe, looking from DllImports in module directory D:\path\to

However these lines look like our extension:

D:\path\to\test.exe matches regex '[Tt]est[s]?\.exe'
ERROR: Exception while discovering tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.GoogleTestDiscoverer.DiscoverTests(IEnumerable`1 executables, ITestFrameworkReporter reporter)
   at GoogleTestAdapter.TestAdapter.TestDiscoverer.DiscoverTests(IEnumerable`1 executables, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)

@JesseOlmer: Could you please double check that you have no other GoogleTest related extension installed/enabled and retry in order to isolate the issue?

@csoltenborn: When we get the stripped-down project, can you try to reproduce it? I still haven't my Windows machine running again...

[JesseOlmer]

My apologies, I did have both extensions installed.
I've tried a few different things with this in mind and have experienced the following:
With GoogleTestRunner enabled, and GoogleTestAdapter disabled, everything works fine (test discovery, not locking the PDB, Run All).
With GoogleTestAdapter enabled, and GoogleTestRunner disabled, everything remains broken.

I'll work on that stripped-down project in a few hours and get that out to you, unless you think somehow GoogleTestRunner is breaking GoogleTestAdapter even when disabled?

[csoltenborn]

@jgefele Sure, I will give it a try... If GoogleTestRunner indeed breaks our adapter when installed in parallel, we have at least learned something. @JesseOlmer Thanks for your efforts!

[JesseOlmer]

Stripped down project is at https://github.com/JesseOlmer/gta-demo. Just needs your own gtestd.lib and includes.

I went ahead and uninstalled both extensions, restarted, re-installed GTA, restarted again, and am still seeing the issues.

[csoltenborn]

@JesseOlmer Thanks for the sample solution! I can indeed reproduce the problem now... stay tuned.

[csoltenborn]

Here's a quick update of what we think we have understood so far:

• It seems that our code is not (directly) responsible for the exception
• An earlier Dia library does not throw an exception, but doesn't find your test either

Thus, we think that there's something special about the PDB generated by your solution. If you have any idea on that, please let us know - we haven't seen anything obvious with your solution and project settings.

We will continue investigation, but that'll take some time. In the meantime, we will release an intermediate version which will ignore the exception and behave as if the Parse symbol information option is set to false. Additionally, we will make sure that the PDB is never locked by our code (that's indeed what one could consider a bug).