Running test suite with /logger:html does not generate HTML files per test case

[dinebhar]

I am trying to run test suite with following command.

COMMAND: "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" C:\MicrosoftProtocolTests\FileServer\Server-Endpoint\1.0.5000.0\Bin\MS-SMB2_ServerTestSuite.dll /TestCaseFilter:"TestCategory=smb21" /Settings:C\MsWinSMBProtocolTest20170111_091422042536\TestSuiteConfigs\ServerLocalTestRun.testrunconfig /Logger:html

test run went good, the results also getting generated but the html files for each test case does not.
./Html folder shows empty. Following is the directory structure getting generated for the results.

dbharti % ls -R
.:
Captures Html index.html js Txt

./Captures:

./Html:

./js:
functions.js

./Txt:
2017-01-11-04-46-06_Passed_BVT_Negotiate_Compatible_Wildcard.txt
2017-01-11-04-46-10_Passed_BVT_Negotiate_Compatible_2002.txt
2017-01-11-04-46-12_Passed_BVT_Negotiate_SigningEnabled.txt
2017-01-11-04-46-14_Passed_BVT_Negotiate_SMB2002.txt
2017-01-11-04-46-16_Passed_BVT_Negotiate_SMB21.txt
2017-01-11-04-46-18_Passed_BVT_Negotiate_SMB30.txt
2017-01-11-04-46-20_Passed_BVT_Negotiate_SMB302.txt
2017-01-11-04-46-22_Passed_BVT_Negotiate_SMB311.txt
2017-01-11-04-46-24_Inconclusive_BVT_Negotiate_SMB311_Preauthentication.txt
2017-01-11-04-46-26_Inconclusive_BVT_Negotiate_SMB311_Preauthentication_Encryption_CCM.txt
2017-01-11-04-46-28_Inconclusive_BVT_Negotiate_SMB311_Preauthentication_Encryption_GCM.txt
2017-01-11-04-46-30_Inconclusive_Negotiate_SMB311_WithoutAnyContexts.txt
2017-01-11-04-46-32_Inconclusive_Negotiate_SMB311_WithEncryptionContextWithoutIntegrityContext.txt
2017-01-11-04-46-34_Inconclusive_Negotiate_SMB311_InvalidHashAlgorithm.txt
2017-01-11-04-46-36_Inconclusive_Negotiate_SMB311_InvalidCipher.txt

please check the attached screen shot

[yazeng]

Hi
I tried to repro this on my environment but failed. Everything works fine to me.
Is that a typo, missing ":" between "C\" in this "/Settings:C\MsWinSMBProtocolTest20170111_091422042536\TestSuiteConfigs\ServerLocalTestRun.testrunconfig "?
And what's the version of PTF (Protocol Test Framework) you are using?

The HTML files should be generated together with .txt files.
You check the function TestResultHandler below:
https://github.com/Microsoft/ProtocolTestFramework/blob/master/src/htmltestlogger/HtmlTestLogger.cs

Thanks,
Yan

[dinebhar]

thanks,

yes there is a typo while filing bug.

i am using PTF version 1.0
Downloaded from - https://github.com/Microsoft/ProtocolTestFramework/releases/download/1.0/ProtocolTestFrameworkInstaller.msi

But i could see the same issue.

[dinebhar]

The thing is the Txt files are getting generated for the test cases but not Html. I am not sure what I am missing here

[yazeng]

Since I cannot repro it in my machine, could you help debug it in your environment?

1. Download code, zip is enough.
   https://github.com/Microsoft/ProtocolTestFramework

2. Unzip the file in your machine.

3. Open "ptf.sln" in src folder by Visual Studio.

4. Modify the output path to the path below of HtmlTestLogger project,
   C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
   

5. Build the solution, then check if the new HtmlTestLogger.dll is generated in the above path:
   

6. Add breakpoint, see below:
   

7. Run your command to repro

8. When the command is running, attach the process vstest in Visual Studio.
   
   

The breakpoint should be hit:

Then you can debug to see what happened.

Thank you in advance!
Yan

[dinebhar]

Hey thank you so much. that helped a lot.

I am able to track down the issue. The method File.WriteAllText() was failing. Please check the following screenshot.

I am not sure whether it is correct or not. but when it executes, the code expects the ptfconfig files must be in "Bin" directory. The code here forces me to put my *.ptfconfigs in ..\Bin\ directory.

what if someone have placed their customize config files in a different folder and updated ServerLocalTestRun.testrunconfig and command for the same.

To fix my issue i have created a bin directory to my homedir and placed all ptfconfig files to it and it worked.

I guess the code must handle the situation irrespective of test suite installation directory path, if the another path is provided.

Thank you very much for your help!

[dinebhar]

for more info:
previously my command was:
cd "C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs\Logs" & "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" C:\MicrosoftProtocolTests\FileServer\Server-Endpoint\1.0.5000.0\Bin\MS-SMB2_ServerTestSuite.dll /TestCaseFilter:"TestCategory=Negotiate" /Settings:C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs\ServerLocalTestRun.testrunconfig /Logger:html

and ptfconfigs were at: C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs*.ptfconfig and *.testrunconfig

Then i have created a bin directory under C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs\Bin
and moved all *.ptfconfig, *.testrunconfig to the same.
New command
cd "C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs" & "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" C:\MicrosoftProtocolTests\FileServer\Server-Endpoint\1.0.5000.0\Bin\MS-SMB2_ServerTestSuite.dll /TestCaseFilter:"TestCategory=Negotiate" /Settings:C:\MsWinSMBProtocolTest20170112_005936513583\TestSuiteConfigs\Bin\ServerLocalTestRun.testrunconfig /Logger:html

[yazeng]

Well, now I can reproduce it in my machine too. :)
I agree with you, we should not expect the *.ptfconfig files are in "bin" folder.
Customized ptfconfig files are possible and we should support this scenario.
I'll file a bug and fix it later. Note, the fix will be delivered to PTF, not this repo.

Thank you very much for reporting and debugging this issue!
Yan

[yazeng]

The bug is fixed and changes are pushed to https://github.com/Microsoft/ProtocolTestFramework
@dinebhar , you can have a try when you have time.

[yazeng]

Close this issue since no update for a long time.