Repository navigation
Invoke-ScriptAnalyzer -Recurse taking forever (60+ seconds) to finish #1189
Description
To be honest, this is my first time ever I tried to use Invoke-ScriptAnalyzer. So please be patient with me and if I am doing things wrong please let me know. Most of the time I use defaults on VSCode but I wanted to play with it today a bit. I have very simple project with just few files. When I used -Recurse it's stuck. I am not sure if this is just my setup or I need to wait much much longer but it would seem weird for it to be waiting like that on 100 lines.
Keep in mind that my C:\Support\Github is actually hardlinked.
Now I am wondering if that is not being a problem overall for all my reports about broken stuff in VSCode where I get lots and lots of perf issues,, broken links. Notice how C: is actually missing from fsutil.exe output. I am not sure if that's supposed to be like that thou and how it would affect your tool.
I did move the project to Desktop or tried using OneDrive folder directly but it didn't change the output so maybe it's not related.
I did try this outside of VSCode with same results.
Steps to reproduce
Invoke-ScriptAnalyzer -Path C:\Support\GitHub\Dashimo -Recurse -VerboseExpected behavior
Finishing up.
Actual behavior
Environment data
> $PSVersionTable
Name Value
---- -----
PSVersion 5.1.17763.316
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.17763.316
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
> (Get-Module -ListAvailable PSScriptAnalyzer).Version | ForEach-Object { $_.ToString() }
1.18.0
1.17.1
1.17.1
Activity
PrzemyslawKlys commentedon Mar 23, 2019
Actually, after waiting a bit more like 60+ seconds, it started returning stuff...
What is funny subsequent retries gets it done in 1 second or less.
2nd try:
[-]Invoke-ScriptAnalyzer -Recurse never ending[/-][+]Invoke-ScriptAnalyzer -Recurse taking forever (60+ seconds) to finish[/+]bergmeister commentedon Mar 23, 2019
1.18.0 is twice as fast for cold runs and hundreds of times faster when re-analyzing due to caching. Please provide a time difference between 1.17.1 and 1.18.0 for a cold run (new shell). For a big project like e.g. the test folder of the PowerShell repo analysis can take 2 minutes. Without seeing the code or a time difference I cannot say anything
PrzemyslawKlys commentedon Mar 23, 2019
The file it was stuck on has 93 lines. Whole project has like 10 files and 1000 lines in total. This is very small project. Unless it analyses all dependencies and loads all required modules I cant see how it can take so long. I’ll publish sources for that project when I get back home.
bergmeister commentedon Mar 23, 2019
Ok, if you've nailed it down to one file (if just analysing this one file in a new shell is still slow) that is problematic that then that file please and a time comparison with 1.17.1 please so that we know if that is a regression or not.
It does traverse some of the function calls and the most expensive part are usually the calls to Get-Command for getting CommandInfo details. Maybe there is some deep recursion going on in your case.
PrzemyslawKlys commentedon Mar 23, 2019
This one gave an error but I actually preloaded Import-Module required modules before running script analyzer. Will prepare more diagnostics soon.
PrzemyslawKlys commentedon Mar 23, 2019
Results:
PrzemyslawKlys commentedon Mar 23, 2019
Here with older version:
PrzemyslawKlys commentedon Mar 23, 2019
Dashimo - Copy.zip
Here is Dashimo project. It's very small. It requires some modules but as you see above it takes like 15 seconds to load them all (they are not compiled (multifile modules) on my computer).
PrzemyslawKlys commentedon Mar 23, 2019
Just to say, if you would like a remote session to see it with your own eyes, play around my system I'm good with that. I can do anything you want when it comes to testing this but I know from experience that it sometimes it's easier to do some debug yourself.
PrzemyslawKlys commentedon Mar 23, 2019
I did some more testing. I was able to bring the scan to 21 seconds. I've noticed the speed up when I've merged all my modules into single file modules. I've noticed that some modules even in single file take a while to load like 7 seconds if they are the first time ever loaded. Subsequent import-module with force takes 0 seconds.
Now to me, it looks like you're triggering Import-Module during scan even thou it's already triggered and commands are available. I can't pinpoint why would my modules behave the way they do when those are loaded for the first time ever.
I can only speculate
Being the reason. Otherwise, it's just a bunch of functions in each one.
bergmeister commentedon Mar 23, 2019
It seems, the 1.18 release made analysis faster as expected, so no regression.
The calls to
Get-Commandare implicitly triggering the module loads, see #841 (comment)On my machine it takes 10 seconds, so I can assume only that you have much more modules installed and your system might be a bit slower as well (as I'm on a 6 core i7).
I googled the error that you had and that comes from the SQL server module that is not thread safe, PSSA invokes all rules in parallel, so technically one would need to call each rule sequentially in this case by calling
Invoke-ScriptAnalyzer -Includeforeach rule but this would slow down execution of course. There are already proposed changes to multi-threading in the pipeline that would speedup cold runs by 80% so I'd expect the behaviour to become slowly better in the future over timePrzemyslawKlys commentedon Mar 23, 2019
I have I9 9900K, with 2500mb/s read/write speeds and 32gb of ram. Updated recently.
But is it calling modules with -Force? After merging every single module to 1 file it takes like 0 seconds per module to load and I load them before executing your script. Or maybe it's doing Import-Module for all my modules in the system?
PrzemyslawKlys commentedon Mar 23, 2019
Also why would it trigger SQL server module? I am not using that in that module?
bergmeister commentedon Mar 24, 2019
I found a reference to the SQL server module here when I googled the error message, but maybe it was a red herring as this thread indicates it might be a threading issue with
-Verbose.You can save some time by importing modules beforehand but PSSA calls
Get-Command -Name $namefor nearly every command that it encounters, which then implicitly leads to module loads. If there is inter-dependencies between modules then it will start loading more modules but that is just the behaviour of PowerShell itself. I plan to replace it with a single call toGet-Commandat some point but this is blocked on this bug of PowerShell (that I actually discussed last with members of the PS team).For the moment, we know that PSSA is not always the fastest for cold runs but that is also because it is doing a lot of work to get as much information out as possible. Things will slowly get better over time like in the last release where speed was doubled and similar improvements are coming in the future.
You could try running this in PowerShell Core where you probably have fewer modules installed that could make it faster.
PrzemyslawKlys commentedon Mar 24, 2019
I understand. This error only showed up when using VSCode, outside of VSCode no error (on same code).
If I understand correctly how PSScriptAnalyzer works, it scans the whole file and then for each command it finds in the file it would do Get-Command. So if I am preloading all modules beforehand everything should be loaded already and take no time.
When I used PS6 it does the scan in 1 second for everything. That's a huge difference :-) 1minute vs 1second. Of course, I don't have my PSWriteExcel, PSWriteWord, PSWinDocumentation and/or PSWinReporting.
Now I am thinking that maybe something else is in play here? What if I would do a short script and loop over each file myself? Is there a way to do more debug as in which command it's stuck on the most (because I see this only on one file or so).
I event went mad on import and did
Get-Module -ListAvailableand put that over to Import-Module. Lots of errors later it still didn't help.msftrncs commentedon May 6, 2019
I found my system got very slow (anything with PowerShell in VS Code) after installing the .NET SDK / Visual Studio Community / or the Visual Studio Build Tools. Haven't narrowed done exactly which.