Update troubleshooting.md

[mklement0]

The Homebrew installation is based on a so-called keg-only formula, which means that symlinks in /usr/local are not established.

Note that with Homebrew installed, sudo is not needed to create the symlinks in the /usr/local subtree.

[msftclas]

Hi @mklement0, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, MSBOT;

[daviwil]

Awesome, thanks Michael!

[mklement0]

My pleasure, @daviwil, but I just stumbled upon this, which suggests that symlinking to /usr/local/lib should be avoided for security reasons (I don't understand the issues involved), and is actually not necessary when .NET Core is installed as part of the PowerShell macOS package (emphasis partly mine):

"
Please ignore .NET Core's installation instructions to manually link the OpenSSL libraries. This is not required for PowerShell as we patch .NET Core's cryptography libraries to find Homebrew's OpenSSL in its installed location. Again, do not run brew link --forcenor ln -s for OpenSSL, regardless of other instructions.

Homebrew previously allowed OpenSSL libraries to be linked to the system library location; however, this created major security holes and is no longer allowed.

Because .NET Core's 1.0.0 release libraries still look in the prior system location for OpenSSL, they will fail to work unless the libraries are manually placed there (security risk), or their libraries are patched (which we do).

To patch .NET Core's cryptography libraries, we use install_name_tool:

find ~/.nuget -name System.Security.Cryptography.Native.dylib | xargs sudo install_name_tool -add_rpath /usr/local/opt/openssl/lib

This updates .NET Core's library to look in Homebrew's OpenSSL installation location instead of the system library location. The PowerShell macOS package come with the necessary libraries patched, and the build script patches the libraries on-the-fly when building from source. You can run this command manually if you're having trouble with .NET Core's cryptography libraries.
"
https://github.com/PowerShell/PowerShell/blob/master/docs/installation/linux.md#os-x-1011

So perhaps including this find ... | xargs sudo install_name_tool ... command is all that should be mentioned in troubleshooting.md - which should work for both Homebrew and MacPorts installations.

One thing that puzzles me, however, is that the find command targets ~/.nuget.
The .NET Core version that comes with the PowerShell for macOS installer package installs to /usr/local/microsoft/powershell instead.

[mklement0]

Oops! Just realized that the VSCode extension at hand apparently still needs the OpenSSL libraries in /usr/local/lib. Is there a way to avoid this, given the security concerns?

[daviwil]

Hey Michael, thanks a ton for digging into that. I totally missed those details on symbolic linking being a security risk. I'm not exactly sure how patching the libraries is considered more secure, but I'll take this information to account to update the troubleshooting guidance to use the recommended approach. Looks like I need to also check for Homebrew's installation path for OpenSSL so I don't incorrectly advise those users to use the symbolic link fix.

I've filed issue #292 to track this. Thanks again!

[mklement0]

My pleasure, @daviwil - thank you for a great extension.