Merge pull request #2175 from andschwa/osx-openssl

Fix macOS CI build and installation instructions

.travis.yml

@@ -5,15 +5,19 @@ git:
 
 os:
   - linux
+  - osx
 sudo: required
 dist: trusty
+osx_image: xcode7.3
 
 addons:
   artifacts:
     paths: $(ls powershell*{deb,pkg} | tr "\n" ":")
 
 install:
-  - (cd tools && ./download.sh)
+  - pushd tools
+  - ./download.sh
+  - popd
   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then rvm use 2.2.1; fi # Default 2.0.0 Ruby is buggy
 
 script: ./tools/travis.sh

docs/building/macos.md

@@ -20,9 +20,8 @@ and use`Start-PSBootstrap` to install the dependencies.
 The `Start-PSBootstrap` function does the following:
 
 - Uses `brew` to install CMake, OpenSSL, and GNU WGet
-- Links OpenSSL
 - Uninstalls any prior versions of .NET CLI
-- Downloads and installs the latest .NET CLI 1.0.0-preview2 SDK to `~/.dotnet`
+- Downloads and installs the latest .NET CLI 1.0.0-preview3 SDK to `~/.dotnet`
 
 If you want to use `dotnet` outside of `Start-PSBuild`,
 add `~/.dotnet` to your `PATH` environment variable.
@@ -40,7 +39,7 @@ We cannot do this for you in the build module due to #[847][].
 [847]: https://github.com/PowerShell/PowerShell/issues/847
 
 error: dotnet restore
--------------------------
+---------------------
 
 If you run `dotnet restore` and get error like
 
@@ -55,18 +54,11 @@ error:   The type initializer for 'Crypto' threw an exception.
 error:   The type initializer for 'CryptoInitializer' threw an exception.
 error:   Unable to load DLL 'System.Security.Cryptography.Native': The specified module could not be found.
 error:    (Exception from HRESULT: 0x8007007E)
-
 ```
 
-Try the following
+These means you did not use our `Start-PSBootstrap` function to setup your environment,
-
+which handles patching .NET Core's bad cryptography libraries.
-* Make sure you have latest openssl and re-link it
+Please see our [macOS installation instructions](../installation/linux.md#openssl) for explanation.
-
-```
-brew update
-brew install openssl
-brew link --force openssl
-```
 
 Build using our module
 ======================

docs/installation/linux.md

@@ -72,9 +72,11 @@ sudo yum install https://github.com/PowerShell/PowerShell/releases/download/v6.0
 [CentOS 7]: https://www.centos.org/download/
 
 macOS 10.11
-==========
+===========
 
-Using macOS 10.11, download the PKG package `powershell-6.0.0-alpha.9.pkg` from the [releases][] page onto the macOS machine.
+Using macOS 10.11, download the PKG package
+`powershell-6.0.0-alpha.9.pkg`
+from the [releases][] page onto the macOS machine.
 
 Either double-click the file and follow the prompts,
 or install it from the terminal:
@@ -83,6 +85,50 @@ or install it from the terminal:
 sudo installer -pkg powershell-6.0.0-alpha.9.pkg -target /
 ```
 
+OpenSSL
+-------
+
+Also install [Homebrew's OpenSSL][openssl]:
+
+```
+brew install openssl
+```
+
+[Homebrew][brew] is the missing package manager for macOS.
+If the `brew` command was not found,
+you need to install Homebrew following [their instructions][brew].
+
+.NET Core requires Homebrew's OpenSSL because the "OpenSSL" system libraries on macOS are not OpenSSL,
+as Apple deprecated OpenSSL in favor of their own libraries.
+This requirement is not a hard requirement for all of PowerShell;
+however, most networking functions (such as `Invoke-WebRequest`)
+do require OpenSSL to work properly.
+
+**Please ignore** .NET Core's installation instructions to manually link the OpenSSL libraries.
+This is **not** required for PowerShell as we patch .NET Core's cryptography libraries to find Homebrew's OpenSSL in its installed location.
+Again, **do not** run `brew link --force` nor `ln -s` for OpenSSL, regardless of other instructions.
+
+Homebrew previously allowed OpenSSL libraries to be linked to the system library location;
+however, this created major security holes and is [no longer allowed][homebrew-patch].
+Because .NET Core's 1.0.0 release libraries still look in the prior system location for OpenSSL,
+they will fail to work unless the libraries are manually placed there (security risk),
+or their libraries are patched (which we do).
+To patch .NET Core's cryptography libraries, we use `install_name_tool`:
+
+```
+find ~/.nuget -name System.Security.Cryptography.Native.dylib | xargs sudo install_name_tool -add_rpath /usr/local/opt/openssl/lib
+```
+
+This updates .NET Core's library to look in Homebrew's OpenSSL installation location instead of the system library location.
+The PowerShell macOS package come with the necessary libraries patched,
+and the build script patches the libraries on-the-fly when building from source.
+You *can* run this command manually if you're having trouble with .NET Core's cryptography libraries.
+
+
+[openssl]: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openssl.rb
+[brew]: http://brew.sh/
+[homebrew-patch]: https://github.com/Homebrew/brew/pull/597
+
 Paths
 =====
 

tools/download.sh

@@ -7,10 +7,9 @@ trap '
   kill -s INT "$$"
 ' INT
 
-# Retrieves asset ID and package name of asset ending in argument
+get_url() {
-# $info looks like: "id": 1698239, "name": "powershell_0.4.0-1_amd64.deb",
+    release=v6.0.0-alpha.9
-get_info() {
+    echo "https://github.com/PowerShell/PowerShell/releases/download/$release/$1"
-    curl -s https://api.github.com/repos/PowerShell/PowerShell/releases/latest | grep -B 1 "name.*$1"
 }
 
 # Get OS specific asset ID and package name
@@ -20,28 +19,30 @@ case "$OSTYPE" in
         # Install curl and wget to download package
         case "$ID" in
             centos*)
-                if [[ -z $(command -v curl) ]]; then
+                if ! hash curl 2>/dev/null; then
                     echo "curl not found, installing..."
                     sudo yum install -y curl
                 fi
-                version=rpm
+
+                package=powershell-6.0.0_alpha.9-1.el7.centos.x86_64.rpm
                 ;;
             ubuntu)
+                if ! hash curl 2>/dev/null; then
+                    echo "curl not found, installing..."
+                    sudo apt-get install -y curl
+                fi
+
                 case "$VERSION_ID" in
                     14.04)
-                        version=ubuntu1.14.04.1_amd64.deb
+                        package=powershell_6.0.0-alpha.9-1ubuntu1.14.04.1_amd64.deb
                         ;;
                     16.04)
-                        version=ubuntu1.16.04.1_amd64.deb
+                        package=powershell_6.0.0-alpha.9-1ubuntu1.16.04.1_amd64.deb
                         ;;
                     *)
                         echo "Ubuntu $VERSION_ID is not supported!" >&2
                         exit 2
                 esac
-                if [[ -z $(command -v curl) ]]; then
-                    echo "curl not found, installing..."
-                    sudo apt-get install -y curl
-                fi
                 ;;
             *)
                 echo "$NAME is not supported!" >&2
@@ -49,7 +50,8 @@ case "$OSTYPE" in
         esac
         ;;
     darwin*)
-        version=pkg
+        # We don't check for curl as macOS should have a system version
+        package=powershell-6.0.0-alpha.9.pkg
         ;;
     *)
         echo "$OSTYPE is not supported!" >&2
@@ -57,15 +59,12 @@ case "$OSTYPE" in
         ;;
 esac
 
-info=$(get_info $version)
+curl -L -o "$package" $(get_url "$package")
 
-# Parses $info for asset ID and package name
+if [[ ! -r "$package" ]]; then
-read asset package <<< $(echo $info | sed 's/[,"]//g' | awk '{ print $2; print $4 }')
+    echo "ERROR: $package failed to download! Aborting..." >&2
-
+    exit 1
-# Downloads asset to file
+fi
-packageuri=$(curl -s -i -H 'Accept: application/octet-stream' https://api.github.com/repos/PowerShell/PowerShell/releases/assets/$asset |
-    grep location | sed 's/location: //g')
-curl -C - -s -o $package ${packageuri%$'\r'}
 
 # Installs PowerShell package
 case "$OSTYPE" in
@@ -88,22 +87,35 @@ case "$OSTYPE" in
                         ;;
                 esac
                 echo "Installing $libicupackage, libunwind8, and $package with sudo ..."
-                sudo apt-get install -y libunwind8 $icupackage
+                sudo apt-get install -y libunwind8 "$icupackage"
                 sudo dpkg -i "./$package"
                 ;;
             *)
         esac
         ;;
     darwin*)
+        if hash brew 2>/dev/null; then
+            if [[ ! -d $(brew --prefix openssl) ]]; then
+               echo "Installing OpenSSL with brew..."
+               if ! brew install openssl; then
+                   echo "ERROR: OpenSSL failed to install! Crypto functions will not work..." >&2
+                   # Don't abort because it is not fatal
+               fi
+            fi
+        else
+            echo "ERROR: brew not found! OpenSSL may not be available..." >&2
+            # Don't abort because it is not fatal
+        fi
+
         echo "Installing $package with sudo ..."
-        sudo installer -pkg ./$package -target /
+        sudo installer -pkg "./$package" -target /
         ;;
 esac
 
 powershell -noprofile -c '"Congratulations! PowerShell is installed at $PSHOME"'
 success=$?
 
-if [[ $success != 0 ]]; then
+if [[ "$success" != 0 ]]; then
-    echo "ERROR! PowerShell didn't install. Check script output" >&2
+    echo "ERROR: PowerShell failed to install!" >&2
-    exit $success
+    exit "$success"
 fi