Update aml branch.

README.md

@@ -2,9 +2,13 @@
 
 `nimbusml` is a Python module that provides Python bindings for [ML.NET](https://github.com/dotnet/machinelearning). 
 
-ML.NET was originally developed in Microsoft Research and is used across many product groups in Microsoft like Windows, Bing, PowerPoint, Excel and others. `nimbusml` was built to enable data science teams that are more familiar with Python to take advantage of ML.NET's functionality and performance. 
+ML.NET was originally developed in Microsoft Research and is used across many product groups in Microsoft like Windows, Bing, PowerPoint, Excel, and others. `nimbusml` was built to enable data science teams that are more familiar with Python to take advantage of ML.NET's functionality and performance. 
 
-This package enables training ML.NET pipelines or integrating ML.NET components directly into [scikit-learn](https://scikit-learn.org/stable/) pipelines (it supports  `numpy.ndarray`, `scipy.sparse_cst`, and `pandas.DataFrame` as inputs).
+`nimbusml` enables training ML.NET pipelines or integrating ML.NET components directly into [scikit-learn](https://scikit-learn.org/stable/) pipelines. It adheres to existing `scikit-learn` conventions, allowing simple interoperability between `nimbusml` and `scikit-learn` components, while adding a suite of fast, highly optimized, and scalable algorithms, transforms, and components written in C++ and C\#.
+
+See examples below showing interoperability with `scikit-learn`. A more detailed example in the [documentation](https://docs.microsoft.com/en-us/nimbusml/tutorials/b_c-sentiment-analysis-3-combining-nimbusml-and-scikit-learn) shows how to use a `nimbusml` component in a `scikit-learn` pipeline, and create a pipeline using only `nimbusml` components.
+
+`nimbusml` supports `numpy.ndarray`, `scipy.sparse_cst`, and `pandas.DataFrame` as inputs. In addition, `nimbusml` also supports streaming from files without loading the dataset into memory with `FileDataStream`, which allows training on data significantly exceeding memory.
 
 Documentation can be found [here](https://docs.microsoft.com/en-us/NimbusML/overview) and additional notebook samples can be found [here](https://github.com/Microsoft/NimbusML-Samples).
 
@@ -84,7 +88,7 @@ To build `nimbusml` from source please visit our [developer guide](docs/develope
 
 ## Contributing
 
-The contributions guide can be found [here](CONTRIBUTING.md). Given the experimental nature of this project, support will be provided on a best-effort basis. We suggest opening an issue for discussion before starting a PR with big changes.
+The contributions guide can be found [here](CONTRIBUTING.md). 
 
 ## Support
 

build.cmd

@@ -26,6 +26,9 @@ set RunExtendedTests=False
 set BuildDotNetBridgeOnly=False
 set SkipDotNetBridge=False
 set AzureBuild=False
+set BuildManifestGenerator=False
+set UpdateManifest=False
+set VerifyManifest=False
 
 :Arg_Loop
 if [%1] == [] goto :Build
@@ -53,6 +56,10 @@ if /i [%1] == [--skipDotNetBridge]     (
     set SkipDotNetBridge=True
     shift && goto :Arg_Loop
 )
+if /i [%1] == [--updateManifest] (
+    set UpdateManifest=True
+    shift && goto :Arg_Loop
+)
 if /i [%1] == [--azureBuild]     (
     set AzureBuild=True
     shift && goto :Arg_Loop
@@ -68,6 +75,7 @@ echo "  --installPythonPackages           Install python packages after build"
 echo "  --includeExtendedTests            Include the extended tests if the tests are run"
 echo "  --buildDotNetBridgeOnly           Build only DotNetBridge"
 echo "  --skipDotNetBridge                Build everything except DotNetBridge"
+echo "  --updateManifest                  Update manifest.json"
 echo "  --azureBuild                      Building in azure devops (adds dotnet CLI to the path)"
 goto :Exit_Success
 
@@ -173,8 +181,6 @@ if "%AzureBuild%" == "True" (
     echo ##vso[task.prependpath]%_dotnetRoot%
 )
 
-set LOCAL_NUGET_PACKAGES_DIR=.\local-nuget-packages
-
 :: Build managed code
 echo ""
 echo "#################################"
@@ -191,6 +197,37 @@ if "%BuildDotNetBridgeOnly%" == "True" (
 call "%_dotnet%" build -c %Configuration% --force "%__currentScriptDir%src\Platforms\build.csproj"
 call "%_dotnet%" publish "%__currentScriptDir%src\Platforms\build.csproj" --force --self-contained -r win-x64 -c %Configuration%
 
+
+if "%Configuration:~-5%" == "Py3.7" set VerifyManifest=True
+if "%VerifyManifest%" == "True" set BuildManifestGenerator=True
+if "%UpdateManifest%" == "True" set BuildManifestGenerator=True
+
+if "%BuildManifestGenerator%" == "True" (
+    echo ""
+    echo "#################################"
+    echo "Building Manifest Generator... "
+    echo "#################################"
+    call "%_dotnet%" build -c %Configuration% -o "%BuildOutputDir%%Configuration%"  --force "%__currentScriptDir%src\ManifestGenerator\ManifestGenerator.csproj"
+)
+
+if "%UpdateManifest%" == "True" (
+    echo Updating manifest.json ...
+    call "%_dotnet%" "%BuildOutputDir%%Configuration%\ManifestGenerator.dll" create %__currentScriptDir%\src\python\tools\manifest.json
+    echo manifest.json updated.
+    echo Run entrypoint_compiler.py --generate_api --generate_entrypoints to generate entry points and api files.
+    goto :Exit_Success
+)
+
+if "%VerifyManifest%" == "True" (
+    echo Verifying manifest.json ...
+    call "%_dotnet%" "%BuildOutputDir%%Configuration%\ManifestGenerator.dll" verify %__currentScriptDir%\src\python\tools\manifest.json
+    if errorlevel 1 (
+        echo manifest.json is invalid.
+        echo Run build --updateManifest to update manifest.json.
+        goto :Exit_Error
+    )
+)
+
 echo ""
 echo "#################################"
 echo "Downloading Dependencies "
@@ -352,13 +389,13 @@ if "%InstallPythonPackages%" == "True" (
     echo "#################################"
     echo "Installing python packages ... "
     echo "#################################"
-    call "%PythonExe%" -m pip install --upgrade pip
-    call "%PythonExe%" -m pip install --upgrade nose pytest graphviz imageio pytest-cov "jupyter_client>=4.4.0" "nbconvert>=4.2.0"
+    call "%PythonExe%" -m pip install --upgrade "pip==19.3.1"
+    call "%PythonExe%" -m pip install --upgrade nose pytest pytest-xdist graphviz imageio pytest-cov "jupyter_client>=4.4.0" "nbconvert>=4.2.0"
 
     if %PythonVersion% == 2.7 (
         call "%PythonExe%" -m pip install --upgrade pyzmq
     ) else (
-        call "%PythonExe%" -m pip install --upgrade "azureml-dataprep>=1.1.12" 
+        call "%PythonExe%" -m pip install --upgrade "azureml-dataprep>=1.1.33"
     )
 
     call "%PythonExe%" -m pip install --upgrade "%__currentScriptDir%target\%WheelFile%"
@@ -379,27 +416,53 @@ set TestsPath1=%PackagePath%\tests
 set TestsPath2=%__currentScriptDir%src\python\tests
 set TestsPath3=%__currentScriptDir%src\python\tests_extended
 set ReportPath=%__currentScriptDir%build\TestCoverageReport
-call "%PythonExe%" -m pytest --verbose --maxfail=1000 --capture=sys "%TestsPath1%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
-if errorlevel 1 (
-    goto :Exit_Error
-)
-call "%PythonExe%" -m pytest --verbose --maxfail=1000 --capture=sys "%TestsPath2%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
+set NumConcurrentTests=%NUMBER_OF_PROCESSORS%
+
+call "%PythonExe%" -m pytest -n %NumConcurrentTests% --verbose --maxfail=1000 --capture=sys "%TestsPath2%" "%TestsPath1%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
 if errorlevel 1 (
-    goto :Exit_Error
+    :: Rerun any failed tests to give them one more
+    :: chance in case the errors were intermittent.
+    call "%PythonExe%" -m pytest -n %NumConcurrentTests% --last-failed --verbose --maxfail=1000 --capture=sys "%TestsPath2%" "%TestsPath1%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
+    if errorlevel 1 (
+        goto :Exit_Error
+    )
 )
 
 if "%RunExtendedTests%" == "True" (
-    call "%PythonExe%" -m pytest --verbose --maxfail=1000 --capture=sys "%TestsPath3%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
+    call "%PythonExe%" -m pytest -n %NumConcurrentTests% --verbose --maxfail=1000 --capture=sys "%TestsPath3%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
     if errorlevel 1 (
-        goto :Exit_Error
+        :: Rerun any failed tests to give them one more
+        :: chance in case the errors were intermittent.
+        call "%PythonExe%" -m pytest -n %NumConcurrentTests% --last-failed --verbose --maxfail=1000 --capture=sys "%TestsPath3%" --cov="%PackagePath%" --cov-report term-missing --cov-report html:"%ReportPath%"
+        if errorlevel 1 (
+            goto :Exit_Error
+        )
     )
 )
 
 :Exit_Success
+call :CleanUpDotnet
 endlocal
 exit /b %ERRORLEVEL%
 
 :Exit_Error
+call :CleanUpDotnet
 endlocal
 echo Failed with error %ERRORLEVEL%
-exit /b %ERRORLEVEL%
+exit /b %ERRORLEVEL%
+
+:CleanUpDotnet
+:: Save the error level so it can be
+:: restored when exiting the function
+set PrevErrorLevel=%ERRORLEVEL%
+
+:: Shutdown all dotnet persistent servers so that the
+:: dotnet executable is not left open in the background.
+:: As of dotnet 2.1.3 three servers are left running in
+:: the background. This will shutdown them all down.
+:: See here for more info: https://github.com/dotnet/cli/issues/9458
+:: This fixes an issue when re-running the build script because
+:: the build script was trying to replace the existing dotnet
+:: binaries which were sometimes still in use.
+call "%_dotnet%" build-server shutdown
+exit /b %PrevErrorLevel%

build.sh

@@ -175,8 +175,6 @@ then
     echo "Installing dotnet SDK ... "
     curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -Version 2.1.701 -InstallDir ./cli
 
-    export LOCAL_NUGET_PACKAGES_DIR=./local-nuget-packages
-
     # Build managed code
     echo "Building managed code ... "
     _dotnet="${__currentScriptDir}/cli/dotnet"
@@ -284,7 +282,7 @@ then
         exit 1
     fi
     # Review: Adding "--upgrade" to pip install will cause problems when using Anaconda as the python distro because of Anaconda's quirks with pytest.
-    "${PythonExe}" -m pip install nose "pytest>=4.4.0" graphviz "pytest-cov>=2.6.1" "jupyter_client>=4.4.0" "nbconvert>=4.2.0"
+    "${PythonExe}" -m pip install nose "pytest>=4.4.0" pytest-xdist graphviz "pytest-cov>=2.6.1" "jupyter_client>=4.4.0" "nbconvert>=4.2.0"
     if [ ${PythonVersion} = 2.7 ]
     then
         "${PythonExe}" -m pip install --upgrade pyzmq
@@ -294,7 +292,7 @@ then
             "${PythonExe}" -m pip install --upgrade pytest-remotedata
         fi
 
-        "${PythonExe}" -m pip install --upgrade "azureml-dataprep>=1.1.12"
+        "${PythonExe}" -m pip install --upgrade "azureml-dataprep>=1.1.33"
     fi
     "${PythonExe}" -m pip install --upgrade "${Wheel}"
     "${PythonExe}" -m pip install "scikit-learn==0.19.2"
@@ -311,12 +309,25 @@ then
     TestsPath2=${__currentScriptDir}/src/python/tests
     TestsPath3=${__currentScriptDir}/src/python/tests_extended
     ReportPath=${__currentScriptDir}/build/TestCoverageReport
-    "${PythonExe}" -m pytest --verbose --maxfail=1000 --capture=sys "${TestsPath1}"
-    "${PythonExe}" -m pytest --verbose --maxfail=1000 --capture=sys "${TestsPath2}"
+    "${PythonExe}" -m pytest -n 4 --verbose --maxfail=1000 --capture=sys "${TestsPath2}" "${TestsPath1}" || \
+        "${PythonExe}" -m pytest -n 4 --last-failed --verbose --maxfail=1000 --capture=sys "${TestsPath2}" "${TestsPath1}" 
 
     if [ ${__runExtendedTests} = true ]
-    then 
-        "${PythonExe}" -m pytest --verbose --maxfail=1000 --capture=sys "${TestsPath3}"
+    then
+        echo "Running extended tests ... " 
+        if [ ! "$(uname -s)" = "Darwin" ]
+        then 
+            # Required for Image.py and Image_df.py to run successfully on Ubuntu.
+            {
+                apt-get update 
+                apt-get install libc6-dev -y
+                apt-get install libgdiplus -y
+            } || { 
+            # Required for Image.py and Image_df.py to run successfully on CentOS.
+                yum install glibc-devel -y
+            }
+        fi
+        "${PythonExe}" -m pytest -n 4 --verbose --maxfail=1000 --capture=sys "${TestsPath3}"
     fi
 fi
 

build/ci/phase-template.yml

@@ -26,7 +26,8 @@ phases:
     - script: $(_buildScript) --configuration $(_configuration) --runTests $(_testOptions)
   # Mac phases
   - ${{ if eq(parameters.name, 'Mac') }}:
-    - script: brew update && brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/f5b1ac99a7fba27c19cee0bc4f036775c889b359/Formula/libomp.rb mono-libgdiplus gettext && brew link gettext --force
+    # Note: Manual defining of the libomp URL below is needed to avoid error at runtime. Installing using 'brew install libomp' results in "Intel MKL FATAL ERROR: Cannot load libmkl_intel_thread.dylib."
+    - script: brew update && brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/f5b1ac99a7fba27c19cee0bc4f036775c889b359/Formula/libomp.rb gettext && brew link gettext --force && brew unlink python@2 && brew install mono-libgdiplus
     - ${{ if eq(parameters.testDistro, 'noTests') }}:
       - script: chmod 777 $(_buildScript) && $(_buildScript) --configuration $(_configuration)
     - ${{ if eq(parameters.testDistro, '') }}:
@@ -50,7 +51,6 @@ phases:
   # Publish build artifacts
   - ${{ if or(eq(parameters.name, 'Linux_Ubuntu16'), eq(parameters.name, 'Windows'), eq(parameters.name, 'Mac')) }}:
     - task: PublishBuildArtifacts@1
-      condition: and(always(), ne(variables['Build.Reason'], 'PullRequest'))
       displayName: Publish wheel file to VSTS artifacts
       inputs:
         pathToPublish: $(Build.SourcesDirectory)/target

build/libs_linux.txt

@@ -1,3 +1,4 @@
+Google.Protobuf.dll
 Newtonsoft.Json.dll
 libCpuMathNative.so
 libFastTreeNative.so

build/libs_mac.txt

@@ -1,3 +1,4 @@
+Google.Protobuf.dll
 Newtonsoft.Json.dll
 libCpuMathNative.dylib
 libFastTreeNative.dylib

docs/developers/linux-build.md

@@ -12,9 +12,9 @@ Building NimbusML from source on Linux
 ## Build
 Run `./build.sh`
 
-This downloads dependencies (.NET SDK, specific versions of Python and Boost), builds native code and managed code, and packages NimbusML into a pip-installable wheel. This produces debug binaries by default, and release versions can be specified by `./build.sh --configuration RlsLinPy3.7` for examle.
+This downloads dependencies (.NET SDK, specific versions of Python and Boost), builds native code and managed code, and packages NimbusML into a pip-installable wheel. This produces debug binaries by default, and release versions can be specified by `./build.sh --configuration RlsLinPy3.7` for example.
 
 For additional options including running tests and building components independently, see `./build.sh -h`.
 
 ### Known Issues
-The LightGBM estimator fails on Linux when building from source. The official NimbusML Linux wheel package on Pypi.org has a working version of LightGBM.
+The LightGBM estimator fails on Linux when building from source. The official NimbusML Linux wheel package on Pypi.org has a working version of LightGBM.

docs/developers/windows-build.md

@@ -7,6 +7,6 @@ Building NimbusML from source on Windows
 ## Build
 Run `build.cmd`
 
-This downloads dependencies (.NET SDK, specific versions of Python and Boost), builds native code and managed code, and packages NimbusML into a pip-installable wheel. This produces debug binaries by default, and release versions can be specified by `build.cmd --configuration RlsWinPy3.7` for examle.
+This downloads dependencies (.NET SDK, specific versions of Python and Boost), builds native code and managed code, and packages NimbusML into a pip-installable wheel. This produces debug binaries by default, and release versions can be specified by `build.cmd --configuration RlsWinPy3.7` for example.
 
 For additional options including running tests and building components independently, see `build.cmd -?`.

docs/release-notes/release-1.5.0.md

@@ -0,0 +1,101 @@
+# [NimbusML](https://docs.microsoft.com/en-us/nimbusml/overview) 1.5.0
+
+## **New Features**
+
+- **Initial implementation of `csr_matrix` output support.**
+
+    [PR#250](https://github.com/microsoft/NimbusML/pull/250)
+    Add support for data output in `scipy.sparse.csr_matrix` format.
+
+    ```python
+    xf = OneHotVectorizer(columns={'c0':'c0', 'c1':'c1'})
+    xf.fit(train_df)
+    result = xf.transform(train_df, as_csr=True)
+    ```
+
+- **Permutation Feature Importance for model interpretibility.**
+
+    [PR#279](https://github.com/microsoft/NimbusML/pull/279)
+    Adds `permutation_feature_importance()` method to `Pipeline` and
+    predictor estimators, enabling evaluation of model-wide feature
+    importances on any dataset with same schema as the dataset used
+    to fit the `Pipeline`.
+
+    ```python
+    pipe = Pipeline([
+        LogisticRegressionBinaryClassifier(label='label', feature=['feature'])
+    ])
+    pipe.fit(data)
+    pipe.permutation_feature_importance(data)
+    ```
+
+- **Initial implementation of DateTime input and output column support.**
+
+    [PR#290](https://github.com/microsoft/NimbusML/pull/290)
+    Add initial support for input and output of Pandas DateTime columns.
+
+- **Initial implementation of LpScaler.**
+
+    [PR#253](https://github.com/microsoft/NimbusML/pull/253)
+    Normalize vectors (rows) individually by rescaling them to unit norm (L2, L1 or LInf).
+    Performs the following operation on a vector X: Y = (X - M) / D, where M is mean and D
+    is either L2 norm, L1 norm or LInf norm.
+
+- **Add support for variable length vector output.**
+
+    [PR#267](https://github.com/microsoft/NimbusML/pull/267)
+    Support output of columns returned from ML.Net which contain variable length vectors.
+
+- **Save `predictor_model` when pickling a `Pipeline`.**
+
+    [PR#295](https://github.com/microsoft/NimbusML/pull/295)
+
+- **Initial implementation of the WordTokenizer transform.**
+
+    [PR#296](https://github.com/microsoft/NimbusML/pull/296)
+
+- **Add support for summary output from tree based predictors.**
+
+    [PR#298](https://github.com/microsoft/NimbusML/pull/298)
+
+## **Bug Fixes**
+
+- **Fixed `Pipeline.transform()` in transform only `Pipeline` fails if y column is provided **
+
+    [PR#294](https://github.com/microsoft/NimbusML/pull/294)
+    Enable calling `.transform()` on a `Pipeline` containing only transforms when the y column is provided 
+
+- **Fix issue when using `predict_proba` or `decision_function` with combined models.**
+
+    [PR#272](https://github.com/microsoft/NimbusML/pull/272)
+
+- **Fix `Pipeline._extract_classes_from_headers` was not checking for valid steps.**
+
+    [PR#292](https://github.com/microsoft/NimbusML/pull/292)
+
+- **Fix BinaryDataStream was not valid as input for transformer.**
+
+    [PR#307](https://github.com/microsoft/NimbusML/pull/307)
+
+- **Fix casing for the installPythonPackages build.sh argument.**
+
+    [PR#256](https://github.com/microsoft/NimbusML/pull/256)
+
+## **Breaking Changes**
+
+- **Removed `y` parameter from `Pipeline.transform()`**
+
+    [PR#294](https://github.com/microsoft/NimbusML/pull/294)
+    Removed `y` parameter from `Pipeline.transform()` as it is not needed nor used for transforming data with a fitted `Pipeline`.
+
+## **Enhancements**
+
+None.
+
+## **Documentation and Samples**
+
+None. 
+
+## **Remarks**
+
+None.