Spark Serving v2#530

Merged

mhamilton723 merged 4 commits intomicrosoft:masterfrom

mhamilton723:serving-refactor

Apr 6, 2019

Contributor

mhamilton723 commented Apr 4, 2019

No description provided.


          Spark Serving v2

269c43c

mhamilton723 commented

View reviewed changes

src/io/http/src/main/scala/HTTPSchema.scala Outdated Show resolved Hide resolved

mhamilton723 commented

View reviewed changes

src/io/http/src/main/scala/HTTPSourceV2.scala Outdated Show resolved Hide resolved

mhamilton723 commented

View reviewed changes

src/io/http/src/main/scala/HTTPSourceV2.scala Outdated Show resolved Hide resolved

mhamilton723 commented

View reviewed changes

src/io/http/src/main/scala/HTTPSourceV2.scala Outdated Show resolved Hide resolved

mhamilton723 commented

View reviewed changes

src/io/http/src/test/scala/ContinuousHTTPSuite.scala Outdated Show resolved Hide resolved

Contributor

mmlspark-bot commented Apr 4, 2019

FAIL Fail! — The build has failed. (269c43c1)


          Fixes for build

c01b01b

Contributor

mmlspark-bot commented Apr 4, 2019

FAIL Fail! — The build has failed. (c01b01b5)

Contributor

mmlspark-bot commented Apr 5, 2019

PASS Pass! — The build has succeeded. (c01b01b5)

MMLSpark 0.16.dev14+2.gc01b01b

This is a build for Github PR #530, changes:

[c01b01b] Mark Hamilton Fixes for build
[269c43c] Mark Hamilton Spark Serving v2

Source: mhamilton723/mmlspark,
serving-refactor at revision c01b01b
(by Mark Hamilton marhamil@microsoft.com).
Build: MMLSpark, 21262728
(built by elivm on elbarzil-vm, 2019-04-04 23:06)
Info:
0.16.dev14+2.gc01b01b: mhamilton723/mmlspark/serving-refactor@c01b01b5; MMLSpark#21262728
Queued by:
Mark Hamilton for Mark Hamilton
Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.16.dev14+2.gc01b01b and --repositories https://mmlspark.azureedge.net/maven.
PIP package uploaded.
R package uploaded.
HDInsight: Copy the link to this Script Action to setup this build on a cluster.
ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).
- HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).
- GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).
- Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as
```
./deploy-arm.sh ... -p <your-parameters>
```
Documentation uploaded.

eisber reviewed

View reviewed changes

src/io/http/src/main/scala/HTTPSource.scala

-                      idRow.update(1, null)
+                      val idRow = new GenericInternalRow(3)
+                      idRow.update(0, null)
+                      idRow.update(1, UTF8String.fromString(id.toString))

Collaborator

eisber Apr 5, 2019

UTF8String.fromString(id.toString) looks expensive. what type is id?

Contributor Author

mhamilton723 Apr 5, 2019

its a long, for some reason Spark's internal row construction needs this UTF8 string instead of a regular string. The HTTP Source is mainly good for debugging though and will be deprecated for v2 stuff soon though

src/io/http/src/main/scala/HTTPSourceV2.scala Outdated Show resolved Hide resolved

src/io/http/src/main/scala/HTTPSourceV2.scala Outdated Show resolved Hide resolved

src/io/http/src/test/scala/ContinuousHTTPSuite.scala Outdated Show resolved Hide resolved


          PR comments

7d86506

Contributor

mmlspark-bot commented Apr 5, 2019

FAIL Fail! — The build has failed. (7d86506a)


          Fixes for build

653af78

Contributor

mmlspark-bot commented Apr 5, 2019

PASS Pass! — The build has succeeded. (653af78b)

MMLSpark 0.16.dev14+4.g653af78

This is a build for Github PR #530, changes:

[653af78] Mark Hamilton Fixes for build
[7d86506] Mark Hamilton PR comments
[c01b01b] Mark Hamilton Fixes for build
[269c43c] Mark Hamilton Spark Serving v2

Source: mhamilton723/mmlspark,
serving-refactor at revision 653af78
(by Mark Hamilton marhamil@microsoft.com).
Build: MMLSpark, 21280164
(built by elivm on elbarzil-vm, 2019-04-05 17:28)
Info:
0.16.dev14+4.g653af78: mhamilton723/mmlspark/serving-refactor@653af78b; MMLSpark#21280164
Queued by:
Mark Hamilton for Mark Hamilton
Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.16.dev14+4.g653af78 and --repositories https://mmlspark.azureedge.net/maven.
PIP package uploaded.
R package uploaded.
HDInsight: Copy the link to this Script Action to setup this build on a cluster.
ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).
- HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).
- GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).
- Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as
```
./deploy-arm.sh ... -p <your-parameters>
```
Documentation uploaded.

mhamilton723 merged commit 5d3aed1 into microsoft:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet