Skip to content

[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference. #4544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference. #4544

wants to merge 1 commit into from

Conversation

yhuai
Copy link
Contributor

@yhuai yhuai commented Feb 12, 2015

No description provided.

@yhuai yhuai mentioned this pull request Feb 12, 2015
Closed
@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27309 has started for PR 4544 at commit 6e2ffc2.

  • This patch merges cleanly.

@rxin
Copy link
Contributor

rxin commented Feb 12, 2015

lgtm - assuming tests pass.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27309 has finished for PR 4544 at commit 6e2ffc2.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27309/
Test FAILed.

@rxin
Copy link
Contributor

rxin commented Feb 12, 2015

cc @mengxr this broke MLlib... thoughts?

@davies
Copy link
Contributor

davies commented Feb 12, 2015

I think some Integers are dumped into JSON, then come out as LongType, then break some APIs.

@mengxr
Copy link
Contributor

mengxr commented Feb 12, 2015

Yes, we use row.getInt() in MLlib. I think we should use json4s directly in MLlib's export/import.

@mengxr
Copy link
Contributor

mengxr commented Feb 12, 2015

@yhuai I created a JIRA to use json4s directly in MLlib: https://issues.apache.org/jira/browse/SPARK-5757. Could you make a JIRA for this PR and link them?

@mengxr mengxr mentioned this pull request Feb 12, 2015
Closed
@yhuai yhuai changed the title [SQL] Use LongType as the default type for integers in JSON schema inference. [SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference. Feb 12, 2015
@yhuai
Copy link
Contributor Author

yhuai commented Feb 12, 2015

@mengxr Sure. I have created https://issues.apache.org/jira/browse/SPARK-5758.

asfgit pushed a commit that referenced this pull request Feb 12, 2015
@mengxr
[SPARK-5757][MLLIB] replace SQL JSON usage in model import/export by …
e26c149 
…json4s

This PR detaches MLlib model import/export code from SQL's JSON support, and hence unblocks #4544 . yhuai

Author: Xiangrui Meng <meng@databricks.com>

Closes #4555 from mengxr/SPARK-5757 and squashes the following commits:

b0415e8 [Xiangrui Meng] replace SQL JSON usage by json4s

(cherry picked from commit 99bd500)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
asfgit pushed a commit that referenced this pull request Feb 12, 2015
@mengxr
[SPARK-5757][MLLIB] replace SQL JSON usage in model import/export by …
99bd500 
…json4s

This PR detaches MLlib model import/export code from SQL's JSON support, and hence unblocks #4544 . yhuai

Author: Xiangrui Meng <meng@databricks.com>

Closes #4555 from mengxr/SPARK-5757 and squashes the following commits:

b0415e8 [Xiangrui Meng] replace SQL JSON usage by json4s
@yhuai
Copy link
Contributor Author

yhuai commented Feb 12, 2015

test this please

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27365 has started for PR 4544 at commit 6e2ffc2.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27365 has finished for PR 4544 at commit 6e2ffc2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27365/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 12, 2015
@yhuai @marmbrus
[SPARK-5758][SQL] Use LongType as the default type for integers in JS…
b0c79da 
…ON schema inference.

Author: Yin Huai <yhuai@databricks.com>

Closes #4544 from yhuai/jsonUseLongTypeByDefault and squashes the following commits:

6e2ffc2 [Yin Huai] Use LongType as the default type for integers in JSON schema inference.

(cherry picked from commit c352ffb)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in c352ffb Feb 12, 2015
@yhuai yhuai deleted the jsonUseLongTypeByDefault branch February 13, 2015 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@yhuai @SparkQA @rxin @AmplabJenkins @davies @mengxr