Produce two artifacts in Spark for scala_2.11 and scala_2.12 #85
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kxu1026
reviewed
Aug 2, 2021
kxu1026
reviewed
Aug 2, 2021
transportable-udfs-examples/transportable-udfs-example-udfs/build.gradle
Outdated
Show resolved
Hide resolved
wmoustafa
reviewed
Aug 5, 2021
transportable-udfs-examples/transportable-udfs-example-udfs/build.gradle
Outdated
Show resolved
Hide resolved
transportable-udfs-plugin/src/main/java/com/linkedin/transport/plugin/Defaults.java
Show resolved
Hide resolved
HotSushi
force-pushed
the
master
branch
2 times, most recently
from
9abb063 to
ade0f04
Compare
|
It seems travis-ci validation did not work. Could you double check by building the |
wmoustafa
approved these changes
Aug 6, 2021
s-pedamallu
pushed a commit
to lyft/transport
that referenced
this pull request
Aug 12, 2021
wmoustafa
pushed a commit
that referenced
this pull request
Oct 12, 2021
* Disable Jacoco for platform tests (#37) * 0.0.46 release (previous 0.0.45) + release notes updated [ci skip] * Fixed presto UDF patch broken link (#41) Co-authored-by: Sushant Raikar <sraikar@sraikar-mn2.linkedin.biz> * FileSystemUtils: remove an unreliable check for unit testing (#42) Why: we don't need it What changed: instead of checking configuration for hints on whether we are unit testing, we trust the URI's protocol Tests performed: ./gradlew build * 0.0.47 release (previous 0.0.46) + release notes updated [ci skip] * Presto: Pass custom configuration object when using FileSystemUtils (#43) * 0.0.48 release (previous 0.0.47) + release notes updated [ci skip] * Presto: Make ScalarFunctionImplementation state independent of StdUdfWrapper (#44) * 0.0.49 release (previous 0.0.48) + release notes updated [ci skip] * Upgrade to PrestoSQL 333 (#45) Some major changes: - `SqlFunction`, `SqlScalarFunction` and `ScalarFunctionImplementation` have evolved in trinodb/trino#1764 - `Metadata::getScalarFunctionImplementation` evolved in trinodb/trino#1039 - Type signature parser was moved to presto-main in trinodb/trino#1738 * 0.0.50 release (previous 0.0.49) + release notes updated [ci skip] * Add support for StdFloat, StdDouble, and StdBinary (#46) * Introduce StdFloat, StdDouble, and StdBinary interfaces * Add implementations of those interfaces in Avro, Hive, Presto, Spark, and Generic type systems * Add examples of transport UDFs on those new types, and add tests for those UDFs * Update documentation * 0.0.51 release (previous 0.0.50) + release notes updated [ci skip] * Allow users to override main and test source set names, output directories * 0.0.52 release (previous 0.0.51) + release notes updated [ci skip] * Empty commit to release new version * Empty commit to release new version [ci skip-compare-publications] * 0.0.53 release (previous 0.0.52) + release notes updated [ci skip] * Plugin: Publish Presto thin jar which allows consumers to control dependency graph (#49) * 0.0.54 release (previous 0.0.53) + release notes updated [ci skip] * Hive: Struct data should not be converted to object array during StdStruct creation (#50) * Remove slf4j-log4j12 from Transport dependency graph (#51) * Bump shipkit (#54) * 0.0.55 release (previous 0.0.54) + release notes updated [ci skip] * Fix test SQL generation for binary inputs (#55) * 0.0.56 release (previous 0.0.55) + release notes updated [ci skip] * Spark: Create index-based iterator for non-mutable map keySet and values access (#58) * 0.0.57 release (previous 0.0.56) + release notes updated [ci skip] * Avro: Support simple union schemas (#60) * 0.0.58 release (previous 0.0.57) + release notes updated [ci skip] * Support conversion of String type to Utf8 in AvroWrapper (#61) * 0.0.59 release (previous 0.0.58) + release notes updated [ci skip] * Add Avro ENUM read support and fix String bug (#62) Co-authored-by: Raymond Lam <ralam@linkedin.com> * 0.0.60 release (previous 0.0.59) + release notes updated [ci skip] * Build: Fail if there are checkstyle violations (#64) - Change the Checkstyle severity level from warning to error - Eliminate all existing checkstyle violations Co-authored-by: Carl Steinbach <cwsteinbach@gmail.com> * 0.0.61 release (previous 0.0.60) + release notes updated [ci skip] * Add travis-build.sh for pre-commit testing from command line (#65) * Add travis-build.sh for pre-commit testing from command line * Fix name of build file in comment Co-authored-by: Carl Steinbach <cwsteinbach@gmail.com> Co-authored-by: Shardul Mahadik <smahadik@linkedin.com> * Upgrade to Gradle 6.7 (#67) * Support builds with platform specific JDK (#69) * Bump Avro dependency to 1.10.2 (from 1.7.7). (#71) There doesn't seem to be any impact to the code. gradle build passes. * Migrate from PrestoSQL to Trino (#68) * Automate artifact publication to Maven Central (#72) * Update ci.yml java version to 8 (#77) skip release * Fix org.pentaho:pentaho-aggdesigner-algorithm sunset problem (#78) * Remove travis build in favor of github actions (#87) * Add scala_2.11 and scala_2.12 support (#85) * Update ci.yml to also build the udf-examples folder (#90) Co-authored-by: Malini Mahalakshmi Venkatachari <malvenkatachari@linkedin.com> * Fix running multiple builds in run step in workflow action (#92) Co-authored-by: Malini Mahalakshmi Venkatachari <malvenkatachari@linkedin.com> * A solution to fix running multiple UDFs in Spark issue (#93) Co-authored-by: Kai Xu <kxu@kxu-mn3.linkedin.biz> Co-authored-by: Shardul Mahadik <smahadik@linkedin.com> Co-authored-by: shipkit-org <shipkit.org@gmail.com> Co-authored-by: Sushant Raikar <sraikar@wish.com> Co-authored-by: Sushant Raikar <sraikar@sraikar-mn2.linkedin.biz> Co-authored-by: Suren Nihalani <1093911+SurenNihalani@users.noreply.github.com> Co-authored-by: Xingyuan Lin <xinlin@linkedin.com> Co-authored-by: Khai Tran <46727493+khaitranq@users.noreply.github.com> Co-authored-by: John Joyce <jjoyce6@nd.edu> Co-authored-by: Raymond <13109642+raymondlam12@users.noreply.github.com> Co-authored-by: curtiscwang <cwang89@gmail.com> Co-authored-by: Raymond Lam <ralam@linkedin.com> Co-authored-by: Carl Steinbach <cwsteinbach+github@gmail.com> Co-authored-by: Carl Steinbach <cwsteinbach@gmail.com> Co-authored-by: Akshay Rai <akrai@linkedin.com> Co-authored-by: Sreeram Ramachandran <sramachandran@linkedin.com> Co-authored-by: Raymond Zhang <razhang@linkedin.com> Co-authored-by: KAI XU <kxu@linkedin.com> Co-authored-by: Sushant Raikar <sraikar@linkedin.com> Co-authored-by: Malini Mahalakshmi Venkatachari <malvenkatachari@linkedin.com> Co-authored-by: Kai Xu <kxu@kxu-mn3.linkedin.biz>
wmoustafa
pushed a commit
that referenced
this pull request
Oct 13, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes that this PR brings:
Outcome of this PR is that:
Testing can be done:
How it works?
Scala plugin's compileScala task automatically configures the classpath required to build scala code. It does so by looking for "scala-library.jar" in all dependencies declared in the compile classpath. If a module declares dependency on all scala_2.12 artifacts, the "scala-library.jar" that gets inferred is for scala_2.12. Similarly for building scala_2.11 artifact, all dependencies should be built against scala_2.11. In this PR we create separate modules for scala_2.11 and scala_2.12 with appropriately versioned dependencies.
cc: @shardulm94 @wmoustafa