Skip to content

Commit

Permalink
[KYUUBI #5427] [AUTHZ] Shade spark authz plugin
Browse files Browse the repository at this point in the history
### _Why are the changes needed?_

This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5427 from Yikf/shade-authz.

Closes #5427

d2f7ea8 [yikaifei] fix
695133d [Kent Yao] Update docs/security/authorization/spark/install.md
f3a6531 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab3 [yikaifei] bundle
2068c98 [yikaifei] relocation
6c6e50e [yikaifei] Shade spark authz plugin

Lead-authored-by: yikaifei <yikaifei@apache.org>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: yikaifei <yikaifei@apache.org>
  • Loading branch information
2 people authored and davidyuan1223 committed Oct 26, 2023
1 parent 32028f9 commit 3af5ed1
Show file tree
Hide file tree
Showing 4 changed files with 332 additions and 1 deletion.
13 changes: 13 additions & 0 deletions docs/security/authorization/spark/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,19 @@ After a while, if everything goes well, you will get the plugin finally in two p
- The main plugin jar, which is under `./extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_${scala.binary.version}-${project.version}.jar`
- The least transitive dependencies needed, which are under `./extensions/spark/kyuubi-spark-authz/target/scala-${scala.binary.version}/jars`

## Build shaded jar with Apache Maven

Apache Kyuubi also provides the shaded jar for the Spark AuthZ plugin, You can run the AuthZ plugin using just a shaded jar without the additional dependency of jars,
To build it, `cd` to the root direct of kyuubi project and run:

```shell
build/mvn clean package -pl :kyuubi-spark-authz-shaded_2.12 -DskipTests -am
```

After a while, if everything goes well, you will get the plugin finally:

- The shaded AuthZ plugin jar, which is under `./extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_${scala.binary.version}-${project.version}.jar`

### Build against Different Apache Spark Versions

The maven option `spark.version` is used for specifying Spark version to compile with and generate corresponding transitive dependencies.
Expand Down
2 changes: 1 addition & 1 deletion docs/security/authorization/spark/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

## Install

With the `kyuubi-spark-authz_*.jar` and its transitive dependencies available for spark runtime classpath, such as
Use either the shaded jar `kyuubi-spark-authz-shaded_*.jar` or the `kyuubi-spark-authz_*.jar` with its transitive dependencies available for spark runtime classpath, such as
- Copied to `$SPARK_HOME/jars`, or
- Specified to `spark.jars` configuration

Expand Down
317 changes: 317 additions & 0 deletions extensions/spark/kyuubi-spark-authz-shaded/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,317 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-parent</artifactId>
<version>1.9.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

<artifactId>kyuubi-spark-authz-shaded_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Kyuubi Dev Spark Authorization Extension Shaded</name>
<url>https://kyuubi.apache.org/</url>

<properties>
<!-- the following components' version may need to tune to align w/ the ranger.version-->
<gethostname4j.version>1.0.0</gethostname4j.version>
<jersey.client.version>1.19.4</jersey.client.version>
<jna.version>5.7.0</jna.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-common</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-bundle</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugin-classloader</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
<exclusion>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
</exclusion>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
</exclusion>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>${jersey.client.version}</version>
<exclusions>
<exclusion>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
<version>${gethostname4j.version}</version>
</dependency>

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version>
</dependency>

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>${jna.version}</version>
</dependency>

<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
<version>${ranger.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-cred</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.client</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.lucene</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>com.carrotsearch</groupId>
<artifactId>hppc</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.kyuubi:kyuubi-util-scala_${scala.binary.version}</include>
<include>org.apache.kyuubi:kyuubi-spark-authz_${scala.binary.version}</include>
<include>org.apache.kyuubi:kyuubi-util</include>
<include>org.apache.ranger:ranger-plugins-common</include>
<include>org.apache.ranger:ranger-plugins-audit</include>
<include>org.codehaus.jackson:jackson-jaxrs</include>
<include>com.sun.jersey:jersey-client</include>
<include>com.sun.jersey:jersey-core</include>
<include>com.kstruct:gethostname4j</include>
<include>net.java.dev.jna:jna</include>
<include>net.java.dev.jna:jna-platform</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/*.proto</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/DEPENDENCIES</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>LICENSE.txt</exclude>
<exclude>NOTICE.txt</exclude>
<exclude>mozilla/**</exclude>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>org.codehaus.jackson.jaxrs</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.codehaus.jackson.jaxrs</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.jersey</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.jersey</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.ws.rs.ext</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.ws.rs.ext</shadedPattern>
</relocation>
<relocation>
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.hadoop.security</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.apache.hadoop.security</shadedPattern>
<includes>
<include>org.apache.hadoop.security.KrbPasswordSaverLoginModule</include>
<include>org.apache.hadoop.security.SecureClientLogin</include>
<include>org.apache.hadoop.security.SecureClientLoginConfiguration</include>
</includes>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
</transformers>
</configuration>
<executions>
<execution>
<goals>
<goal>shade</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
1 change: 1 addition & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
<module>extensions/server/kyuubi-server-plugin</module>
<module>extensions/spark/kyuubi-extension-spark-jdbc-dialect</module>
<module>extensions/spark/kyuubi-spark-authz</module>
<module>extensions/spark/kyuubi-spark-authz-shaded</module>
<module>extensions/spark/kyuubi-spark-connector-common</module>
<module>extensions/spark/kyuubi-spark-connector-tpcds</module>
<module>extensions/spark/kyuubi-spark-connector-tpch</module>
Expand Down

0 comments on commit 3af5ed1

Please sign in to comment.