Skip to content

RANGER-2128: Implementation of Ranger Spark SQL plugin #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 35 commits into from

Conversation

yaooqinn
Copy link
Member

@yaooqinn yaooqinn commented Jun 26, 2018

pre-work

Basic concepts and introductions can be found in spark-authorizer's documentations.

additionals

apache/spark#17724 exposed a new experimental develop api SparkSessionExetensions, which is able to add user supplied extensions to SparkSession object during instantiation via program api or the spark property named spark.sql.extensions.

This PR uses spark.sql.extensions and other necessary ranger-hive-plugin settings to enable Ranger security support for Spark SQL with hive as external catalog.

spark.sql.extensions=
org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension

@yaooqinn yaooqinn changed the title [WIP][RANGER-2128]Implementation of Spark SQL plugin with hive as external catalog RANGER-2128: Implementation of Spark SQL plugin with hive as external catalog Jun 27, 2018
@tooptoop4
Copy link

@boscodurai ship it?

@tooptoop4
Copy link

keen to see this in action! do you have any jar and setup notes for this?

@tooptoop4
Copy link

@yaooqinn can u create apache review?

@yaooqinn
Copy link
Member Author

@tooptoop4 review request created.

@yaooqinn yaooqinn changed the title RANGER-2128: Implementation of Spark SQL plugin with hive as external catalog RANGER-2128: Implementation of Ranger Spark SQL plugin May 19, 2019
@ptallada
Copy link

ptallada commented Apr 8, 2020

Hi!

This work is very interesting!
Any progress on the merge?

@rimolive
Copy link

rimolive commented Apr 27, 2020

@yaooqinn Can you fix the conflicts?

@shgriffi
Copy link

@yaooqinn any thoughts on resolving the conflicts?

@simonvanderveldt
Copy link

simonvanderveldt commented Nov 25, 2022

What does this PR do/accomplish that isn't already possible with the existing Hive support? We're currently running Spark Thriftserver (3.2.x) with the kyuubi plugin against Ranger where in Ranger we've defined the service as a Hive service and everything with regards to authentication and authorization seems to be working as expected.

The only thing that I've observed that doesn't work is the auto-complete when creating policies via the Ranger UI, I assume this is a slight dialect difference in the response from the Spark Thriftserver vs a real HiveServer2 since the query being run by Ranger (show databases like "*") returns the databases just fine when I run it myself.

@bhavikpatel9977
Copy link
Contributor

What does this PR do/accomplish that isn't already possible with the existing Hive support? We're currently running Spark Thriftserver (3.2.x) with the kyuubi plugin against Ranger where in Ranger we've defined the service as a Hive service and everything with regards to authentication and authorization seems to be working as expected.

The only thing that I've observed that doesn't work is the auto-complete when creating policies via the Ranger UI, I assume this is a slight dialect difference in the response from the Spark Thriftserver vs a real HiveServer2 since the query being run by Ranger (show databases like "*") returns the databases just fine when I run it myself.

Thanks @simonvanderveldt. I think than we can include Kyuubi plugin in Ranger for Spark.
If you are familiar with Kyuubi than Can you please raise demo PR?

@waywtdcc
Copy link

+1

@ManoharVanam
Copy link

will this work with spark-submit cluster mode without passing keytab?

@yaooqinn
Copy link
Member Author

I am going to close this in favor of the kyuubi spark authz plugin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants