[KYUUBI #6803][AUTHZ] Add Paimon authz plan-level, row-filter and data-mask tests - #7596
Draft
bowenliang123 wants to merge 3 commits into
Draft
[KYUUBI #6803][AUTHZ] Add Paimon authz plan-level, row-filter and data-mask tests#7596bowenliang123 wants to merge 3 commits into
bowenliang123 wants to merge 3 commits into
Conversation
bowenliang123
marked this pull request as draft
July 29, 2026 14:08
bowenliang123
changed the title
…st coverage to Iceberg parity Test-only changes; no production code touched. * New PaimonCatalogPrivilegesBuilderSuite mirrors IcebergCatalogPrivilegesBuilderSuite: DeleteFromPaimonTable, UpdatePaimonTable, MergeIntoPaimonTable (with PK target), PaimonCallProcedure, plus 30 inherited V2 plan tests (Paimon-specific TRUNCATE and dynamic partition overwrite skipped via the new supportsTruncateTable / supportsOverwritePartitionsDynamic flags). * New RowFilteringForPaimonSuite and DataMaskingForPaimonSuite mirror the Iceberg counterparts; gated by @PaimonTest. * PaimonCatalogRangerSparkExtensionSuite: rename the typo'd 'Producers' test to 'Procedures' so it matches the Iceberg CALL pattern. * PaimonCommands spec builder: propagate 'comment = Paimon' to all four tableDescs and regenerate the table_command_spec.json golden file. * V2CommandsPrivilegesSuite base: add supportsTruncateTable and supportsOverwritePartitionsDynamic (default true) following the existing supportsX flag pattern; consumed by the Paimon subclass only.
bowenliang123
force-pushed
the
improve-paimon-authz
branch
from
July 29, 2026 14:51
613f2b1 to
da56176
Compare
…es on supported versions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
PaimonCatalogRangerSparkExtensionSuiteis the only Paimon test layer in Spark authz; the three Iceberg-specific layers (*PrivilegesBuilderSuite,RowFilteringFor*,DataMaskingFor*) are absent. A regression intable_command_spec.jsonor the Paimon extractor therefore surfaces only as a wrong error message in the e2e suite, not as a precise plan-level diff, and Paimon users running with row-filter and data-masking policies get no coverage at all.What this PR does
PaimonCatalogPrivilegesBuilderSuite: four Paimon-specific plan tests (DeleteFromPaimonTable,UpdatePaimonTable,MergeIntoPaimonTable,PaimonCallProcedure) plus the 30 inherited V2 plan tests. Paimon TRUNCATE and dynamic partition overwrite skip via two new opt-out flags on the shared base.RowFilteringForPaimonSuiteandDataMaskingForPaimonSuitemirroring the Iceberg counterparts.Producerstest toProcedures(typo) and propagatecomment = "Paimon"to all fourtableDescsfor consistency with Iceberg / Hudi / Delta; regenerate the spec JSON.V2CommandsPrivilegesSuite: addsupportsTruncateTableandsupportsOverwritePartitionsDynamic(defaulttrue) following the existingsupportsXflag pattern.No production code is changed.
How was this patch tested?
Default Spark 3.5 / Paimon 0.8.2 / Scala 2.12 build, one Paimon suite per command plus regression runs against the two existing plan suites that share
V2CommandsPrivilegesSuite:mvn -pl extensions/spark/kyuubi-spark-authz \ -Pspark-3.5,hive-provided,spark-provided,flink-provided \ -Dtest=none -DwildcardSuites=... testCancelations are intentional:
Proceduresis gated on Spark 3.4+, Paimon has no permanent-view support,SetNamespaceLocationis hive-catalog-only.RowFilteringForPaimonSuiteandDataMaskingForPaimonSuiteare also gated onisScalaV212 || isSparkV40OrGreater(same rule asPaimonCatalogRangerSparkExtensionSuite). On Scala 2.13 + Spark < 4.0, Paimon 0.8.2 ships generatedpaimon-codegen/scala/Serializable.classthat references the Scala 2.12scala.Serializabletrait, removed in Scala 2.13 — touchingSparkCatalogtriggersNoClassDefFoundError: scala/Serializable. On Spark 4.0+ Paimon also requiresPaimonSparkSessionExtensionsto be registered. The two suites therefore skip test registration andbeforeAll/afterAllon unsupported configurations, matching the existing Paimon pattern.Spotless across all six Spark profiles:
mvn spotless:apply \ -Pflink-provided,hive-provided,spark-provided,spark-4.2,spark-4.1,spark-4.0,spark-3.5,spark-3.4,spark-3.3,tpcds,kubernetes-itWas this patch authored or co-authored using generative AI tooling?
Yes.
Assisted-by: MiniMax-M3