HADOOP-19471. ABFS: Support Fixed SAS Token at Container Level #7461
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manika137
changed the title
|
💔 -1 overall
This message was automatically generated. |
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
c5b14bb to
4a182ac
Compare
|
🎊 +1 overall
This message was automatically generated. |
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
Outdated
Show resolved
Hide resolved
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
Test Results============================================================
|
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
b2cb26d to
6915d0e
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| @@ -150,7 +150,9 @@ public void testBothProviderFixedTokenConfigured() throws Exception { | |||
| * Helper method to get the Fixed SAS token value | |||
| */ | |||
| private String getFixedSASToken(AbfsConfiguration config) throws Exception { | |||
| return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), "read"); | |||
| String readPermission = "read"; | |||
There was a problem hiding this comment.
this should be a static string used as a constant.
anmolanmol1234
approved these changes
Mar 11, 2025
| testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS); | ||
|
|
||
| // Assert that Container Specific Fixed SAS is used | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("sr=c"); |
There was a problem hiding this comment.
Please add description for this.
|
|
||
| // Assert that Account Specific Fixed SAS is used if container SAS isn't set | ||
| testAbfsConfig.unset(containerProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getFileSystemName(), this.getAccountName())); | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("ss=bf"); |
There was a problem hiding this comment.
Same as above, please add description in all the assert.
| //Assert that Account-Agnostic fixed SAS is used if no other fixed SAS configs are set. | ||
| // The token is the same as the Account Specific Fixed SAS. | ||
| testAbfsConfig.unset(accountProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getAccountName())); | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("ss=bf"); |
| * Helper method to get the Fixed SAS token value | ||
| */ | ||
| private String getFixedSASToken(AbfsConfiguration config) throws Exception { | ||
| return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), |
There was a problem hiding this comment.
Please review the format. This line has too many characters.
| @@ -322,6 +322,10 @@ public static String accountProperty(String property, String account) { | |||
| return property + "." + account; | |||
| } | |||
|
|
|||
| public static String containerProperty(String property, String fsName, String account) { | |||
| return property + "." + fsName + "." + account; | |||
There was a problem hiding this comment.
We can use DOT constant here (AbfsHttpConstants)
| * @return Container-specific configuration key | ||
| */ | ||
| public String containerConf(String key) { | ||
| return key + "." + fsName + "." + accountName; |
There was a problem hiding this comment.
We can use DOT constant here (AbfsHttpConstants)
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
0969033 to
f4c6458
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anujmodi2021
requested changes
Mar 12, 2025
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Outdated
Show resolved
Hide resolved
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Show resolved
Hide resolved
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Show resolved
Hide resolved
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Outdated
Show resolved
Hide resolved
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
22851b7 to
a800cda
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anujmodi2021
requested changes
Mar 27, 2025
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
f87a493 to
0e1ce6b
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
bhattmanish98
approved these changes
Mar 31, 2025
YanivKunda
pushed a commit
to YanivKunda/hadoop
that referenced
this pull request
Apr 1, 2025
…he#7461) Contributed by Manika Joshi Reviewed by Anuj Modi, Anmol Asrani, Manish Bhatt Signed off by Anuj Modi <anujmodi@apache.org>
manika137
added a commit
to ABFSDriver/AbfsHadoop
that referenced
this pull request
Apr 10, 2025
…he#7461) Contributed by Manika Joshi Reviewed by Anuj Modi, Anmol Asrani, Manish Bhatt Signed off by Anuj Modi <anujmodi@apache.org>
manika137
added a commit
to ABFSDriver/AbfsHadoop
that referenced
this pull request
Apr 10, 2025
…he#7461) Contributed by Manika Joshi Reviewed by Anuj Modi, Anmol Asrani, Manish Bhatt Signed off by Anuj Modi <anujmodi@apache.org>
anujmodi2021
pushed a commit
that referenced
this pull request
Apr 14, 2025
… (#7597) Contributed by Manika Joshi Reviewed by Anuj Modi, Anmol Asrani, Manish Bhatt Signed off by Anuj Modi <anujmodi@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The ABFS driver currently lacks support for multiple SAS tokens for the same storage account across different containers.
Introducing this support through this PR.
To use fixed SAS token at container level the configuration to be used is:
fs.azure.sas.fixed.token.<container-name>.<storage-account-name>How was this patch tested?
Adding the test results in the comments below.