HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation #1785
Conversation
|
LGTM; one minor tweak and then it's good to go in. Troubleshooting docs are, provided kept current, wonderful |
|
🎊 +1 overall
This message was automatically generated. |
| @@ -857,6 +857,46 @@ signon page for humans, even though it is a machine calling. | |||
| 1. The URL is wrong —it is pointing at a web page unrelated to OAuth2.0 | |||
| 1. There's a proxy server in the way trying to return helpful instructions. | |||
|
|
|||
| ### `java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/user1/.staging is not as expected. It is owned by <principal_id>. The directory must be owned by the submitter user1 or user1` | |||
|
|
|||
There was a problem hiding this comment.
There are a couple of options to deal with the issue like deleting staging dir before running the job or changing the staging directory in config before running the job. The configs identity.transformer listed here are also providing a workaround specific to ABFS driver to let the client assume the ownership is with current local user.
As this is not really a store issue, please reword to highlight that this is a workaround.
There was a problem hiding this comment.
@snvijaya The options suggested are not a suitable fix for the exception. I think identity.transformer is more of a fix rather to be considered as a workaround. I have added a short message on the fix in my last commit. Let me know for any further comments?
| </description> | ||
| </property> | ||
| <property> | ||
| <name>fs.azure.use.upn</name> |
There was a problem hiding this comment.
This config is not required to be set as it does not contribute to the conversion of the AAD OID to local user. Can be removed from this PR.
There was a problem hiding this comment.
@snvijaya Removed this config in the new commit
|
🎊 +1 overall
This message was automatically generated. |
|
@steveloughran @snvijaya - Please can you review the changes and merge if you're happy? Thanks! |
DadanielZ
added
the
fs/azure
There was a problem hiding this comment.
LGTM, +1.
@karthick-rn for this PR could you create a subtask under HADOOP-15763 so I can help to commit using its JIRA id?
DadanielZ
changed the title
DadanielZ
changed the title
DadanielZ
changed the title
|
committed, thank you @karthick-rn |
In this PR, I have updated the
abfs.mdto include some key configurations required when submitting jobs to YARN using hadoop-azure module. Currently, these properties aren't documented anywhere in Apache Hadoop. I have them included under 'Troubleshooting' section ofabfs.md, please review and also let me know if you think they need to be documented elsewhere?As the PR is only an update to the existing doc I have not opened any JIRA ticket, let me know if you think a JIRA is still required?
Many thanks!