Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Windows pathmapping rules #135

Merged
merged 2 commits into from
May 1, 2024

Conversation

lucaseck
Copy link
Contributor

@lucaseck lucaseck commented Apr 29, 2024

What was the problem/requirement? (What/Why)

It was noticed that absolute Windows paths weren't getting mapped at all on the workers. This is because the pathmapping rules were being changed to using double backslashes \\ as part of a dictionary to string conversion which when set in HOUDINI_PATHMAP Houdini converts them internally into double forward slashes //. This meant that the paths would not be mapped as Houdini uses a single forward slash / as the separator when selecting files and folders on Windows.

What was the solution? (How)

All rules now have a find/replace done on them for \\->/

What is the impact of this change?

Windows absolute paths are now mapped correctly by the adaptor

How was this change tested?

  • Created a scene which referenced an absolute texture path and had an absolute path on the output of the Mantra node
  • Ran a submission from a Windows machine to a Linux worker with the current adaptor and saw that the texture was not mapped and the output not found since it was also not mapped
  • Tested again after these changes and saw that the texture was mapped correctly and the output file was found and uploaded as output correctly.

Was this change documented?

No

Is this a breaking change?

No

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@lucaseck lucaseck requested a review from a team as a code owner April 29, 2024 22:22
destination_path = rule.destination_path
rule_dict = rule.to_dict()

# Convert Windows paths to Posix as HOUDINI_PATHMAP turns \\ in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think instead of conditionally calling as_posix() when it's Windows, you can just call it regardless of OS. From the Python docs, it indicates that the function is a member of the parent PurePath. That would simplify the code to something like:

# Houdini uses forward slashes on all OS
source_path = source_path.as_posix()
destination_path = destination_path.as_posix()

Copy link
Contributor Author

@lucaseck lucaseck Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had tried that, but it doesn't seem to work Windows->Linux. I believe what's happening is when the rules are loaded on Linux it defaults to a PurePosixPath underneath which treats the \ as valid path characters and not a separator to swap, so nothing is changed in the path.

The implementation of as_posix does a .replace(sep, "/") where sep is the value of os.path.sep which on Linux is already / so nothing is being replaced.
For example, running on Linux:

>>> from pathlib import PurePath
>>> test_path = "C:\\folder1\\folder2\\output.png"
>>> p_test_path = PurePath(test_path)
>>> p_test_path
PurePosixPath('C:\\folder1\\folder2\\output.png')
>>> p_test_path.as_posix()
'C:\\folder1\\folder2\\output.png'
>>> 

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, then I think doing a .replace(os.path.sep, "/") would be nicer than the if/else on the OS type. That reduces the dependencies on details in the adaptor library, so helps us refactor it.

epmog
epmog previously approved these changes Apr 30, 2024
Signed-off-by: lucaseck <117225985+lucaseck@users.noreply.github.com>
@epmog epmog merged commit d24499d into aws-deadline:mainline May 1, 2024
9 checks passed
@lucaseck lucaseck deleted the fix_windows_pathmapping branch May 1, 2024 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants