Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DCE rules for custom_jvp and custom_vjp #25963

Merged
merged 1 commit into from
Jan 24, 2025

Conversation

dfm
Copy link
Collaborator

@dfm dfm commented Jan 17, 2025

All of the custom_* primitives are currently treated as opaque by DCE but, in some cases, it would be useful to make sure that DCE passes through these ops properly. This PR adds DCE rules for custom_jvp and custom_vjp.

Related to #25956

@dfm dfm requested a review from mattjj January 17, 2025 19:20
@dfm dfm self-assigned this Jan 17, 2025
@dfm dfm force-pushed the dce-custom-star branch from 7fc3d95 to 1acc944 Compare January 22, 2025 20:11
@dfm dfm changed the title Add DCE rule for custom_jvp Add DCE rules for custom_jvp and custom_vjp Jan 22, 2025
@dfm
Copy link
Collaborator Author

dfm commented Jan 22, 2025

Based on offline discussions with @mattjj, I updated this PR to include DCE rules for both custom_jvp and custom_vjp. The key simplifying assumption is that DCE happens late enough (when lowering) that we don't actually need to have the DCE rule produce a new custom_* primitive with updated rules. Instead, we can blow away the custom AD, and just return a closed_call primitive with the DCEed jaxpr. I can't think of cases where this would be a problem, but I'll do some testing!

Scratch that: it looks like this will break higher order remat!

@dfm dfm added the pull ready Ready for copybara import and testing label Jan 22, 2025
@dfm dfm force-pushed the dce-custom-star branch from 1acc944 to 62b8f17 Compare January 22, 2025 20:56
@dfm dfm force-pushed the dce-custom-star branch from 62b8f17 to 28d5733 Compare January 23, 2025 20:23
Copy link
Collaborator

@mattjj mattjj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Dan!

@copybara-service copybara-service bot merged commit 7a23d1d into jax-ml:main Jan 24, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pull ready Ready for copybara import and testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants