Unable to use custom domain for OAuth2 (JWT Profile) token endpoint

[sunnyyip]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

https://naughty-tesla-4oqisau3a4.projects.oryapis.com

Describe the bug

I have a custom domain set up and was trying to use it for my JWT Profile OAuth2 flow (Using JWT as Authorization Grants - https://www.ory.sh/docs/hydra/guides/jwt#using-jwts-as-authorization-grants).
i.e. make an access token request to https://[custom_domain]/oauth2/token with grant_type and assertion parameters.

This worked with the Ory project oauth2 token endpoint https://[project_slug].projects.oryapis.com/oauth2/token but not with the cname/custom domain. This is the error that I got with HTTP 400 status code:
{ "error": "invalid_grant", "error_description": "The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. The JWT in 'assertion' request parameter MUST contain an 'aud' (audience) claim containing a value 'https://[custom_domain]/oauth2/token' that identifies the authorization server as an intended audience." }

Whatever custom domain I set to and sent an access token request to, it gets echo’d back in the error message stating that the 'aud' must match the custom domain token endpoint. I have confirmed that the JWT sent in assertion has the audience (aud) set to the token endpoint.

I followed the troubleshooting page at https://www.ory.sh/docs/troubleshooting/oauth2-trust-audience and https://www.ory.sh/docs/oauth2-oidc/issuer-url and set the issuer url matching my custom domain but the error persists.
I confirmed via the Ory CLI and also the .well-known/openid-configuration endpoint that the issuer url got updated.

Also, it works - I got the access token in response - when a JWT with “aud” set to the project-slug token endpoint was sent to the cname’d endpoint. It looks like the token endpoint doesn’t recognize the cname (as the “aud”) and refused to process the request.

Reproducing the bug

1. Set up a custom domain for the project
2. CNAME the custom domain to the project slug endpoint
3. Update the project's issuer url to the custom domain per https://www.ory.sh/docs/oauth2-oidc/issuer-url.
4. Send an access token request to the CNAME'd token endpoint https://[custom_domain]/oauth2/token with grant_type and assertion parameters per https://www.ory.sh/docs/hydra/guides/jwt#using-jwts-as-authorization-grants. The assertion parameter should be a JWT ID token with "aud" set to the CNAME'd token endpoint.
5. Observe the error message being returned

Relevant log output

HTTP 400

{
	"error": "invalid_grant",
	"error_description": "The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. The JWT in 'assertion' request parameter MUST contain an 'aud' (audience) claim containing a value 'https://[custom_domain]/oauth2/token' that identifies the authorization server as an intended audience."
}

Relevant configuration

No response

Version

Ory Network

On which operating system are you observing this issue?

Ory Network

In which environment are you deploying?

Ory Network

Additional Context

No response

[aeneasr]

Thank you for the context! It appears that setting the issuer URL does not change the token URL, which seems to be causing this issue and can be considered a bug!

[aeneasr]

https://naughty-tesla-4oqisau3a4.projects.oryapis.com/.well-known/openid-configuration shows that issuer is set correctly, but token_endpoint is set to the public URL.

This comes from here

https://github.com/ory/hydra/blob/4b8c9715684c29e89d06fbaa8178d8f57a9ca979/oauth2/handler.go#L484C53-L484C53

which in turn comes from here

https://github.com/ory/hydra/blob/4b8c9715684c29e89d06fbaa8178d8f57a9ca979/driver/config/provider.go#L433

which uses the public URL, and not the issuer URL. I think this is a bug, but changing this default value could have some problematic effects. If we change it, it should be marked as a breaking change, and we should check in Ory Network how many customers would be affected by this change.

[sunnyyip]

Thanks for looking into this, @aeneasr! Any idea on the potential timeframe to fix this? I ask because we're looking to roll out to production with the custom domain endpoint in a few weeks. Thanks.

[aeneasr]

It's triaged internally and someone will pick it up soon.

[aeneasr]

Fixed now and will be released in the coming days to production.

[sunnyyip]

@aeneasr may I ask if the fix has been deployed to Ory Network? I'm still getting the same error.
If it hasn't, may I ask for the ETA? Thanks.

[aeneasr]

If a custom domain is set up, then the URLs will use the custom domain instead of the oryapis one. You can see this here:

https://ory.aeneas.io/.well-known/openid-configuration

I checked for your project and it appears that no custom domain is configured for this project?

https://naughty-tesla-4oqisau3a4.projects.oryapis.com/.well-known/openid-configuration

[sunnyyip]

I forgot to reset the issuer url to use the custom domain. But it still doesn't help now that I did.

The token_endpoint is still using the project slug url even when the issuer changed -
https://naughty-tesla-4oqisau3a4.projects.oryapis.com/.well-known/openid-configuration

[aeneasr]

Try exporting your config with ory get oauth2-config <project-id> --format yaml and check that all the URLs are pointing to the correct domain :)

[sunnyyip]

I did and only urls.self.issuer changed to my custom domain. I set it per https://www.ory.sh/docs/oauth2-oidc/issuer-url

What am I missing?

My custom domain config looks ok -
https://auth.kusari.cloud/.well-known/openid-configuration

[aeneasr]

Ok, this probably needs further investigation. I assume you are making your API calls against the custom domain for token exchange etc?

[sunnyyip]

Yes, just double checked that I was using the custom domain url. Thanks.

[sunnyyip]

@aeneasr did we get a chance to investigate on this?

[sunnyyip]

@aeneasr any update on this issue? Thanks.

[aeneasr]

Hello, sorry for being slow on this. I'm prioritizing this in our backlog!

[aeneasr]

Hello, this is now fixed on production!