Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Investigate if we can improve Pydantic's error messages for user consumption #13337

Open
DMRobertson opened this issue Jul 20, 2022 · 2 comments
Open

Investigate if we can improve Pydantic's error messages for user consumption #13337

DMRobertson opened this issue Jul 20, 2022 · 2 comments
Labels
A-Validation 500 (mostly) errors due to lack of event/parameter validation T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact
Milestone
Validate data pas...

Comments

@DMRobertson
Copy link
Contributor

DMRobertson commented Jul 20, 2022 •
edited
Loading

If anyone wants to see the kind of error messages Pydantic produces, try running this test case on my other pydantic experiment branch.

I think the name of the model isn't user-relevant. The "error context" (stuff in brackets) also loks like noise to an end-user. But we should be able to inspect the ValidationError and format it as we please.

/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-7yaa6cKe-py3.10/bin/python /home/dmr/.local/share/JetBrains/Toolbox/apps/PyCharm-C/ch-0/221.5787.24/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py --path /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py
Testing started at 19:38 ...
Launching unittests with arguments python -m unittest /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py in /home/dmr/workspace/synapse-2/tests/config



Ran 13 tests in 0.033s

OK

Process finished with exit code 0

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements -> 0
  value is not a valid dict (type=type_error.dict)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)

2 validation errors for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> answer
  extra fields not permitted (type=value_error.extra)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  expected string or bytes-like object (type=type_error)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI '' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'notaurl' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'https://example.com' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'mxc://mxc://mxc://' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_id
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at least 1 characters (type=value_error.any_str.min_length; limit_value=1)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
idp_id
  string does not match regex "^[A-Za-z0-9._~-]+$" (type=value_error.str.regex; pattern=^[A-Za-z0-9._~-]+$)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  field required (type=value_error.missing)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)
idp_name
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

Originally posted by @DMRobertson in #13188 (comment)
@DMRobertson DMRobertson added A-Validation 500 (mostly) errors due to lack of event/parameter validation T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact labels Jul 20, 2022
@DMRobertson
Copy link
Contributor Author

DMRobertson commented Jul 20, 2022 •
edited
Loading

Ahh, there is a model config option error_msg_templates which might be useful here. And we could set this globally by defining our own BaseModel, see https://pydantic-docs.helpmanual.io/usage/model_config/#change-behaviour-globally

Edit: I ended up creating RequestBodyModel

@DMRobertson DMRobertson mentioned this issue Aug 10, 2022
Merged
@DMRobertson
Copy link
Contributor Author

Rich notes:

I don't love the multiline error messages, particularly in the logs (where it is useful to be able to grep ERROR and hope to see most of the useful text). It might be better just to complain about the first error rather than trying to list all of them (which is what the jsonschema-based validation for config settings does).

We can iterate on this in future though.

@DMRobertson DMRobertson mentioned this issue Sep 8, 2022
Merged
@matrixbot matrixbot mentioned this issue Dec 21, 2023
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Validation 500 (mostly) errors due to lack of event/parameter validation T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact
Projects
None yet
Milestone
Validate data passed to REST endpoints
Development

No branches or pull requests
1 participant
@DMRobertson