Argument validation decorator #185

justcallmegreg · 2016-07-10T21:27:36Z

Introducing decorator that is able to validate input arguments and return value
of the decorated function.

Before calling the wrapped function, the validators specified in the
decorator's argument list will be tested against the values passed at
the function call.

@validate_schema(arg1=int, arg2=int)
def foo(arg1, arg2):
  return arg1 * arg2

After calling the function with the validated arguments, schema
specified in RETURNS_KEY (currently __returns__) will be applied
against the output.

@validate_schema(arg1=int, arg2=int, __returns__=int)
def foo(arg1, arg2):
  return arg1 * arg2

See more in the related test cases.

jstewmon · 2016-07-11T16:15:24Z

FWIW, I implemented something similar for one of my projects, which I'd like to share here to add some color to how such a decorator should work if it's added to voluptuous.

My implementation works with named args, varargs and keyword args. Although I'm new to voluptuous, I think this is the right feature set for a decorator that is going to ship with the library.

For my solution, I used inspect, which I realize may not be the right solution for python 3. I don't think my current solution will work with keyword-only args in python 3.5. I think that to properly support python 3.5, I'd need to have two discrete decorator implementations, so that the one for python 3.5 uses inspect.signature.

Here's my implementation and tests:

import functools
import inspect
import itertools
from collections import OrderedDict

from future.utils import iteritems
from future.utils import viewkeys

def apply_schema(**schema_map):

    def decorator(func):
        arg_spec = inspect.getargspec(func)
        varargs = arg_spec.varargs
        keywords = arg_spec.keywords
        arg_count = len(arg_spec.args)
        arg_positions = OrderedDict(
            (arg, pos)
            for pos, arg in enumerate(arg_spec.args)
        )

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            converted = {}
            for arg_name, pos in iteritems(arg_positions):
                if pos >= len(args):
                    break
                if arg_name in schema_map:
                    converted[pos] = schema_map[arg_name](args[pos])
            if varargs in schema_map:
                for pos, arg in enumerate(itertools.islice(args, arg_count, None), arg_count):
                    converted[pos] = schema_map[varargs](arg)
            if keywords in schema_map:
                for kw in viewkeys(kwargs):
                    if kw in arg_positions:
                        if kw in schema_map:
                            kwargs[kw] = schema_map[kw](kwargs[kw])
                    else:
                        kwargs[kw] = schema_map[keywords](kwargs[kw])
            else:
                for kw in viewkeys(kwargs) & viewkeys(schema_map):
                    kwargs[kw] = schema_map[kw](kwargs[kw])
            if converted:
                validated_args = (converted.get(i, arg) for i, arg in enumerate(args))
            else:
                validated_args = args

            return func(*validated_args, **kwargs)
        return wrapper
    return decorator


def test_apply_schema():
    schema = voluptuous.Schema({'foo': voluptuous.Coerce(int)})

    input_arg = {'foo': '1'}
    expect = {'foo': 1}

    @apply_schema(arg=schema)
    def func_named(arg=None):
        assert arg == expect

    @apply_schema(arg=schema)
    def func_keywords(**kwargs):
        assert kwargs['arg'] == expect

    @apply_schema(arg=schema)
    def func_positional(arg):
        assert arg == expect

    @apply_schema(args=schema)
    def func_varargs(*args):
        for arg in args:
            assert arg == expect

    @apply_schema(named=voluptuous.Schema(int), kwargs=schema)
    def func_all_keywords(ignored=None, named=None, **kwargs):
        assert type(named) is int
        for arg in kwargs.values():
            assert arg == expect

    func_positional(input_arg)
    func_positional(arg=input_arg)

    func_named(input_arg)
    func_named(arg=input_arg)

    func_keywords(arg=input_arg)
    func_all_keywords(named=1, ignored=None, a=input_arg, b=input_arg)

    func_varargs(input_arg, input_arg)

Edit: noticed that I was missing coverage on the case where a named argument should not have a schema applied, so I fixed the test case for func_all_keywords and fixed an issue in the implementation around this case.

justcallmegreg · 2016-07-13T14:34:18Z

👍 Thanks for the ideas!

In my opinion using func_code over inspect should be preferred for the func_code can be ported to python 3 with a little effort. (func_xxx has been renamed to xxx)

Although I like the idea of calling the decorator apply_schema with the named, kwargs parameters. I'd add some more optional parameters like ret for return value.

Sum up:

@apply_schema(
    args=schema1,
    kwargs=schema2,
    ret=schema3
)

jstewmon · 2016-07-13T16:15:48Z

Hmm... my decorator expects the kwargs with which it's invokes to match the names of the args used in the function signature, so that it can apply the schema strictly to the args passed at the call site. This precludes any reserved kwargs, such as ret for a return value. I think it's reasonable to have a separate decorator that only applies to the return value.

I don't see an obvious way to use __code__ to properly map the per-arg schema passed to the decorator to the args passed at the call site since the name used for varargs and kwargs is discretionary.

I think that succinct compatibility is possible with my solution by changing the decorator setup to be like so:

    def decorator(func):
        if hasattr(inspect, 'getfullargspec'):
            arg_spec = inspect.getfullargspec(func)
            keywords = arg_spec.varkw
        else:
            arg_spec = inspect.getargspec(func)
            keywords = arg_spec.keywords
        varargs = arg_spec.varargs
        arg_count = len(arg_spec.args)
        arg_positions = OrderedDict(
            (arg, pos)
            for pos, arg in enumerate(arg_spec.args)
        )

Edit: I confirmed that this compatibility update works with my existing tests, plus this new case:

    if PY3:
        @apply_schema(args=schema, arg=schema)
        def func_keywords_only(ignored, *args, arg=None, **kwargs):
            assert arg == expect

        func_keywords_only(None, input_arg, input_arg, extra=None, arg=input_arg)

justcallmegreg · 2016-08-18T12:31:02Z

Hey! Here is one of my projects called ratatoskr. It has a decorator called protectron that realizes this functionality. It can also be used with voluptuous.

tusharmakkar08 · 2016-09-15T14:09:31Z

Hey @ngergo

I personally like the idea of decorator apply_schema. Please resolve the conflicts and get this merged to master.

Thanks

justcallmegreg · 2016-09-17T21:38:06Z

Hey @tusharmakkar08 ,

I'll refresh the PR and rebase to the current master.

alecthomas · 2016-09-18T10:04:29Z

I like the idea of a function validator, but I don't really like the usage of this implementation because it's more verbose than it needs to be IMO. I'd prefer something like this:

@validate(arg0=int, arg1=int, __returns__=string)
def func(arg0, arg1=1):
  return "hello"

justcallmegreg · 2016-09-18T14:15:47Z

Hey @alecthomas , @tusharmakkar08

I've refactored my patch according to your proposal. I like this way of specifying conditions to inputs and outputs... Honestly, I'd prefer passing parameters as a dict for that's much easier to review and can be generalised easier if a validator needs to be used more than one time.

There is only one thing I'd change here... We should use __return__ instead of __returns__. Specifying the wrong parameter could lead to a long debugging session... and I think it's not just me who will forget the s from the end of the word.

alecthomas · 2016-09-18T14:22:53Z

I'm happy with __return__.

justcallmegreg · 2016-09-18T14:24:57Z

Done then.

alecthomas

This is a really great change, thanks for putting in the effort. Just a couple of minor changes.

alecthomas · 2016-09-18T23:53:41Z

voluptuous/schema_builder.py

@@ -1017,16 +1017,75 @@ def wrapper(*args, **kwargs):

    return decorator

+def _args_to_dict(func, args):
+    """


Can you make this docstring style (here and elsewhere) consistent with the rest of Voluptuous please?

alecthomas · 2016-09-18T23:55:26Z

voluptuous/schema_builder.py

+    ret = args_dict.copy()
+    ret.update(kwargs_dict)
+    return ret
+

 def validate_schema(*a, **kw):


I realise this is not part of your change, but would you remind renaming this to just "validate"? It is currently undocumented, so I'm not concerned with breaking backwards compatibility.

alecthomas · 2016-09-18T23:59:10Z

voluptuous/schema_builder.py

+            if len(schema_arguments) == 0:
+                output = func(**arguments)
+            else:
+                input_schema = Schema(schema_arguments)


Pull this out of the inner function, otherwise it will be rebuilding the schema on every call. To handle the main if case (len(schema_arguments) == 0), make it a no-op lambda returning its arg in that situation.

alecthomas · 2016-09-18T23:59:24Z

voluptuous/schema_builder.py

+                output = func(**validated_arguments)
+
+            if returns_defined:
+                output_schema = Schema(returns)


Same as above; pull this out of the inner function.

alecthomas · 2016-09-19T00:40:04Z

voluptuous/schema_builder.py

+    """
+        Returns argument names as values as key-value pairs.
+
+        :input `func` that is a function object


If you want to document the parameters it should be in correct RST format.

alecthomas · 2016-09-19T00:40:29Z

voluptuous/schema_builder.py


 def validate_schema(*a, **kw):
-    schema = Schema(*a, **kw)
+    """


This docstring should be a bit more descriptive I think.

justcallmegreg · 2016-09-19T11:49:20Z

Hey @alecthomas,

Finished the changes u'd requested. I've put them into separated patches to make it easier for u to review. If ur ok with it I'll merge them into one commit.

alecthomas · 2016-09-19T13:14:29Z

Looks good! Once you've squashed the commits I'll merge it in. Thanks for doing all that.

Introducing decorator that is able to validate input arguments and return value of the decorated function. Before calling the wrapped function, the validators specified in the decorator's argument list will be tested against the values passed at the function call. ``` @validate_schema(arg1=int, arg2=int) def foo(arg1, arg2): return arg1 * arg2 ``` After calling the function with the validated arguments, schema specified in `RETURNS_KEY` (currently `__return__`) will be applied against the output. ``` @validate_schema(arg1=int, arg2=int, __returns__=int) def foo(arg1, arg2): return arg1 * arg2 ``` See more in the related test cases. Signed-off-by: Gergő Nagy <grigori.grant@gmail.com>

justcallmegreg · 2016-09-19T19:03:02Z

Done. :)

alecthomas requested changes Sep 19, 2016

View reviewed changes

alecthomas reviewed Sep 19, 2016

View reviewed changes

alecthomas approved these changes Sep 19, 2016

View reviewed changes

alecthomas merged commit 6802ce0 into alecthomas:master Sep 19, 2016

tusharmakkar08 mentioned this pull request Sep 27, 2016

Decorating functions #216

Closed

pyup-bot mentioned this pull request Feb 15, 2018

Update voluptuous to 0.11.1 sohonetlabs/generate-ipdatabase#27

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Argument validation decorator #185

Argument validation decorator #185

justcallmegreg commented Jul 10, 2016 •

edited

Loading

jstewmon commented Jul 11, 2016 •

edited

Loading

justcallmegreg commented Jul 13, 2016

jstewmon commented Jul 13, 2016 •

edited

Loading

justcallmegreg commented Aug 18, 2016 •

edited

Loading

tusharmakkar08 commented Sep 15, 2016 •

edited

Loading

justcallmegreg commented Sep 17, 2016 •

edited

Loading

alecthomas commented Sep 18, 2016

justcallmegreg commented Sep 18, 2016

alecthomas commented Sep 18, 2016

justcallmegreg commented Sep 18, 2016

alecthomas left a comment

alecthomas Sep 18, 2016 •

edited

Loading

alecthomas Sep 18, 2016 •

edited

Loading

alecthomas Sep 18, 2016 •

edited

Loading

alecthomas Sep 18, 2016

alecthomas Sep 19, 2016

alecthomas Sep 19, 2016

justcallmegreg commented Sep 19, 2016

alecthomas commented Sep 19, 2016

justcallmegreg commented Sep 19, 2016

Argument validation decorator #185

Argument validation decorator #185

Conversation

justcallmegreg commented Jul 10, 2016 • edited Loading

jstewmon commented Jul 11, 2016 • edited Loading

justcallmegreg commented Jul 13, 2016

jstewmon commented Jul 13, 2016 • edited Loading

justcallmegreg commented Aug 18, 2016 • edited Loading

tusharmakkar08 commented Sep 15, 2016 • edited Loading

justcallmegreg commented Sep 17, 2016 • edited Loading

alecthomas commented Sep 18, 2016

justcallmegreg commented Sep 18, 2016

alecthomas commented Sep 18, 2016

justcallmegreg commented Sep 18, 2016

alecthomas left a comment

Choose a reason for hiding this comment

alecthomas Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

alecthomas Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

alecthomas Sep 18, 2016 • edited Loading

Choose a reason for hiding this comment

alecthomas Sep 18, 2016

Choose a reason for hiding this comment

alecthomas Sep 19, 2016

Choose a reason for hiding this comment

alecthomas Sep 19, 2016

Choose a reason for hiding this comment

justcallmegreg commented Sep 19, 2016

alecthomas commented Sep 19, 2016

justcallmegreg commented Sep 19, 2016

justcallmegreg commented Jul 10, 2016 •

edited

Loading

jstewmon commented Jul 11, 2016 •

edited

Loading

jstewmon commented Jul 13, 2016 •

edited

Loading

justcallmegreg commented Aug 18, 2016 •

edited

Loading

tusharmakkar08 commented Sep 15, 2016 •

edited

Loading

justcallmegreg commented Sep 17, 2016 •

edited

Loading

alecthomas Sep 18, 2016 •

edited

Loading

alecthomas Sep 18, 2016 •

edited

Loading

alecthomas Sep 18, 2016 •

edited

Loading