skip_missing is gone, how to skip None elements ?

[touilleMan]

I saw that v2.0.0b removed the skip_missing option, claiming it skips the missing entries by default.
However in my case (using mongoengine's Document) a field is never missing: it returns None which is, strictly speaking, something...

I can see from the commit (a3908d3) that previous behavior of skipping empty values (like None or empty list/tuple) has been obliterated. Is there now a new way to do this ?

I'm thinking for now to overload the _serialize method of each field to check for the empty value and return the missing singleton in such a case... but this sound really cumbersome to me, there must be a better way !

Same problem with nested Schema representing lazy loaded SqlAlchemy entities during serialization.

[mcquay239]

I beleive this is a relevant test:

assert not Marshaller()({"v": None}, {"v": fields.Integer()})

[VladimirPal]

@sloria Tell us please how can we skip None elements?
I've created this PR #254 and it works for me, it will skip None elements, but it breaks a lot of tests, looks like i don't understand something

[VladimirPal]

I think we need to have skip_if argument:

    name = fields.Str(skip_if=None)

@sloria If it's ok i can create PR

[sloria]

I'm not sure that a user would want to configure skippable values at the individual field level; it might make more sense to configure at the field class level.

Something like:

class SkippableString(fields.Str):
    skip_values = [None, '']

# Or, if you prefer
fields.Str.skip_values = [None, '']

Thoughts?

[VladimirPal]

What if i need all my fields with None value have been ignored in schema? I need to create some Meta class?
Here is my case:
In frontend i have a backbone model and it fetch data from backend, backend returns this None values and when i save model, i get errors - "Field may not be null", because i sent the data that i got from backend, i think it's a weird behaviour

[touilleMan]

I solve this trouble in my library (marshmallow intregration with mongoengine) by creating a mixin class which overload the _serialize method

class SkipEmptyClass(object):
    def __init__(self, *args, **kwargs):
        skip_empty = kwargs.pop('skip_empty', True)
        super(SkipEmptyClass, self).__init__(*args, **kwargs)
        self.skip_empty = skip_empty

    def _serialize(self, value, attr, obj):
        value = super(SkipEmptyClass, self)._serialize(value, attr, obj)
        if (self.skip_empty and
                (value is None or isinstance(value, (list, tuple)) and not value)):
            return missing
        return value

# Overload your marshmallow field class here
class String(SkipEmptyClass, fields.String):  # watch out inheritance order with MRO
    pass

see: https://github.com/touilleMan/marshmallow-mongoengine/blob/master/marshmallow_mongoengine/fields.py#L7

[VladimirPal]

@touilleMan Thanks, it will help me.
But i think it's really ultimate issue and we need to have more easy way.

[sloria]

Sorry for the delayed response on this.

How about using a post-dump method to remove any null values? Something like:

from marshmallow import Schema, fields, post_dump

class BaseSchema(Schema):
    SKIP_VALUES = set([None])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }


class MySchema(BaseSchema):
    foo = fields.Field()
    bar = fields.Field()


sch = MySchema()
sch.dump({'foo': 42, 'bar': None}).data  # {'foo': 42}

[Bachmann1234]

Howdy! Thanks @sloria for this. Though you will run into problems if you have dicts or anything not hashable. Luckily I just needed to filter Nones out so I could change it to

if value is not None

[makmanalp]

A related but tangential trouble is the default behavior for missing inputs - I'm not sure that missing fields do get skipped by default, specifically during dumps() I get AttributeError: "foo" is not a valid field for <obj> and such if a field is missing.

I think before this hits 2.0 final, it might be a decent idea to enumerate all the possibilities - serialization, deserialization, missing field, null field, field with type specified, field without type specified (in fields Meta). I volunteer myself for the task unless anyone objects.

Also, hi @Bachmann1234! :)

[Bachmann1234]

howdy!

[sloria]

@Bachmann1234

Though you will run into problems if you have dicts or anything not hashable.

True; in these cases, you can just make SKIP_VALUES a list.

@makmanalp

I get AttributeError: "foo" is not a valid field for and such if a field is missing.

Would you mind posting code that produces this error?

I think before this hits 2.0 final, it might be a decent idea to enumerate all the possibilities

Are you planning on updating the docs? Making code changes?

I volunteer myself for the task unless anyone objects.

Thank you!