Improve performance of EDTFField #14
Comments
jmurty
added a commit
that referenced
this issue
May 29, 2017
Improve performance of EDTFField when populating model instances from DB values by storing EDTF data in pickled format, not as string values that need to be re-parsed every time they are loaded. This implementation is naive and could be improved with a more sophisticated serialization approach for EDTF fields, though it works for now. This change also handles the issue in #15 where already- parsed EDTF field values are re-parsed when an instance is saved, unless there is a `natural_text_field` value present to override and reset the EDTF field value.
jmurty
added a commit
that referenced
this issue
Jun 5, 2017
Bump the character field size up to 2000 to avoid hitting character limits with this temporary fix, until we have a better serialization mechanism.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
EDTFFieldstores parsed EDTF values as text in the underlying database, and converts this stored text value back into a python object when a model instance's data is loaded from the DB.Unfortunately the conversion of text data into Python does a full pass through the EDTF grammar parsing process, which as noted in #11 is quite slow. This means that it is slow just to populate a model instance using an
EDTFFieldfrom the database, even if that field is not used.In anecdotal testing I have seen it take from 1/4 to 1/2 a second to load each model instance with a non-empty EDTF field from the database, which can be crippling when performing operations on many items such as listings.
To improve performance the
EDTFFieldshould probably be changed to stored pre-parsed values – as generated by thepre_savemethod – in the DB in a way that can be very quickly and directly converted back to Python by thefrom_db_valuemethod without needing to be parsed.The text was updated successfully, but these errors were encountered: