Improve prefetch by the Project model

[moshthepitt]

class PrefetchManager(models.Manager):
    def get_queryset(self):
        from onadata.apps.logger.models.xform import XForm
        from onadata.apps.api.models.team import Team
        return super(PrefetchManager, self).get_queryset().select_related(
            'created_by', 'organization'
        ).prefetch_related(
            Prefetch('xform_set',
                     queryset=XForm.objects.filter(deleted_at__isnull=True)
                     .select_related('user')
                     .prefetch_related('user')
                     .prefetch_related('dataview_set')
                     .prefetch_related('metadata_set')
                     .only('id', 'user', 'project', 'title', 'date_created',
                           'last_submission_time', 'num_of_submissions',
                           'downloadable', 'id_string', 'is_merged_dataset'),
                     to_attr='xforms_prefetch')

The above prefetch code will cause serious memory issues when dealing with a project that has a lot of dataviews.

In the case of one user who had over 400,000 dataviews in one project, she was never able to load the projects endpoint for that specific project as the onadata application would always get OOM killed.

[ukanga]

Begs the question, should we return filtered dataviews at the project end point?
Should we also return forms at this end point?

Cc @ivermac

[ukanga]

We do get the same issue even when the prefetch is not being used, right? Was the issue that we request xforns for each filtered dataview - XML + JSON?

[moshthepitt]

@ukanga @lincmba @ivermac @JohnMwashuma @jasonrogena

I think we have one major question before us:

Do we need to return dataviews from the projects endpoint?

Regardless of whether we do or not, I believe that we need to modify the prefetch block of dataview i.e. replace .prefetch_related('dataview_set') with something custom that only gets the XForm fields that we need. Similar to how xform_set is prefetched in the code above.

To illustrate, if you run:

print(Project.prefetched.get(id=1337).xform_set.all().query)

You get:

SELECT "logger_xform"."id", "logger_xform"."xls", "logger_xform"."json", "logger_xform"."description", "logger_xform"."xml", "logger_xform"."user_id", "logger_xform"."require_auth", "logger_xform"."shared", "logger_xform"."shared_data", "logger_xform"."downloadable", "logger_xform"."allows_sms", "logger_xform"."encrypted", "logger_xform"."deleted_by_id", "logger_xform"."sms_id_string", "logger_xform"."id_string", "logger_xform"."title", "logger_xform"."date_created", "logger_xform"."date_modified", "logger_xform"."deleted_at", "logger_xform"."last_submission_time", "logger_xform"."has_start_time", "logger_xform"."uuid", "logger_xform"."bamboo_dataset", "logger_xform"."instances_with_geopoints", "logger_xform"."instances_with_osm", "logger_xform"."num_of_submissions", "logger_xform"."version", "logger_xform"."project_id", "logger_xform"."created_by_id", "logger_xform"."has_hxl_support", "logger_xform"."last_updated_at", "logger_xform"."hash", "logger_xform"."is_merged_dataset" FROM "logger_xform" WHERE "logger_xform"."project_id" = 1337 ORDER BY "logger_xform"."id" ASC

Which is not desirable because we pull all of the XForm fields.

[ivermac]

Begs the question, should we return filtered dataviews at the project end point?
Should we also return forms at this end point?

The data endpoint uses the stream http response if it has been configured. We could change all our endpoints to have this option as well. With this, and I stand to be corrected, it won't be that much of a concern to have dataviews and forms in the projects endpoint.

[ukanga]

In terms of how the data has been structured that aspect of streaming might not be possible on the projects endpoints, it will, however, work well on dataviews and forms endpoints.