optimize UX for loading pipeline pages

[IronPan]

speed up the query time for list runs

For example:
Before

SELECT subq.uuid, 
       "[" 
       || Group_concat(r.payload, ",") 
       || "]" AS refs 
FROM   (SELECT rd.*, 
               "[" 
               || Group_concat(m.payload, ",") 
               || "]" AS metrics 
        FROM   (SELECT * 
                FROM   run_details 
                WHERE  uuid IN (SELECT resourceuuid 
                                FROM   resource_references AS rf 
                                WHERE  ( rf.resourcetype = 'Run' 
                                         AND rf.referenceuuid = 'b7cc0970-e823-431a-a7d9-6b34407bbe17' 
                                         AND rf.referencetype = 'Experiment' ))) AS rd 
               LEFT JOIN run_metrics AS m 
                      ON rd.uuid = m.runuuid 
        GROUP  BY rd.uuid) AS subq 
       LEFT JOIN (SELECT * 
                  FROM   resource_references 
                  WHERE  resourcetype = 'Run') AS r 
              ON subq.uuid = r.resourceuuid 
GROUP  BY subq.uuid 
ORDER  BY createdatinsec ASC, 
          uuid ASC 
LIMIT  40;

After

SELECT subq.uuid, 
       "[" 
       || Group_concat(r.payload, ",") 
       || "]" AS refs 
FROM   (SELECT rd.*, 
               "[" 
               || Group_concat(m.payload, ",") 
               || "]" AS metrics 
        FROM   (SELECT * 
                FROM   run_details 
                WHERE  uuid IN (SELECT resourceuuid 
                                FROM   resource_references AS rf 
                                WHERE  ( rf.resourcetype = 'Run' 
                                         AND rf.referenceuuid = 'b7cc0970-e823-431a-a7d9-6b34407bbe17' 
                                         AND rf.referencetype = 'Experiment' )) ORDER  BY createdatinsec ASC, uuid ASC LIMIT  40) AS rd 
               LEFT JOIN run_metrics AS m 
                      ON rd.uuid = m.runuuid 
        GROUP  BY rd.uuid) AS subq 
       LEFT JOIN (SELECT * 
                  FROM   resource_references 
                  WHERE  resourcetype = 'Run') AS r 
              ON subq.uuid = r.resourceuuid 
GROUP  BY subq.uuid; 
ORDER  BY createdatinsec ASC, uuid ASC;

---

This change is 

[IronPan]

/assign @hongye-sun

[IronPan]

/hold
more optimization might be needed for further speedup

[rileyjbauer]

I think we should hold off on this for now, but you change it if you think it's alright. The Run (not detail) part of the API is also used for the experiment list page, and I'm not sure yet that we'll always prefer the full run detail. it seems like the core part of this PR is still valuable without this.

[IronPan]

reverted.

[IronPan]

/hold cancel

[hongye-sun]

/lgtm

[hongye-sun]

Why is this removed?

[hongye-sun]

Just wonder if it's still required to add sorting query if inner result has already been sorted.

[IronPan]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [IronPan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [IronPan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment