[Improvement-17755][Dao] Optimize pagination query performance of projects/workflow-instances/task-instances

[fanzhenyu95]

Purpose of the pull request

close #17755

Brief change log

Add the following indexes to improve the pagination query performance:

For t_ds_workflow_instance table:

• idx_project_code_start_time (project_code, start_time DESC, id DESC)
• idx_workflow_definition_code_start_time (workflow_definition_code, start_time DESC)

For t_ds_task_instance table:

• idx_project_code_submit_time (project_code, submit_time DESC)
• idx_project_code_start_time (project_code, start_time DESC)

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

[SbloodyS]

Please describe what issues you have encountered so that everyone can understand clearly what problems this PR has solved. @fanzhenyu95

[ruanwenjun]

Adding indexes alone won't solve all problems; we still need to optimize some query methods to achieve better results. Please first describe which interfaces you are currently experiencing slow query issues with.

[fanzhenyu95]

@SbloodyS @ruanwenjun Thank you for your feedback! The pagination queries for project list, workflow instance list, and task instance list are significantly slow in production environments with large datasets, leading to long page loading times and poor user experience.

Frequently used interfaces with slow query performance include:

1. Project list API: GET /projects
2. Workflow instance list API: GET /projects/{projectCode}/workflow-instances
3. Task instance list API: GET /projects/{projectCode}/task-instances

While adding indexes alone may not solve all performance issues, in our specific scenario with tables containing millions of records, list queries were taking over 10 seconds to execute and sometimes even failed to return results. After implementing the proposed indexes, all list queries now perform normally with acceptable response times.

[ruanwenjun]

@fanzhenyu95 Thanks for the clarification. I've updated the issue and PR titles.

[SbloodyS]

Schema Check failed. Please fix it. @fanzhenyu95

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[SbloodyS]

Schema Test failed. Please fix it. @fanzhenyu95