Enhancing Kedro-Viz Performance with Lazy Loading

[rashidakanchwala]

Description

The current loading strategy for Kedro-Viz can lead to significant delays. It requires all data, including session information like the catalog and session_store for experiment tracking, to be fully loaded before the visualization is rendered. This process can slow down the display of the Kedro-Viz flowchart, which only needs the quicker-to-load pipeline_data for initial rendering.

To improve the performance, we need to -

1. Refactor Kedro-Viz CLI - Refactor CLI with lazy subcommands and deferring imports #1920
2. Refactor Kedro-Viz FAST API app creation - Refactor Kedro-Viz FAST API app creation #2061
3. Asynchronously load pipeline data - I tested this while profiling work and created a test branch with some changes in server.py - https://github.com/kedro-org/kedro-viz/blob/test/large-pipeline-profiling/package/kedro_viz/server.py . I did not see a huge improvement in load time, but we can take this up as a minor improvement. I will create a ticket if needed. cc: @rashidakanchwala

Testing the performance improvements -

1. Test against a massive pipeline once created - Create massive pipeline to test with flowchart on Kedro-viz  #1064
2. Test against a complex pipeline once created - [Stress Testing] - Create example projects to assess Kedro performance for complex pipelines kedro#3866

Checklist

• Include labels so that we can categorise your feature request