[serve] Remove dashboard's dependency on Serve

[shrekris-anyscale]

Why are these changes needed?

The Ray dashboard imports Serve even when it doesn't need to start the Serve controller. This change makes this a delayed import and removes the hard dependency.

It also adds a new job to the CI that uses Ray's default dependencies (i.e. the ones installed by "pip install ray[default]").

Related issue number

Closes #23309.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
    • New unit tests are added to test_dashboard.py.

[simon-mo]

ray dashboard doesn't work with minimal install. ray dashboard only works when ray[default] is installed

[shrekris-anyscale]

Got it, I added a script to perform the test with a default installation.

[simon-mo]

can you check regular dashboard process works (e.g. /api/snapshot) and serve doesn't?

[simon-mo]

can you drop all the

@pytest.mark.skipif(
    os.environ.get("RAY_DEFAULT") == "1",
    reason="This test may not work with default installation.",
)

ray dashboard (all the tests) should work with ray default install

[simon-mo]

can you check regular dashboard process works (e.g. /api/snapshot) and serve doesn't?