API v2

[karolz-ms]

Rationale

After shipping several Aspire releases we have learned enough, and accumulated enough outstanding, breaking-change work items, to develop a new version of DCP API.

Must-have work items and issues

• Create v2 version of all current DCP resources and make all object kinds namespaced. This will allow us to use DCP as a long-running application orchestrator, supporting multiple application runs simultaneously.
• Make sure that all object references are using NamespacedName, or--if the referenced object kind can vary (rare)--that they use NamespacedName and GroupVersionResource combination. We mostly conform to this rule already in API v1 but not always, e.g. ContainerExecSpec refers to its owning Container by name only.
• Make sure all current integration tests have their v2 equivalent and are passing
• Account for, and report, all relevant network addresses for Services, Executables, and Containers (eliminate single "effective address" fiction) Service status contains only one effective address #10
• Introduce a "physical" container resource/controller differentiated from the current "logical" container type Introduce a "physical" container resource/controller differentiated from the current "logical" container type #11 An immediate benefit of doing this is logic reuse between Container and ContainerNetworkTunnelProxy controllers.
• Introduce a "process" resource differentiated from the current, logical Executable to be able to model Executables that are suspended under debugger/restarted after code change (watch mode). Relevant Aspire issue: Add Suspended state for projects/Executables aspire#9321
• Support for multiple networks Support for multiple networks #18
• Resource change tracking Track Executable and Container state changes #34
• Add "container image" resource Introduce a dedicated resource for container image management (API v2) #8
• Allow more than one ExecutionType to be specified for an Executable (to solve [DCP] Executables that are not able to be run via IDE should fall back to process execution aspire#12522)

Should-consider work items and issues

These are issues that should be considered when designing details of v2 API and may benefit from being implemented on top of the new v2 API (not supported by v1 API), but are not blocking with regards to shipping v2 API

• Support for persistent executables #13
• Headless Service mode #30
• Sticky TCP sessions #31
• Automatic restarts for Containers and Executables #32
• Delay startup till dependencies ready #28
• [DCP] Executables that are not able to be run via IDE should fall back to process execution aspire#12522
• Persistent containers using container tunnel #77
• Ephemeral network resource #137
• Change EnvVar.Value definition to be a pointer, with the idea that if the pointer is nil, the presence of EnvVar record indicates the environment variable should be removed from the process/container environment if present in the ambient environment.

[karolz-ms]

David Negstad (@danegsta) and I had a long brainstorming session about supporting the new "dotnet watch" (a.k.a. Aspire watch) experience with DCP. Conclusions follow.

Goals

We assume that the main goals for the new experience are:

1. Be able to automatically restart, as necessary, both the Aspire application host, and individual services that comprise the application ("Aspire watch"). Restart will occur automatically in reaction to relevant source code change. Developer may want to confirm a restart.
2. Be able to see the dashboard (and preserve telemetry) across application restarts.
3. Avoid the need for "Aspire watch" to implement the IDE protocol. Individual services are started directly via "Aspire watch" CLI.
4. (eventually) Be able to debug an application running in watch mode. More specifically, we assume that Aspire watch will want to support the debug adapter protocol (as a "bridge") so that individual services are both debugable and can be hot-reloaded.

Design Issue 1: dashboard surviving app host restarts

David and I discussed the idea of making DCP lifetime tied to the Aspire CLI lifetime and solving the dashboard survival problem by having some DCP resources lifetime (dashboard in particular) be tied to the CLI lifetime. We do not think this is the best solution long-term. We eventually want DCP to be able to run in a truly independent fashion (a.k.a. "daemon mode") and making CLI own it does not help with that. It also adds complexity by forcing CLI to learn the DCP API without relieving the application host from the same.

Instead, we propose that we implement the "persistent Executable" feature #13 and run Aspire dashboard as persistent Executable. This will allow it to survive application host restarts and keep the rest of Aspire app model largely intact. And we can enhance the DCP auto-cleanup capability to ensure that the dashboard is stopped when the corresponding CLI instance is gone--more on that below.

Design Issue 2: supporting Aspire watch

The new Aspire watch implementation requires the watch server to be started before individual services (.NET projects) are run in watch mode. We think this requirement is best satisfied by introducing new AspireWatchServer and DotNetService types. Under the covers, these types would create appropriate lower-level Process types to model invocations of "aspire watch server" and "aspire watch ", respectively. The resulting type hierarchy (what Aspire app model would create) would look like this:

AspireWatchServer name: watchserver (represent demand for an Aspire watch server instance)
    Process (invocation of "aspire watch server")
DotNetService name: service1 (represents demand to run Service1 project; execution type = "Aspire watch", watch server="watchserver")
    Process (invocation of "aspire watch --server watchserver service1.csproj)
DotNetService name: service 2
    Process (invocation of "aspire watch --server watchserver service2.csproj)

etc.

With Debug Adapter Protocol support included, individual service object hierarchy may become as follows:

Project name:"service1 project" adapter: dotnet-vsdbg (represents a debuggable program with debug adapter configured for C#, is responsible for setting up DAP bridge to the actual adapter, in this case the aspire watch bridge to C# debug adaper)
    DotNetService name: service1 (represents demand to run Service1 project; execution type = "Aspire watch", watch server="watchserver")
        Proces (invocation of "aspire watch --server watchserver --dap service1.csproj)

The shape of these types is very different from what we currently have in DCP (API v1). We think the Aspire watch support is great reason to introduce the second major version of DCP API (a.k.a. API v2 #9).

API v2 MVP

We want to make rapid progress with API v2 and Aspire watch support, but there is a set of functionalities that must be delivered in a single chunk to make it possible to switch Aspire app model from API v1 to API v2. We propose the following as this minimal chunk (minimal v2 API):

1. Types

   • High-level workload types: DotNetService, AspireWatchServer, Executable (including persistent Executable), Container, ContainerVolume, ContainerExec, Project (for debuggable Executables)
   • Low-level types: Process, ContainerInstance
   • Networking types: Service (with full multi-address binding), Endpoint, ContainerNetworkTunnelProxy, ContainerNetwork, ContainerNetworkConnection

2. All types are namespaced (allowing multiple workloads to run simultaneously). We have a minimal Namespace controller that initiates resource cleanup for all objects inside a deleted namespace.

3. For persistent Executables, we need to enhance the DCP process monitor command to support re-attaching to I/O streams when a new DCP instance comes online and determines that associated process is still running and can be reused.

4. To facilitate terminating Aspire dashboard when "aspire run" terminates, we propose to enhance DCP process/resource harvesting capability. Currently DCP can harvest resources based on single (monitored) PID, and it is all-or-nothing proposition (all DCP resources are harvested). The proposed enhancement is to allow multiple PIDs to be monitored, and each would be associated with a different "resource harvesting group". For example, most DCP resources would be harvested when app host process exit, but the dashboard (persistent Executable) would be harvested when the "aspire run" CLI instance exits.

5. For persistent Executables, we need to upgrade the DCP data store. This store needs to keep data between DCP invocations, support concurrent access from multiple DCP instances, and auto-cleanup of stale data. Currently we use a set of JSON files, locked on demand with filesystem locks. This solution has been working remarkably error-free so far, but it is rapidly approaching its limits in terms of scenarios supported, ease of use, and maintainability. We propose replacing those with SQLite (fully embedded in DCP executable).

API v2 versioning

We envision that in the period of transition between API v1 and API v2 we will want to support both. DCP has support for K8s API versioning by virtue of being based on K8s API server libraries. This means we theoretically could continue to support API v1 as a façade built on top of API v1 (using API v2 as "storage API version") and dynamically translate to/from API v1. However, we don't think this is a practical solution for us. The proposed v2 API is significantly different from v1 API, so much that auto-translation will be increasingly difficult and error-prone while not buying us much, since we won't need to support API v1 and API v2 clients simultaneously.

Instead, we propose to make a "clean break" by freezing the current API v1 (and its controllers) and starting API v2 from scratch. They can be easily hosted side-by-side in the same DCP codebase or instance. This allows us to continue to support Aspire app hosts talking API v1 while allowing rapid development of API v2 and associated Aspire app host/CLI functionality.