0.20.28

Runs

dstack inside dstack

Sometimes, runs may need to call dstack themselves, for example, to inspect runs, submit another run, or attach to one.

This update makes this easy:

type: task
image: dstackai/dstack
dstack: true
env:
  - DSTACK_TOKEN
commands:
  - dstack ps

In this case, dstack exposes mounts /run/dstack/server.sock inside the container and sets DSTACK_SERVER_URL and DSTACK_PROJECT accordingly.

This gives the run access to the dstack server, allowing it to manage existing runs, submit new ones with dstack apply, and attach to them with dstack attach.

Authentication remains explicit, so DSTACK_TOKEN must be passed through env or secrets.

See documentation for more details.

Server

OpenTelemetry

The dstack server now integrates with OpenTelemetry SDK for exporting server traces, logs, and metrics. This can be used for monitoring the server via a LGTM (Grafana) observability stack or any other solution supporting OLTP. See documentation for more details.

Backends

Vast.ai spot instances

The vastai backend now supports interruptible spot instances, offering more competitive prices.

$ dstack offer -b vastai --gpu B200,B300 --spot-auto
 #   BACKEND               RESOURCES                                              INSTANCE TYPE  PRICE
 1   vastai (us-virginia)  cpu=96 mem=510GB disk=100GB gpu=B200:180GB:4 (spot)    45069637       $25.1215
 2   vastai (us-virginia)  cpu=96 mem=510GB disk=100GB gpu=B200:180GB:4           45069637       $28.6215
 3   vastai (ca-)          cpu=192 mem=2059GB disk=100GB gpu=B300:270GB:8 (spot)  43033760       $40.0694
 4   vastai (ca-)          cpu=192 mem=2059GB disk=100GB gpu=B300:270GB:8         43033760       $62.5694
 5   vastai (us-virginia)  cpu=192 mem=2043GB disk=100GB gpu=B200:180GB:8 (spot)  45069635       $50.1215
 6   vastai (us-virginia)  cpu=192 mem=2043GB disk=100GB gpu=B200:180GB:8         45069635       $57.1215
 7   vastai (us-oregon)    cpu=24 mem=31GB disk=100GB gpu=B200:180GB:1 (spot)     33945613       $5.651
 8   vastai (us-oregon)    cpu=24 mem=31GB disk=100GB gpu=B200:180GB:1            33945613       $6.901
 9   vastai (us-virginia)  cpu=48 mem=127GB disk=100GB gpu=B200:180GB:2 (spot)    45081639       $15.1215
 10  vastai (us-virginia)  cpu=48 mem=127GB disk=100GB gpu=B200:180GB:2           45081639       $16.3715

What's changed

• Ignore cachetools-related typing error in AWS backend by @un-def in #4022
• Fix Git.ls_remote() return type error by @un-def in #4024
• Pass effective Requirements to Compute's run_job/run_jobs by @un-def in #4020
• Kubernetes: Rework offers and resource handling by @un-def in #4021
• Do not ignore Vast.ai instance termination errors by @jvstme in #4030
• Migrate off deprecated Vast.ai /api/v0/instances by @jvstme in #4031
• [Feature] Supporting dstack-inside-dstack for runs by @peterschmidt85 in #4023
• Vast.ai: Support spot instances by @trashhalo in #4025
• Update Vast.ai spot instances support by @jvstme in #4036
• Update gateway connection error log level by @jvstme in #4035
• Detect Vast.ai interruptions during provisioning by @jvstme in #4037
• Support server monitoring with OpenTelemetry by @r4victor in #4034
• Update VastAIAPIClient by @jvstme in #4038
• [Feature] Endpoint presets by @peterschmidt85 in #4028

New contributors

• @trashhalo made their first contribution in #4025

Full changelog: 0.20.27...0.20.28