Skip to content

Releases: dstackai/dstack

Release list

0.21.1

0.21.1 Latest
Latest

Choose a tag to compare

View all tags
@jvstme jvstme released this 14 Aug 16:57
0.21.1
7767e22
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Presets

Patching framework

Presets can now patch source code (e.g. serving frameworks, kernels, etc). Patches are stored with the preset as unified diffs and applied when the preset starts, so dstack preset apply reproduces the optimized environment.

type: service

files:
  - local_path: patches/rocm_aiter_mla.py.patch
    path: /patches/rocm_aiter_mla.py.patch

commands:
  - for p in /patches/*.patch; do patch -p1 -d / < "$p"; done
  - vllm serve ...

Previous sessions

Agents can now use results from previous preset sessions to continue optimization instead of starting from scratch. Pass one or more session IDs with --previous:

$ dstack preset create -f config.yml --previous e8b7e09c

The previous configuration property can also be used to provide multiple sessions.

baseline now defaults to true, ensuring every session has an anchor for comparison. The first trial uses the framework's recommended configuration, or reproduces the previous best when building on a previous session.

Benchmark datasets

Presets can benchmark against real datasets instead of synthetic prompts. Set dataset to a supported benchmark dataset or a Hugging Face dataset ID:

dataset: sharegpt

When a dataset is specified, it provides the requests, so input_tokens, output_tokens, and shared_prefix_tokens are no longer set explicitly. The preset records the measured means instead. Gated datasets can be accessed by providing HF_TOKEN in env.

dstack preset now displays the dataset in place of the synthetic request shape.

Failed preset creations are also now shown by dstack preset, making unsuccessful optimization attempts visible instead of silently disappearing.

Metrics

dstack metrics can now display all jobs and replicas of a run at once.

$ dstack metrics vllm-qwen

                         UTILIZATION                  MEMORY
 replica=0 job=0  cpu    ▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ 20%  ▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ 85GB/440GB
                  gpu=0  ▄▄▃▃▃▃▃▃▃▃▄▄▅▅▅▅▅▅▅▄▄▄▄ 51%  ▃▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ 60GB/80GB

 replica=1 job=0  cpu    ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 6%   ▁▁▁▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁ 73GB/440GB
                  gpu=0  ▂▂▁▁▁▁▁▁▁▁▂▂▂▂▃▃▃▃▃▂▂▁▁ 14%  ▃▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ 60GB/80GB

 replica=2 job=0  cpu    ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 1%   ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 61GB/440GB
                  gpu=0  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 2%   ▃▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ 60GB/80GB

                         6 Aug 17:11 ┄┄┄┄┄┄┄ now      6 Aug 17:11 ┄┄┄┄┄┄┄ now

Troubleshooting

Provisioning failures now provide more actionable status and error information when all provisioning attempts fail.

$ dstack apply -y

my-task provisioning completed (failed)
No capacity
Failed to provision in fleet 'cloud-fleet': tried 1 of 1 offers, all failed.
Errors: p5.48xlarge in aws/us-west-1: NoCapacityError.
See https://dstack.ai/docs/guides/troubleshooting/#provisioning-fails

Gateways

Scaling

Gateways can now be scaled out and scaled in by updating the replicas property in-place:

$ dstack apply -f my-gateway.dstack.yml
Found gateway my-gateway. Detected changes that can be updated in-place:
- replicas

Update the gateway? [y/n]:

When scaling in, dstack stops the oldest gateway replicas first.

Fault tolerance

An unhealthy gateway replica no longer prevents the service from provisioning, as long as the service can be registered on at least one healthy gateway replica. Similarly, an unhealthy gateway replica does not prevent service replica provisioning, as long as the service replica can be registered on another gateway replica.

Registration progress for each gateway replica can be tracked in events.

$ dstack event --within-run my-service | grep Service
[2026-08-14 17:51:10] [run my-service, gateway test-gateway] Service registered on gateway replica 0
[2026-08-14 17:51:10] [run my-service, gateway test-gateway] Service registered on gateway replica 1
[2026-08-14 17:52:03] [job my-service-0-1] Service replica ready to receive requests
[2026-08-14 17:52:03] [job my-service-0-0] Service replica ready to receive requests
[2026-08-14 17:52:23] [job my-service-0-1, gateway test-gateway] Service replica registered on gateway replica 1
[2026-08-14 17:52:23] [job my-service-0-0, gateway test-gateway] Service replica registered on gateway replica 1
[2026-08-14 17:52:23] [job my-service-0-1, gateway test-gateway] Service replica registered on gateway replica 0
[2026-08-14 17:52:23] [job my-service-0-0, gateway test-gateway] Service replica registered on gateway replica 0

During rolling deployments, however, the old service replica is not terminated until the new service replica has been registered on all gateway replicas.

Backends

AWS

AWS p5en.48xlarge instances are now supported, with EFA configured automatically for improved connectivity in clusters.

$ dstack offer -b aws --instance-type p5en.48xlarge --max-offers 5
 #   BACKEND           RESOURCES                                              INSTANCE TYPE  PRICE
 1   aws (us-east-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $26.5714
 2   aws (us-west-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $26.9369
 3   aws (us-east-1)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $27.1988
 4   aws (eu-north-1)  cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $39.8158
 5   aws (us-east-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8         p5en.48xlarge  $63.296

Runpod

dstack no longer provides Runpod spot offers, since Runpod announced that Spot and Interruptible Pods will no longer be available after Friday, September 11th.

Breaking changes

The deprecated dstack_server_request* Prometheus metrics are no longer available. Use the OpenTelemetry http_server_* metrics instead.

What's changed

Full Changelog: 0.21.0...0.21.1

Contributors

un-def, r4victor, and 2 other contributors
Assets 2
Loading

0.21.0

0.21.0

Choose a tag to compare

View all tags
@r4victor r4victor released this 06 Aug 13:31
0.21.0
7aaae67
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

dstack 0.21.0 is a major release that migrates the codebase to Pydantic v2, introduces new features, and removes a number of previously deprecated features. New 0.21.x CLIs do not work with older 0.20.x servers. New servers continue to support older CLIs, so ensure to update the server and the CLI at the same time or update the server first.

Pydantic v2

dstack has finally migrated from Pydantic v1 to Pydantic v2. Besides faster validation and serialization and faster CLI startup, the migration unblocks Python 3.14 support and integration with latest Python libraries.

If you use the Python API or have dstack plugins installed, ensure the code works with Pydantic v2 models before upgrading dstack.

Python 3.14

dstack now supports Python 3.14 both for installing dstack and inside runs:

type: dev-environment
ide: vscode
python: "3.14"

Gateways

Replicated gateways with HTTPS

Replicated gateways previously required an external load balancer for TLS termination. AWS gateways with an acm certificate can now have more than one replica with HTTPS handled by dstack:

type: gateway
name: example-gateway

backend: aws
region: eu-west-1

domain: example.com

certificate:
  type: acm
  arn: arn:aws:acm:eu-west-1:164099421079:certificate/3670388f-f43b-4872-aaf8-907b107a170d

replicas: 2

Load balancing across replicas is performed by a single ALB associated with the gateway, which becomes the gateway's hostname:

$ dstack gateway list
 NAME             BACKEND          HOSTNAME                                                  DOMAIN       DEFAULT  STATUS
 example-gateway                   dstack-qe1na76o-lb-187858581.eu-west-1.elb.amazonaws.com  example.com  ✓        running
    replica=0     aws (eu-west-1)  18.202.25.65                                                                    running
    replica=1     aws (eu-west-1)  3.255.100.238                                                                   running

Replicated gateways remain experimental. For other backends and certificate types, an external load balancer is still required for TLS termination.

CLI

Improved dstack metrics

dstack metrics now shows CPU, memory, and GPU utilization over the last hour of the job, allowing to track changes in metrics:

$ dstack metrics gentle-mayfly-1

        UTILIZATION                             MEMORY
 cpu    ▅▄▄▄▃▃▃▃▃▃▃▃▃▃▃▃▅▅▄▃▃▃▃▃▃▃▃ 41% of 128  ▃▃▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ 581GB/960GB

 gpu=0  ▁▂▃▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 89%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
 gpu=1  ▁▂▆▆▆▅▆▆▆▆▆▆▆▆▆▆▁▁▅▆▆▅▆▆▆▆▆ 84%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
 gpu=2  ▁▂▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆▆ 87%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
 gpu=3  ▂▃▆▅▅▅▅▅▅▆▅▅▆▆▆▆▁▅▅▅▅▅▆▅▅▅▅ 82%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB

        4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now             4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now

Like dstack logs, the command now shows a single job: --replica and --job select one, both defaulting to 0.

GPU driver

dstack fleet -v now shows a DRIVER column with the accelerator driver installed on the host (NVIDIA, AMD, or Tenstorrent), also available as Instance.gpu_driver in the API. The driver is detected by dstack-shim and refreshed on instance checks, so it is filled in for existing instances after an upgrade and updated after a driver upgrade on the host. Container-based backends report no driver for now.

Offers

dstack offer and dstack apply gain two new flags: --full-offers and --unallocated (and corresponding full_offers and unallocated_resources API parameters):

  • --full-offers requests not to adjust offers by requirements on backends that pack multiple jobs onto a node and return full node resources. Currently supported by Kubernetes and Slurm backends.
  • --unallocated subtracts already-allocated resources on backends that pack multiple jobs onto a node, so offers reflect only the available capacity. Currently supported by Kubernetes backend.

The flags are discovery-only.

Presets

dstack preset create gains a pinned benchmark workload, a baseline trial to measure against, and a per-trial record of what each trial taught. The properties that decide what "verified" means are now required rather than optional, so two creations from the same configuration produce comparable presets.

type: preset
name: dsv4-flash

# The agent picks a compatible variant of the base model
base: deepseek-ai/DeepSeek-V4-Flash

# Consider only the specified fleets
fleets: [b200-2x]

# The requirements the preset must meet (time to first token is in milliseconds)
min_context_length: 1048576
max_ttft: 675

# The number of simultaneous requests every benchmark uses
concurrency: 1

# The request shape every benchmark uses (defaults to 1024 and 1024)
input_tokens: 10000
output_tokens: 1500

# Make the first trial a reference point rather than an optimization attempt
baseline: true

# The number of benchmarked trials
trials: 10

Each trial now records what it learned, whether it broke a constraint, and the largest context it actually handled, so context_length on a saved preset is measured rather than assumed. A trial that breaks a constraint keeps its benchmark for the next trial to learn from and is excluded from best-trial selection.

dstack preset gains ps-style filtering with -a, -n, --base, and --repo, CONSTRAINTS and BENCHMARK columns, and one glyph per trial:

$ dstack preset -a --base deepseek-ai/DeepSeek-V4-Flash
 ID        BASE                           GPU           CONSTRAINTS                   BENCHMARK                          STATUS              SUBMITTED
 c83375b4  deepseek-ai/DeepSeek-V4-Flash  B200:180GB:2  io=10000/1500 conc=1          tok/s/user=319 ttft=220ms ctx=1M    interrupted (5/10)  8 hours ago
 de7a07c1  deepseek-ai/DeepSeek-V4-Flash  B200:2        io=10000/1500 conc=1          tok/s/user=140 ttft=519ms ctx=1M    verified (5)        19 hours ago
 014c3216  deepseek-ai/DeepSeek-V4-Flash  H100:80GB:4   io=8K/1K prefix=90% conc=648  tok/s/user=11.2 ttft=4.89s ctx=1M   verified (8)        yesterday

Presets remain an experimental feature. The configuration properties changed in a backward-incompatible way, see the breaking changes below.

Backends

AWS

The AWS backend gains an experimental_instance_types setting that allows provisioning instance types outside the standard supported families, as long as they are present in dstack's pricing catalog:

projects:
  - name: main
    backends:
      - type: aws
        creds:
          type: default
        experimental_instance_types: [p5en.48xlarge]
$ dstack offer -b aws --instance-type p5en.48xlarge
 #   BACKEND           RESOURCES                                              INSTANCE TYPE  PRICE
 1   aws (us-east-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $26.6319
 2   aws (us-west-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8 (spot)  p5en.48xlarge  $26.9512
 3   aws (us-east-2)   cpu=192 mem=2048GB disk=100GB gpu=H200:141GB:8         p5en.48xlarge  $63.296

Slurm

Slurm offers no longer report a fake disk size derived from the requested disk.size range. Disk size is now reported as unknown, and offers are no longer filtered by the disk requirement instead of promising storage that may not be there.

Breaking changes

  • The dstack Python API and plugins now work with Pydantic v2 models. Update the code before upgrading dstack.
  • Dropped the /api/project/{project_name}/runs/submit endpoint, deprecated in favor of /api/project/{project_name}/runs/apply.
  • Dropped the /api/project/{project_name}/fleets/create endpoint, deprecated in favor of /api/project/{project_name}/fleets/apply.
  • Dropped the top-level router property from gateway and run configurations, deprecated since 0.20.17 in favor of replica-based routers.
  • The server no longer accepts resources.cpu as an integer range in the request format used by clients older than 0.19.8. Upgrade the CLI and API clients before upgrading the server.
  • UTC datetimes are serialized as Z instead of +00:00.
  • Preset configurations (experimental): max_trials is now trials, context_length is now min_context_length, and max_ttft, min_context_length, and concurrency no longer have defaults. Existing configurations fail with extra fields not permitted; no aliases were added.

Deprecations

  • Resources.description, Gateway.backend, and Gateway.region are no longer populated by the server and are excluded by the client. They will be removed in 0.22.

What's Changed

  • Use test log storage in test_updates_running_job by @un-def in #4065
  • Add full_offers option to Compute.get_offers() by @un-def in #4060
  • Add in-place update for gateway replicas by @jvstme in #4014
  • Set dstack.user.name attr on OTel traces by @r4victor in #4068
  • Add support for offers with unknown disk size by @un-def in #4066
  • Add unallocated_resources option to Compute.get_offers() by @un-def in #4067
  • Add AWS experimental_instance_types setting by @jvstme in #4069
  • Prepare for Pydantic v2 migration by @r4victor in #4072
  • Support replicated AWS gateways with ACM by @jvstme in #4071
  • ...
Read more

Contributors

complynx, un-def, and 4 other contributors
Loading

0.20.29

0.20.29

Choose a tag to compare

View all tags
@peterschmidt85 peterschmidt85 released this 24 Jul 15:55
0.20.29
2f9618f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Presets

This release introduces dstack preset, an agent-driven inference optimization toolkit. A headless agent runs sequential trials on your fleets, benchmarks serving configurations, and saves the best one as a reusable preset.

To create a preset, define a preset configuration and pass it to dstack preset create:

type: preset
name: qwen3-coder

# The agent picks a compatible variant of the base model
base: Qwen/Qwen3-Coder-30B-A3B-Instruct

# Consider only the specified fleets
fleets: [h200-fleet]

env:
  - HF_TOKEN

# The number of benchmarked trials
max_trials: 5

By default, the agent may choose the hardware among the fleet's offers, the serving framework and its flags, and, when base is specified, any compatible variant of the model, including a different precision or quantization. The prompt property can change the objectives and extend what the agent is allowed to do, from custom target metrics to experiments it would not attempt on its own.

$ dstack preset create -f preset.dstack.yml
...
Create the preset qwen3-coder? [y/n]: y
[2026-07-23 09:11:40] Submitted trial task qwen3-coder-4c9a17d5-1 (vllm/vllm-openai:latest, H100:80GB). Trial 1: bf16, defaults, max-model-len 32768, establishing the baseline.
[2026-07-23 09:46:02] Trial 1 complete. Baseline: 780 tok/s, TTFT 610ms (concurrency 8, 4096 in / 512 out). Decode underutilizes the card; bf16 weights cap context at 32K.
[2026-07-23 10:12:29] Trial 2 complete (official FP8 variant, chunked prefill tuned). 1240 tok/s, TTFT 340ms; max-model-len now 131072.
[2026-07-23 10:41:55] Trial 3 complete (same config on H200 141GB). 1685 tok/s, TTFT 295ms; KV headroom allows the full native 262144 context at concurrency 8.
[2026-07-23 11:07:13] Trial 4 complete (prefix caching + cascade attention). 1608 tok/s (-5%). Rejected: random benchmark prompts get zero cache hits and pay the hashing overhead.
[2026-07-23 11:34:47] Trial 5 complete (batched tokens 16384, full CUDA graphs, async scheduling). 2140 tok/s, TTFT 262ms. Best config: trial 5, with 2.7x throughput over baseline.
[2026-07-23 11:52:24] Final service verified: readiness probe passed, context length confirmed at 262144. Final benchmark through the service URL: 2050 tok/s, TTFT p50 318ms. Report written.

dstack preset lists saved presets along with in-progress creations:

$ dstack preset list
 BASE                                ID        GPU                  BENCHMARK                     STATUS          SUBMITTED    NAME
 Qwen/Qwen3-Coder-30B-A3B-Instruct
                                     4c9a17d5  H200:141GB:1         con=8 2140 tok/s TTFT 262ms   verified (5/5)  3 min ago    qwen3-coder
 Qwen/Qwen3.5-27B
                                     b7e42f19  RTXPRO4500:32GB:1..  con=8 342 tok/s TTFT 389ms    verified (5/5)  yesterday    qwen35-27b

To deploy a preset as a standard dstack service, pass its ID to dstack preset apply:

$ dstack preset apply -f preset.dstack.yml --id 4c9a17d5

Presets are an experimental feature. See the documentation for more details, and share feedback via GitHub issues.

Backends

AWS

NVIDIA RTX PRO 4500

dstack now supports AWS G7 instance types with NVIDIA RTX PRO 4500 Blackwell GPUs, from the single-GPU g7.2xlarge up to the 8-GPU g7.48xlarge, including spot:

$ dstack offer -b aws --gpu RTXPRO4500
 #   BACKEND          RESOURCES                                                  INSTANCE TYPE  PRICE
 1   aws (us-east-2)  cpu=8 mem=32GB disk=100GB gpu=RTXPRO4500:32GB:1 (spot)     g7.2xlarge     $0.3963
 2   aws (us-east-2)  cpu=32 mem=128GB disk=100GB gpu=RTXPRO4500:32GB:1 (spot)   g7.8xlarge     $0.4086
 3   aws (us-east-2)  cpu=16 mem=64GB disk=100GB gpu=RTXPRO4500:32GB:1 (spot)    g7.4xlarge     $0.4965
 4   aws (us-east-2)  cpu=48 mem=192GB disk=100GB gpu=RTXPRO4500:32GB:2 (spot)   g7.12xlarge    $0.7128
 5   aws (us-east-2)  cpu=96 mem=384GB disk=100GB gpu=RTXPRO4500:32GB:4 (spot)   g7.24xlarge    $1.4257
 6   aws (us-east-2)  cpu=192 mem=768GB disk=100GB gpu=RTXPRO4500:32GB:8 (spot)  g7.48xlarge    $2.8513
 7   aws (us-east-1)  cpu=8 mem=32GB disk=100GB gpu=RTXPRO4500:32GB:1            g7.2xlarge     $2.52
 8   aws (us-east-2)  cpu=16 mem=64GB disk=100GB gpu=RTXPRO4500:32GB:1           g7.4xlarge     $3.0421

What's Changed

  • Refactor get_job_plans() by @un-def in #4042
  • Set DSTACK_OTEL_METRICS_EXPORTERS=otlp by default by @r4victor in #4041
  • Do not instrument http+unix:// requests by @r4victor in #4044
  • CLI: add --no-profile option by @un-def in #4043
  • Use OTel LoggingHandler from opentelemetry-instrumentation-logging by @r4victor in #4050
  • Don't request backend offers if creation_policy: reuse or instances is set by @un-def in #4046
  • CLI: rework offer command by @un-def in #4047
  • Update gpus/list API method to match runs/get_plan by @un-def in #4051
  • Fix creation_policy: reuse ignored by @r4victor in #4053
  • [UI] Change GPU offer resources spec to match dstack offer command by @un-def in #4052
  • Forbid updating failed gateways in-place by @jvstme in #4055
  • Enforce /api/runs/list job_submissions_limit on the server side by @r4victor in #4057
  • Delete dull job and instance models created on no capacity retry by @r4victor in #4059
  • [Feature] Presets by @peterschmidt85 in #4058
  • Support AWS G7 instances with NVIDIA RTX PRO 4500 by @jvstme in #4063
  • Reject StaticFiles websocket requests gracefully by @r4victor in #4064
  • Implement within_fleets filter via entity_fleet_id by @r4victor in #4062

Full Changelog: 0.20.28...0.20.29

Contributors

un-def, r4victor, and 2 other contributors
Loading

0.20.28

0.20.28

Choose a tag to compare

View all tags
@jvstme jvstme released this 16 Jul 14:58
0.20.28
b2459c7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Runs

dstack inside dstack

Sometimes, runs may need to call dstack themselves, for example, to inspect runs, submit another run, or attach to one.

This update makes this easy:

type: task
image: dstackai/dstack
dstack: true
env:
  - DSTACK_TOKEN
commands:
  - dstack ps

In this case, dstack exposes mounts /run/dstack/server.sock inside the container and sets DSTACK_SERVER_URL and DSTACK_PROJECT accordingly.

This gives the run access to the dstack server, allowing it to manage existing runs, submit new ones with dstack apply, and attach to them with dstack attach.

Authentication remains explicit, so DSTACK_TOKEN must be passed through env or secrets.

See documentation for more details.

Server

OpenTelemetry

The dstack server now integrates with OpenTelemetry SDK for exporting server traces, logs, and metrics. This can be used for monitoring the server via a LGTM (Grafana) observability stack or any other solution supporting OLTP. See documentation for more details.

Backends

Vast.ai spot instances

The vastai backend now supports interruptible spot instances, offering more competitive prices.

$ dstack offer -b vastai --gpu B200,B300 --spot-auto
 #   BACKEND               RESOURCES                                              INSTANCE TYPE  PRICE
 1   vastai (us-virginia)  cpu=96 mem=510GB disk=100GB gpu=B200:180GB:4 (spot)    45069637       $25.1215
 2   vastai (us-virginia)  cpu=96 mem=510GB disk=100GB gpu=B200:180GB:4           45069637       $28.6215
 3   vastai (ca-)          cpu=192 mem=2059GB disk=100GB gpu=B300:270GB:8 (spot)  43033760       $40.0694
 4   vastai (ca-)          cpu=192 mem=2059GB disk=100GB gpu=B300:270GB:8         43033760       $62.5694
 5   vastai (us-virginia)  cpu=192 mem=2043GB disk=100GB gpu=B200:180GB:8 (spot)  45069635       $50.1215
 6   vastai (us-virginia)  cpu=192 mem=2043GB disk=100GB gpu=B200:180GB:8         45069635       $57.1215
 7   vastai (us-oregon)    cpu=24 mem=31GB disk=100GB gpu=B200:180GB:1 (spot)     33945613       $5.651
 8   vastai (us-oregon)    cpu=24 mem=31GB disk=100GB gpu=B200:180GB:1            33945613       $6.901
 9   vastai (us-virginia)  cpu=48 mem=127GB disk=100GB gpu=B200:180GB:2 (spot)    45081639       $15.1215
 10  vastai (us-virginia)  cpu=48 mem=127GB disk=100GB gpu=B200:180GB:2           45081639       $16.3715

What's changed

New contributors

Full changelog: 0.20.27...0.20.28

Contributors

trashhalo, un-def, and 3 other contributors
Loading

0.20.27

0.20.27

Choose a tag to compare

View all tags
@r4victor r4victor released this 09 Jul 13:10
0.20.27
82f9620
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Backends

Slurm

This release adds Slurm as a new backend, allowing dstack to orchestrate container-based runs across your existing Slurm clusters. A single slurm backend can manage one or many clusters — dstack connects to each cluster's login node over SSH and submits runs as Slurm jobs, with every cluster becoming its own dstack region.

projects:
- name: main
  backends:
    - type: slurm
      clusters:
        - name: gpu-cluster-a
          hostname: login.example.com
          user: admin
          private_key:
            path: ~/.ssh/id_rsa
          gpu_partitions:
            - gpu: H100
              partitions: [gpu]

Refer to the Backends docs for prerequisites, partition configuration, and known limitations.

Runs

CUDA 13.0

dstack base Docker images have CUDA runtime updated from version 12.8 to 13.0.

Note: CUDA 13 dropped support for Nvidia Volta (V100) and older architectures. If your workload depends on dstack-shipped CUDA 12.8 runtime, install the CUDA runtime manually or use a custom Docker image.

Breaking changes

  • Dropped P100 provisioning from GCP and OCI.

Deprecations

  • dstack gateway update: the --domain argument is deprecated in favor of gateway configuration in-place update.
  • /api/project/{project_name}/gateways/create and /api/project/{project_name}/gateways/set_wildcard_domain API methods are deprecated in favor of plan-based API: /api/project/{project_name}/gateways/get_plan and /api/project/{project_name}/gateways/apply

What's Changed

Full Changelog: 0.20.26...0.20.27

Contributors

un-def, Bihan, and 4 other contributors
Loading

0.20.26

0.20.26

Choose a tag to compare

View all tags
@jvstme jvstme released this 25 Jun 19:29
0.20.26
5e01dff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Server

SSH pool

The server now keeps a pool of reusable SSH connections to instances, enabled by default. Previously, the server opened a fresh SSH connection for every operation against an instance. Reusing pooled connections removes this per-operation overhead and delivers a significant performance boost — runs, dev environments, and services become noticeably more responsive, especially on servers managing many instances.

ssh-pool

The SSH pool is on by default and requires no configuration. If needed, you can opt out by setting the DSTACK_SERVER_SSH_POOL_DISABLED environment variable:

DSTACK_SERVER_SSH_POOL_DISABLED=1

Faster run listing

The /api/runs/list endpoint has been optimized to load jobs more efficiently. Listing runs — including in the UI and via dstack ps — is now faster, particularly for projects with a large number of runs.

Backends

AWS

Capacity Reservations

dstack now applies the tenancy of an EC2 On-Demand Capacity Reservation when launching instances into it. Because a Capacity Reservation only accepts instances whose attributes — instance type, platform, Availability Zone, and tenancy — match the reservation, this ensures instances with a dedicated tenancy reservation launch correctly instead of being rejected.

What's changed

Full changelog: 0.20.25...0.20.26

Contributors

Bihan, r4victor, and 2 other contributors
Loading

0.20.25

0.20.25

Choose a tag to compare

View all tags
@r4victor r4victor released this 18 Jun 11:12
0.20.25
9c1e460
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Runs

Ubuntu 24.04

dstack's base Docker images have been upgraded from Ubuntu 22.04 to Ubuntu 24.04. This means runs are now executed in the Ubuntu 24.04 containers unless image is specified. See the Ubuntu 24.04 LTS release notes for more details.

Note: If your runs hard-depend on the previous Ubuntu version, specify image in the run configuration explicitly:

type: task
image: dstackai/base:0.13-base-ubuntu22.04
commands: ...

Instances

Run configurations now support instances property that allows provisioning runs only on the specified instances:

type: dev-environment
ide: vscode
instances: [{fleet: my-ssh-fleet, instance: 0}]

This can be useful if, for example, a run depends on an Instance Volume that exists on a specific SSH instance.

See the reference for different syntax options supported by instances.

Gateways

Replicas

A gateway can now have multiple replicas for improved availability and scalability:

type: gateway
name: example-gateway

backend: aws
region: eu-west-1

domain: example.com

certificate: null
replicas: 2

To balance requests between gateway replicas, add DNS records for each replica or set up a load balancer outside of dstack.

Note: Automatic HTTPS is not supported for replicated gateways. Use an external load balancer for TLS termination.

Replicated gateways are an experimental feature. See the docs for all the limitations.

Backends

AWS

NVIDIA B200 and B300

dstack now supports AWS p6-b200 and p6-b300 instance types, with max-throughput EFA networking setup out-of-the-box. p6-b300 is the first instance type natively supported by dstack that comes with NVIDIA Blackwell Ultra B300 GPUs and 6,400 Gbps networking. Give it a try:

✗ dstack apply -f b300-fleet.dstack.yml
...
 #  BACKEND  REGION     INSTANCE          RESOURCES                                         SPOT  PRICE
 1  aws      us-east-1  p6-b300.48xlarge  cpu=192 mem=4096GB disk=100GB gpu=B300:268.6GB:8  yes   $33.082
 2  aws      us-west-2  p6-b300.48xlarge  cpu=192 mem=4096GB disk=100GB gpu=B300:268.6GB:8  yes   $34.4876
 3  aws      us-west-2  p6-b300.48xlarge  cpu=192 mem=4096GB disk=100GB gpu=B300:268.6GB:8  no    $142.416
    ...
 Shown 3 of 4 offers, $142.416 max

What's changed

Full changelog: 0.20.24...0.20.25

Contributors

Bihan, r4victor, and 2 other contributors
Loading

0.20.25rc1

0.20.25rc1 Pre-release
Pre-release

Choose a tag to compare

View all tags
@peterschmidt85 peterschmidt85 released this 12 Jun 15:38
0.20.25rc1
c1d9fa9

Instances

Run configurations now support the instances property for targeting specific fleet instances.

When instances is set, the run is placed only on matching existing fleet instances. If the specified instances cannot be used, the run fails instead of provisioning new instances.

Target by instance name:

instances:
  - name: my-fleet-0

The short syntax is an instance name string:

instances:
  - my-fleet-0

Target by hostname or IP address:

instances:
  - hostname: 10.0.1.42

Target by fleet and instance number:

instances:
  - fleet: my-fleet
    instance: 0

For fleets from another project, use the <project>/<fleet> reference:

instances:
  - fleet: shared-project/my-fleet
    instance: 0

Multiple instances can be specified:

instances:
  - my-fleet-0
  - my-fleet-1

What's changed

Full changelog: 0.20.24...0.20.25rc1

Note

The public documentation will be updated when the release becomes GA.

Contributors

fededagos and peterschmidt85
Loading

0.20.24

0.20.24

Choose a tag to compare

View all tags
@peterschmidt85 peterschmidt85 released this 11 Jun 13:55
0.20.24
cd0e93c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Dev environments

Zed

dstack now supports Zed as a dev environment IDE:

type: dev-environment
ide: zed
resources:
  gpu: L4

Once the dev environment is up, the CLI prints a zed:// link that opens the remote project in Zed over SSH. Since Zed doesn't require any plugins, no server pre-installation is needed — the Zed server is installed automatically on first connect.

✗ dstack apply
...
Submit a new run? [y/n]: y
 NAME                     BACKEND                  GPU                     PRICE       STATUS      SUBMITTED
 fast-fly-1               aws (us-east-2)          gpu=L4:24GB:1           $0.1838     running     16:36
                                                                           (spot)

fast-fly-1 provisioning completed (running)
pip install ipykernel...

To open in Zed, use link below:

  zed://ssh/fast-fly-1/dstack/run

To connect via SSH, use: `ssh fast-fly-1`

To exit, press Ctrl+C.

Services

Replica groups

The spot_policy and reservation properties can now be specified at the replica group level. This allows distributing replicas across reserved and spot capacity, e.g., running baseline replicas on a reservation while autoscaling overflow replicas on spot instances:

type: service
image: my-image
port: 80

replicas:
  - name: baseline
    reservation: my-reservation
    count: 1

  - name: overflow
    spot_policy: auto
    count: 0..3
    scaling:
      metric: rps
      target: 1

Shepherd Model Gateway

Services using Shepherd Model Gateway now support gRPC communication with both vLLM and SGLang workers. Previously, only the SGLang runtime with the HTTP connection mode was supported.

Below is an example service configuration running vLLM gRPC workers:

type: service
name: prefill-decode

env:
  - HF_TOKEN
  - MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
  - count: 1
    image: python:3.12-slim
    commands:
      - pip install smg
      - |
          smg launch \
            --pd-disaggregation \
            --model-path $MODEL_ID \
            --enable-igw \
            --host 0.0.0.0 \
            --port 8000 \
            --prefill-policy cache_aware
    router:
      type: sglang
    resources:
      cpu: 4

  - count: 1
    image: vllm/vllm-openai:latest
    commands:
      - pip install -U "vllm[grpc]"
      - |
          python3 -m vllm.entrypoints.grpc_server \
            --model $MODEL_ID \
            --host 0.0.0.0 \
            --port 8000 \
            --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
    resources:
      gpu: H200

  - count: 1
    image: vllm/vllm-openai:latest
    commands:
      - pip install -U "vllm[grpc]"
      - |
          python3 -m vllm.entrypoints.grpc_server \
            --model $MODEL_ID \
            --host 0.0.0.0 \
            --port 8000 \
            --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
    resources:
      gpu: H200

port: 8000

dstack automatically detects each worker's runtime (vLLM or SGLang) and connection mode (HTTP or gRPC) by probing it. With gRPC, the SMG router tokenizes requests once and routes on tokens instead of raw text, reducing duplicate work and making cache_aware routing more effective.

JarvisLabs

The jarvislabs backend now supports offers with RTXPRO6000 GPUs.

Azure

subnet_ids

Similarly to vpc_ids, the azure backend now allows selecting specific subnets to be attached to dstack VMs via the new subnet_ids property, mapping regions to subnets in the <resource-group>/<vnet>/<subnet> format:

projects:
  - name: main
    backends:
      - type: azure
        subscription_id: ...
        tenant_id: ...
        creds:
          type: default
        regions: [westeurope]
        subnet_ids:
          westeurope: my-resource-group/my-vnet/my-subnet

This is useful when the VNet contains subnets that dstack shouldn't pick automatically, e.g. subnets delegated to other Azure services.

What's changed

  • Fix zero scaled services assigned to wrong fleets by @r4victor in #3939
  • Set runner/shim default compiled versions to latest by @r4victor in #3941
  • Implement SSH connection pool for runner instances by @r4victor in #3936
  • [chore]: Move format_backend() to common utils by @jvstme in #3942
  • Drop non-linux runner builds and local backend by @r4victor in #3944
  • Support Zed as dev-environment IDE by @r4victor in #3947
  • Fix dropping ssh connections to non-provisioned terminating instances by @r4victor in #3948
  • Replica group spot_policy and reservation by @jvstme in #3932
  • Fix jpd.hostname AssertionError on container stop by @r4victor in #3951
  • Add NVIDIA Dynamo blog post by @peterschmidt85 in #3949
  • Support gRPC communication with SMG (Shepherd Model Gateway) workers by @Bihan in #3946
  • Allow configuring subnet_ids in Azure settings by @jvstme in #3955
  • [JarvisLabs] Support RTX PRO 6000; update gpuhunt dependency by @peterschmidt85 in #3943

Full changelog: 0.20.23...0.20.24

Contributors

Bihan, r4victor, and 2 other contributors
Loading

0.20.23

0.20.23

Choose a tag to compare

View all tags
@jvstme jvstme released this 04 Jun 10:20
0.20.23
60bbb7e
This commit was signed with the committer’s verified signature.
jvstme
GPG key ID: A211055E243C1667
Verified
Learn about vigilant mode.

This release includes several bug fixes and performance optimizations.

What's Changed

  • [Internal]: Fix OCI image publishing script by @jvstme in #3915
  • Update Docker and cloud images to 0.13 by @jvstme in #3916
  • [shim] Pass proxy variables to the container by @un-def in #3917
  • Fix image pull progress when reported in seconds by @jvstme in #3921
  • Skip getting backend offers when instance offers suffice by @r4victor in #3923
  • Reduce run provisioning pipeline processing latency by @r4victor in #3922
  • Do not generate RSA key for runner sshd by @r4victor in #3926
  • Handle repo patch with non-UTF8 sequences by @un-def in #3918
  • Fix Verda spot offers marked unavailable due to on-demand-only availability check by @IA386 in #3928

New Contributors

Full Changelog: 0.20.22...0.20.23

Contributors

un-def, r4victor, and 2 other contributors
Loading