Skip to content

[Feature] Allow configuring multiple repos per run #2738

Open
@peterschmidt85

Description

@peterschmidt85

Currently, dstack allows to configure a single repo which is used with all runs. This repo can be a remote url, a remote cloned repo dir, or a local folder. Moreover, this repos is always mountd into /workflow which is the default working directory.

This behaviour is not very flexible as users may want to mount multiple repos per run configuration. And each run configuration may use different repos.

Below is the suggested change:

Iteration 1 (now)

  1. (Required) Introduce the files property
files:
  - .

or

files:
  - .:/workflow

or

files:
  - examples/
    path: /workflow/examples

or

files:
  - ./examples
    path: ./examples
  1. (Required) Deprecate local repos with dstack init and dstack apply
  • Drop the --local argument
  1. (Required) Make repo configuration more explicit
  • Ensure dstack init explicitly tells if a repo was configured or not (in addition to OK)
  • Ensure the dstack apply command tells if a repo configured or not (as a part of the plan)
  1. (Recommended) Improve the working_dir behaviour
  • Automatically detect the working_dir from the Docker image
  • Allow to set an absolute path to working_dir

Without it, specifying a path under files can be quite counter-intuitive:

working_dir: /examples # if not specified, taken from the image
files:
  - examples
    path: .
  1. (Optional) See if there are better alternative namings to files

Iteration 2 (later)

1. Introduce the repos property (similar to files)

repos:
  - .:/workflow

or

repos:
  - .

or

repos:
  - repo_path:  .
     path: .
     identity_key: ~/.ssh/id_rsa

or

repos:
  - repo_path:  .
     identity_key: ~/.ssh/id_rsa

or

repos:
  - repo_path:  .
     token: {env. GH_TOKEN}

Note, if the repo is remote, dstack will try to use the default Git credentials (as it does now) using ~/.config/gh/hosts.yml or ~/.ssh/id_rsa.

A repo path can be also a GitHub URL (as it's supported now):

repos:
  - repo_path:  https://github.com/huggingface/open-r1

or

repos:
  - https://github.com/huggingface/open-r1:/workflow

2. Introduce ssh_config.identity_file
3. Deprecate dstack init

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions