chore: remove docker-compose from det deploy local [MLG-37]

[azhou-determined]

Description

replace docker-compose dependency (in det deploy local) with docker run, which is already used to spin up agents.

Test Plan

[ ] det deploy local cluster-up/down
[ ] det deploy local master-up/down

Migration path:
[ ] det deploy local cluster-up before this change, then det deploy local cluster-up after this change. DB should be persisted unless delete-db was specified.

Commentary (optional)

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

[rb-determined-ai]

you might consider the the whole comment and associated requirement, rather than just the line containing the text docker-compose lol

[rb-determined-ai]

What is the purpose of adding this one?

[azhou-determined]

1. cluster-up and agent-up both have these, and 2) the method this CLI calls handle_master_up actually takes in args.image_repo_prefix even though we never let it be specified.

[rb-determined-ai]

So, this cli is a public interface.

Do we want to offer an --image-repo-prefix? Why not just offer e.g. a --master-image and an --agent-image or something more direct? Or not add it to the public interface if we haven't thought out fully?

[azhou-determined]

yeah, i don't disagree that master-image and agent-image might be better. my reasoning for this was simply to make it more consistent with what the API already does. we already take in image-repo-prefix for the other CLI calls, and it seemed like a bug that we didn't have this here. before, we still technically accepted the argument, we would just default inside of the method, making the defaults not visible and not configurable by the user. i can remove it if you think it's not an overall gain.

[rb-determined-ai]

I'll trust your judgement on this.

[rb-determined-ai]

where is this from? Don't we always use ruamel.yaml?

[azhou-determined]

nope. do a global search for import yaml, you'll find a ton of references to pyyaml (yaml).

[rb-determined-ai]

can you just fix those instead? The pyyaml library has issues, and we set up a better solution that is much more widely used:

rb@bearasaurusrex ~/code/determined/harness/determined (git)-[master]
% git grep 'common.*import.*yaml'
cli/cli.py:from determined.common import api, yaml
cli/command.py:from determined.common import api, context, util, yaml
cli/experiment.py:from determined.common import api, context, set_logger, util, yaml
cli/master.py:from determined.common import api, yaml
cli/render.py:from determined.common import util, yaml
cli/template.py:from determined.common import api, util, yaml
common/experimental/determined.py:from determined.common import api, context, util, yaml
common/util.py:from determined.common import yaml
deploy/gke/cli.py:from determined.common import yaml
deploy/gke/cli.py:from determined.common.util import safe_load_yaml_with_exceptions
deploy/local/cluster_utils.py:from determined.common import yaml

vs

rb@bearasaurusrex ~/code/determined/harness/determined (git)-[master]
% git grep '^import yaml'
cli/job.py:import yaml
deploy/aws/gen_vcpu_mapping.py:import yaml
deploy/aws/preflight.py:import yaml

[rb-determined-ai]

Looks great! New det deploy local operates perfectly on old-det deploy local-deployed cluster. Users are unlikely to even notice.

I'm still suspicious of that pyyaml dependency though.

[rb-determined-ai]

so to be clear, this is something we already depended on, indirectly, only now we are less strict about the version overall, and also it's a dependency we're going to remove in the nearish future, right?

[rb-determined-ai]

so long overdue!