Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] Limit num gpus if specified for A100 #5694

Merged
merged 3 commits into from
Jun 19, 2024
Merged

[ci] Limit num gpus if specified for A100 #5694

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7f913d9
p
khluu Jun 19, 2024
1f950ee
p
khluu Jun 19, 2024
363640f
p
khluu Jun 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/test-template-aws.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ steps:
- "'cd {{ (step.working_dir or default_working_dir) | safe }} && {{ step.command or (step.commands | join(' && ')) | safe }}'"
resources:
limits:
Comment on lines 75 to 77
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more nit comment: is it necessary that we duplicate part of code here? is it possible to reuse the existing code, and only modify say queue based on the step.gpu value?

Copy link
Collaborator Author

@khluu khluu Jun 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our A100 is set up in the EKS cluster so it needs the k8s plugin here to configure the pod (see podSpec) that the cluster launchs to run the job. Other jobs run in a Docker container inside an individual EC2 instance launched by AWS autoscaling group so it's using Docker plugin.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the plugins used are different, I don't think we can reuse most of the code here

nvidia.com/gpu: 8
nvidia.com/gpu: {{ step.num_gpus or 8 }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be 1 by default?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes. updated to 1

volumeMounts:
- name: devshm
mountPath: /dev/shm
Expand Down
Loading