Closed
Description
For now we should do:
num_gpus=self.cache_config.gpu_memory_utilization if self.parallel_config.tensor_parallel_size < 2 else 1
Originally posted by @Yard1 in #1821 (comment)
Metadata
Metadata
Assignees
Labels
No labels
For now we should do:
num_gpus=self.cache_config.gpu_memory_utilization if self.parallel_config.tensor_parallel_size < 2 else 1
Originally posted by @Yard1 in #1821 (comment)