Support OCI spot instances (#2337)

Migrate to v2 `gpuhunt` catalogs and enable the
`oci-spot` flag to see OCI spot offers. The actual
support for provisioning spot instances was
implemented earlier.

Also handle out of capacity errors.
Spot capacity does not always match the capacity
returned in OCI capacity reports, so handle out of
capacity errors gracefully. OCI returns these
errors with the `InternalError` code, so we'll
need to rely on error messages to detect them.

Author: jvstme

contributing/GPUHUNT.md

@@ -95,3 +95,18 @@ Some providers offer extreme flexibility in possible configurations, but not all
 The offline catalog is built in GitHub Actions every night. Every offline provider produces a CSV file with offers. Later, those files get compressed into a zip archive and uploaded to the public S3 bucket.
 
 To ensure data quality, there is a catalog integrity testing step. It uses some simple heuristics to avoid empty catalog files, zero prices, or missing regions.
+
+### Backward compatibility
+
+The same `gpuhunt` version can be used by different `dstack` versions.
+Additionally, offline catalogs are produced by the latest `gpuhunt` version, but used by all `dstack` versions.
+
+These mechanisms are used to preserve backward compatibility:
+
+- **`gpuhunt` version**: The interfaces in the `gpuhunt` package preserve backward compatibility
+  within a minor version (`X` in `0.X.Y`).
+- **Offer flags**: If an offer breaks older `dstack` versions, it is marked with a flag in `RawCatalogItem.flags`
+  and the flag is added to the list of supported flags in `dstack`.
+  Older `dstack` versions that don't support this flag will not see the respective offers.
+- **Offline catalog versions**: If a breaking change in the structure or content of an offline catalog is unavoidable,
+  a new version of the catalog can be introduced. Catalog versions are published at `s3://dstack-gpu-pricing/v{N}`.

setup.py

@@ -48,7 +48,7 @@ def get_long_description():
     "python-multipart>=0.0.16",
     "filelock",
     "psutil",
-    "gpuhunt>=0.0.19,<0.1.0",
+    "gpuhunt>=0.1.0,<0.2.0",
     "argcomplete>=3.5.0",
 ]
 

src/dstack/_internal/core/backends/base/offers.py

@@ -14,6 +14,12 @@
 from dstack._internal.core.models.resources import DEFAULT_DISK, Memory, Range
 from dstack._internal.core.models.runs import Requirements
 
+# Offers not supported by all dstack versions are hidden behind one or more flags.
+# This list enables the flags that are currently supported.
+SUPPORTED_GPUHUNT_FLAGS = [
+    "oci-spot",
+]
+
 
 def get_catalog_offers(
     backend: BackendType,
@@ -110,7 +116,7 @@ def offer_to_catalog_item(offer: InstanceOffer) -> gpuhunt.CatalogItem:
 
 
 def requirements_to_query_filter(req: Optional[Requirements]) -> gpuhunt.QueryFilter:
-    q = gpuhunt.QueryFilter()
+    q = gpuhunt.QueryFilter(allowed_flags=SUPPORTED_GPUHUNT_FLAGS)
     if req is None:
         return q
 

src/dstack/_internal/core/backends/oci/compute.py

@@ -163,7 +163,7 @@ def create_instance(
                 image_id=package.image_id,
             )
         except oci.exceptions.ServiceError as e:
-            if e.code in ("LimitExceeded", "QuotaExceeded"):
+            if e.code in ("LimitExceeded", "QuotaExceeded") or "Out of host capacity" in e.message:
                 raise NoCapacityError(e.message)
             raise