[DE-204] Parse requirements when using a package with commit hash

[mimarcel]

Why

• Building a model with requirement clarifai@git+https://github.com/Clarifai/clarifai-python.git@9131f191fa06ec14cb9efe65ae5d4135f989bdbe returns

  Exception: clarifai not found in requirements.txt, please add clarifai to the requirements.txt file with a fixed version. Current version is clarifai==11.9.0
  

How

• support @ separator when parsing requirements

Tests

• manual test: upload model using the following files

requirements.txt

clarifai@git+https://github.com/Clarifai/clarifai-python.git@9131f191fa06ec14cb9efe65ae5d4135f989bdbe

Dockerfile

# syntax=docker/dockerfile:1.13-labs

# Build stage: install git and find dependencies
FROM python:3.12-slim-bookworm as build
RUN apt-get update && \
    apt-get install -y --no-install-recommends git && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /git-libs && \
    ldd /usr/bin/git | awk '/=>/ {print $3}' | xargs -I '{}' cp --parents '{}' /git-libs/
RUN mkdir -p /git-remote-https-libs && \
    ldd /usr/lib/git-core/git-remote-https | awk '/=>/ {print $3}' | xargs -I '{}' cp --parents '{}' /git-remote-https-libs/

# Final stage: distroless image
FROM --platform=$TARGETPLATFORM public.ecr.aws/clarifai-models/python-base:3.12-df565436eea93efb3e8d1eb558a0a46df29523ec as final

# Copy git binary and its libraries
COPY --from=build /usr/bin/git /usr/bin/git
COPY --from=build /git-libs/ /
# Copy git-remote-https and its dependencies
COPY --from=build /usr/lib/git-core/git-remote-https /usr/lib/git-core/git-remote-https
COPY --from=build /git-remote-https-libs/ /

COPY --link requirements.txt /home/nonroot/requirements.txt

# Update clarifai package so we always have latest protocol to the API. Everything should land in /venv
RUN ["pip", "install", "--no-cache-dir", "-r", "/home/nonroot/requirements.txt"]
RUN ["pip", "show", "clarifai"]

# Set the NUMBA cache dir to /tmp
# Set the TORCHINDUCTOR cache dir to /tmp
# The CLARIFAI* will be set by the templaing system.
ENV NUMBA_CACHE_DIR=/tmp/numba_cache \
    TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_cache \
    HOME=/tmp \
    DEBIAN_FRONTEND=noninteractive

#####
# Copy the files needed to download
#####
# This creates the directory that HF downloader will populate and with nonroot:nonroot permissions
# up.
COPY --chown=nonroot:nonroot downloader/unused.yaml /home/nonroot/main/1/checkpoints/.cache/unused.yaml

#####
# Download checkpoints if config.yaml has checkpoints.when = "build"
COPY --link=true config.yaml /home/nonroot/main/
RUN ["python", "-m", "clarifai.cli", "model", "download-checkpoints", "/home/nonroot/main", "--out_path", "/home/nonroot/main/1/checkpoints", "--stage", "build"]
#####

# Copy in the actual files like config.yaml, requirements.txt, and most importantly 1/model.py
# for the actual model.
# If checkpoints aren't downloaded since a checkpoints: block is not provided, then they will
# be in the build context and copied here as well.
COPY --link=true 1 /home/nonroot/main/1
# At this point we only need these for validation in the SDK.
COPY --link=true requirements.txt config.yaml /home/nonroot/main/

# Add the model directory to the python path.
ENV PYTHONPATH=${PYTHONPATH}:/home/nonroot/main \
    CLARIFAI_PAT=${CLARIFAI_PAT} \
    CLARIFAI_USER_ID=${CLARIFAI_USER_ID} \
    CLARIFAI_RUNNER_ID=${CLARIFAI_RUNNER_ID} \
    CLARIFAI_NODEPOOL_ID=${CLARIFAI_NODEPOOL_ID} \
    CLARIFAI_COMPUTE_CLUSTER_ID=${CLARIFAI_COMPUTE_CLUSTER_ID} \
    CLARIFAI_API_BASE=${CLARIFAI_API_BASE:-https://api.clarifai.com}

# Finally run the clarifai entrypoint to start the runner loop and local dev server.
# Note(zeiler): we may want to make this a clarifai CLI call.
ENTRYPOINT ["python", "-m", "clarifai.runners.server"]
CMD ["--model_path", "/home/nonroot/main"]
#############################

config.yaml

model:
  id: "python_string_cat"
  user_id: "user_id"
  app_id:  "app_id"
  model_type_id: "text-to-text"

build_info:
  python_version: "3.12"

inference_compute_info:
  cpu_limit: "500m"
  cpu_memory: "500Mi"
  num_accelerators: 0

1/model.py

from clarifai.runners.models.model_class import ModelClass


class MyModel(ModelClass):
  @ModelClass.method
  def predict(self, request: str) -> str:
    out = request + "Hello World"
    self.set_output_context(
      prompt_tokens=len(request),
      completion_tokens=len(out),
    )
    return out

[Copilot]

Pull Request Overview

This PR enhances the requirements.txt parsing logic in the model builder to support URL-based package specifications using the @ syntax (e.g., package @ https://...). It also refactors the code to strip whitespace from the package name once before comparison rather than stripping it from the dependency name after matching.

Key changes:

• Adds support for @ syntax in package specifications for URL-based dependencies
• Moves package name stripping to occur before comparison with dependencies list
• Returns the stripped package name instead of the dependency name

[github-actions]

Package	Line Rate	Health
clarifai	45%	❌
clarifai.cli	44%	❌
clarifai.cli.templates	33%	❌
clarifai.client	67%	➖
clarifai.client.auth	67%	➖
clarifai.constants	100%	✔
clarifai.datasets	100%	✔
clarifai.datasets.export	80%	✔
clarifai.datasets.upload	75%	➖
clarifai.datasets.upload.loaders	37%	❌
clarifai.models	100%	✔
clarifai.modules	0%	❌
clarifai.rag	72%	➖
clarifai.runners	53%	➖
clarifai.runners.models	59%	➖
clarifai.runners.pipeline_steps	41%	❌
clarifai.runners.pipelines	70%	➖
clarifai.runners.utils	63%	➖
clarifai.runners.utils.data_types	72%	➖
clarifai.schema	100%	✔
clarifai.urls	60%	➖
clarifai.utils	60%	➖
clarifai.utils.evaluation	67%	➖
clarifai.workflows	95%	✔
Summary	62% (8210 / 13327)	➖

Minimum allowed line rate is 50%