feat: optional term_filter for run-inference and latest-inference-cohort endpoint

[chapmanhk]

changes

To review only the latest commit: see Checkpoint term support and cohort_labels below.

Run-inference: optional term_filter list (API → Databricks)

• API (routers/models.py)

  • InferenceRunRequest: added optional term_filter: list[str] | None = None.
  • When term_filter is present: require at least one non-empty label; reject empty list (400) and empty or whitespace-only labels (400), with logging before each raise.
  • Build DatabricksInferenceRunRequest with term_filter=req.term_filter and pass it to run_pdp_inference.
  • After a successful run, log once: user, term_filter (cohorts), and job_run_id.
  • Docstring updated to describe the optional term_filter field.
  • Batch-not-found error message fixed to use len(batch_result) instead of len(inst_result).
  • Renamed res to inference_run_response in trigger_inference_run for clarity.

• Databricks (databricks.py)

  • DatabricksInferenceRunRequest: added optional term_filter: list[str] | None = None.
  • In run_pdp_inference(): build job_params dict; when req.term_filter is not None, set job_params["term_filter"] = json.dumps(req.term_filter). Omit the key when term_filter is None.
  • Improved logging: use %-style and include exception in log messages for job lookup and job run failures.

• Tests (routers/models_test.py)

  • No term_filter in request → 200 and mock called with term_filter=None.
  • Single and multiple cohorts in term_filter → 200 and mock receives the correct list.
  • Empty list, empty string in list, and whitespace-only label in term_filter → 400 with the expected detail messages.

Latest-inference-cohort endpoint and config from Databricks

• API (routers/data.py)

  • New GET /{inst_id}/latest-inference-cohort returning LatestInferenceCohortResponse (cohort_labels, valid_student_count, status, batch_name, reason). Always returns 200; uses status='invalid' with reason when no batch, missing config, or no cohort has course data.
  • Optional query params: model_name (optional): if omitted, the institution must have exactly one registered model (valid, non-deleted); otherwise invalid. When provided, the model must exist for the institution or the endpoint returns invalid. batch_name (optional): when provided (URL-decoded), cohort selection uses that batch; when omitted, uses the most recent batch that has both STUDENT and COURSE files.
  • Config is read from the model's training run in the silver volume via DatabricksControl.read_volume_training_config(inst.name, model_run_id). model_run_id is the latest model version's run ID for the chosen model.
  • New/refactored helpers: _resolve_model_name_and_run_id_for_inference (resolve model name, validate existence, fetch run ID), _load_batch_and_dataframes_for_inference (resolve batch, load config, read batch files, validate STUDENT/COURSE), get_batch_for_inference, get_batch_by_name_with_student_and_course, _batch_has_student_and_course, plus existing cohort logic split into _latest_inference_cohort_validation_response, _first_valid_from_ordered_terms, and _resolve_latest_inference_cohort_from_dataframes (≤50 lines per function). Mypy type-narrowing assertions added where optional values are guaranteed after error checks.

• Config (config.py)

  • Added ENV_TO_VOLUME_SCHEMA = {"DEV": "dev_sst_02", "STAGING": "staging_sst_01"} for Databricks volume paths. Used by databricks.read_volume_training_config and front_end_tables.get_model_cards (replacing a local dict there).

• Databricks (databricks.py)

  • New read_volume_training_config(inst_name, model_run_id): reads from /Volumes/{env_schema}/{slug}_silver/silver_volume/{model_run_id}/, finds any .toml under that path that contains [preprocessing.selection], and returns that section only (or None if missing/unreadable). Uses ENV_TO_VOLUME_SCHEMA; returns None for unsupported ENV (e.g. LOCAL). New helpers: _parse_config_toml_to_selection(raw: bytes) for TOML parsing and section extraction, and _find_selection_in_toml_under(w, directory_path, inst_name) to list directory and scan .toml files. Guard for response.contents is None before calling .read() (mypy-safe).

• Front-end (routers/front_end_tables.py)

  • get_model_cards now uses ENV_TO_VOLUME_SCHEMA from config instead of a local schema dict.

• Tests

  • routers/data_test.py: Tests for get_latest_inference_cohort (unauthorized; no model_name and no models / models exist but none registered; missing cohort column/term/config; valid; valid with batch_name; model_name not found; batch_name not found; loop-back when no course data for most recent; no cohort has course data; missing student id column; zero students meeting criteria; student_criteria references missing column); for get_latest_batch_with_student_and_course (returns batch when it has STUDENT and COURSE, returns None when no batch or no batch has both); and for apply_student_criteria_count, filter_to_most_recent_checkpoint_term, get_ordered_checkpoint_terms.
  • databricks_test.py: Tests for _parse_config_toml_to_selection (valid TOML with section, invalid/missing section); for read_volume_training_config (empty inst_name, empty model_run_id, ENV not DEV/STAGING, databricksify raises, WorkspaceClient raises, list_raises, download raises, TOML missing [preprocessing.selection], success when .toml found under run dir). Added return type -> list[DirectoryEntry] to _one_toml_entry for mypy.

Checkpoint term support and cohort_labels (latest commit)

• API (routers/data.py)

  • Config normalization: Added _normalize_selection_config_for_inference() so pipeline config can use checkpoint_term in student_criteria; at inference entry it is normalized to cohort_term once. Downstream code and DB column stay cohort_term. _allowed_checkpoint_terms_from_config now reads only cohort_term (config is always normalized first).
  • Naming: Renamed inference-path naming to "checkpoint term": e.g. get_ordered_checkpoint_terms, filter_to_most_recent_checkpoint_term, _try_checkpoint_term_candidate, _first_valid_from_ordered_terms, CHECKPOINT_TERM_REQUIRED_MSG, _TERM_ORDER, _UNKNOWN_TERM_SORT_RANK. Column and schema remain cohort_term.
  • Response model: LatestInferenceCohortResponse.cohort_label → cohort_labels: list[str]; all construction sites updated. User-facing message for missing column says "cohort term" (matches column); other messages keep "checkpoint term."
  • Logic: Defensive fallback message in filter_to_most_recent_checkpoint_term; explicit fields on early "Institution not found" response. Extracted _try_checkpoint_term_candidate, _ordered_checkpoint_terms_from_cohort_column, and _ordered_checkpoint_terms_from_entry_year for single-purpose helpers and 50-line limit.

• Databricks (databricks.py)

  • _parse_config_toml_to_selection: accepts checkpoint_term in config (normalized to cohort_term by API); behavior unchanged for cohort_term.

• Tests

  • routers/data_test.py: Normalizer (valid, non-dict student_criteria); checkpoint_term in config; cohort_term as string in config; cohort_term empty list (valid, count 0); response contract (cohort_labels list).
  • databricks_test.py: test_parse_config_toml_to_selection_accepts_checkpoint_term; return type -> None on the other parse test.

context

• Run-inference term_filter: Callers need to be able to choose which cohorts (e.g. "fall 2024-25") to run inference on. This PR adds an optional term_filter list to the run-inference request and passes it through to the Databricks job so the pipeline can filter by those terms/cohorts. When term_filter is omitted, no term_filter key is sent to Databricks and behavior is unchanged (pipeline uses its config default).

• Latest-inference-cohort: The UI needs a single "latest inference-ready cohort" (label, valid student count, status) for an institution. This endpoint supports optional model_name and batch_name so the UI can show a specific model/batch or rely on defaults (single registered model, latest batch with student+course files). It reads the chosen model's training/preprocessing selection config from Databricks (silver volume, per model run) and returns the first cohort term that has course data and meets the config criteria. Refactors and tests align with project standards (e.g. function length, naming, error-path coverage).

• Checkpoint term and cohort_labels: Pipelines may specify the inference cohort via checkpoint_term in config; the API normalizes this to cohort_term at entry so the rest of the stack stays column-aligned. The response now exposes cohort_labels (list) for consistency and future multi-cohort use. Naming ("checkpoint term") is aligned with product language while keeping the DB column name cohort_term.

questions

• None.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1213317049256276
  • https://app.asana.com/0/0/1213087332160187

[kaylawilding]

Sorry what is checkpoint_term here? Is it meant to be the term config field we have added listing the terms for inference?
If so, a couple thoughts, it shouldn't be a term the same as cohort_term field is simply FALL or SPRING, but instead it also includes the year so it will be like FALL 2023-24, eg.

[kaylawilding]

We should talk about how to square this with that were using terms now, do we want both?

[kaylawilding]

ah i see you are doing both, i will need to change the config to terms i believe

[kaylawilding]

lets make sure this is not the student criteria field

[kaylawilding]

adding comments but this is on hold