🆕 Simplified SAM segmentation in TIAViz

[measty]

This PR takes the sam-viz PR and fixes and substantially simplifies it.

I think the original PR was going for a more general prompt segmentor engine that would in theory cope with prompts dispersed over a whole WSI, but I think that approach has issues in practice (and it would also need some significant work to fix from its current state, I think). This PR focusses on segmenting prompts made within a specific view window, which I think is more practical and allows for a much simplified approach.

To use, draw a few points (and/or a box) on some object(s) in the field of view in TIAViz using this widget:

Then select SAM in the model select box and click run. The segmetations should be displayed after a short time.

Examples:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.38%. Comparing base (5f4b454) to head (44e99d1).

Additional details and impacted files

@@            Coverage Diff            @@
##           develop     #968    +/-   ##
=========================================
  Coverage    99.37%   99.38%            
=========================================
  Files           71       73     +2     
  Lines         9175     9312   +137     
  Branches      1197     1206     +9     
=========================================
+ Hits          9118     9255   +137     
  Misses          31       31            
  Partials        26       26            

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[measty]

PR is now ready for review

[shaneahmed]

Suggested change

	python: python3.12
	python: python3.10

Can we set this to the base supported version i.e., 3.10.

[Copilot]

Pull request overview

This PR adds simplified SAM (Segment Anything Model) integration to TIAViz, enabling interactive segmentation using point and bounding box prompts within a specific view window. The implementation includes a new SAM architecture wrapper, a PromptSegmentor engine, and UI integration for interactive use.

Changes:

• Adds SAM model architecture and PromptSegmentor engine for interactive segmentation with point/box prompts
• Integrates SAM into TIAViz UI with callback for running segmentation on selected regions
• Extends utility functions to support offset parameter for annotation coordinate transformations

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tiatoolbox/models/architecture/sam.py	New SAM model architecture using transformers library for zero-shot segmentation
tiatoolbox/models/engine/prompt_segmentor.py	New PromptSegmentor engine that handles SAM inference and output conversion
tiatoolbox/visualization/bokeh_app/main.py	Adds sam_segment callback and SAM model option to UI
tiatoolbox/utils/misc.py	Extends process_contours and dict_to_store_semantic_segmentor to support offset parameter
tiatoolbox/visualization/tileserver.py	Filters out temporary bokeh databases from commit operations
tiatoolbox/models/init.py	Exports SAM and PromptSegmentor classes
requirements/requirements.txt	Adds transformers>=4.51.1 dependency
.pre-commit-config.yaml	Sets default Python version to 3.12
tests/models/test_prompt_segmentor.py	Unit tests for PromptSegmentor functionality
tests/models/test_arch_sam.py	Unit tests for SAM architecture
tests/test_app_bokeh.py	Integration test for SAM in TIAViz UI

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

When both point_coords and box_coords are provided for the same image, this code will call _process_prompts twice and append results to masks and scores lists separately. This means if a user provides both points and boxes, the concatenation at line 164 will have double the expected number of masks (one set from boxes, one set from points), rather than combining them as joint prompts. Consider either: 1) combining prompts in a single _process_prompts call, or 2) documenting that only one prompt type should be provided at a time, or 3) adding logic to handle this case appropriately.

[Copilot]

The save_dir directory is not created before use. If the directory doesn't exist, dict_to_store_semantic_segmentor will fail when trying to save the database file. Add save_dir.mkdir(parents=True, exist_ok=True) after line 82 to ensure the directory exists before attempting to save files to it.

[Copilot]

The offset attribute is initialized as a tuple (0, 0) but is later assigned as a numpy array in main.py line 1294 (offset = np.array([x_start, y_start])). This type inconsistency could cause issues. The offset is expected to be used in arithmetic operations with numpy arrays in process_contours. Consider initializing offset as np.array([0, 0]) for type consistency, or document that it can be either a tuple or numpy array.

Suggested change

	self.offset = (0, 0)
	self.offset = np.array([0, 0])

[Copilot]

If neither point_coords nor box_coords is provided for an image (both are None), the masks and scores lists will remain empty for that image, leading to an empty concatenation at line 164 which will likely fail. While infer_batch has validation to check that at least one is provided globally, this doesn't ensure that each individual image in the batch has at least one prompt. Consider adding validation within the loop to ensure each image has at least one prompt type, or handle the case where both are None for a specific image.

Suggested change

	if not masks or not scores:
	raise ValueError(
	"No segmentation masks were generated. Ensure that each image in "
	"`imgs` has at least one corresponding prompt in `point_coords` "
	"or `box_coords`."
	)

[Copilot]

The return type annotation indicates this method returns a float, but it actually returns a tuple of (float, float) containing both the calculated mpp and the scale factor. Update the return type annotation to tuple[float, float] to match the actual return value.

[Copilot]

The variable name 'height' is reused at line 1330, shadowing the height variable defined at line 1319. This causes the box height calculation to always use the height from index 0 (UI["box_source"].data["height"][0]) instead of the correct index i. This will produce incorrect box coordinates when multiple boxes are present. Change line 1330 to use index i instead of 0: height = [round(UI["box_source"].data["height"][i]) for i in range(len(x))]

Suggested change

	height = [round(UI["box_source"].data["height"][0]) for i in range(len(x))]
	height = [round(UI["box_source"].data["height"][i]) for i in range(len(x))]

[Copilot]

The scale_factor parameter should be a tuple of floats (tuple[float, float]) as expected by dict_to_store_semantic_segmentor and process_contours functions, but self.scale is a single float value. This will cause the multiplication in process_contours at line 1236 (scaled_coords = np.array([np.array(scale_factor) * coords])) to fail with a shape mismatch error. Consider changing this to scale_factor=(self.scale, self.scale) to provide the expected 2D scale factor.

Suggested change

	scale_factor=self.scale,
	scale_factor=(self.scale, self.scale),

[Copilot]

The docstring incorrectly states the return type as "float" when the method actually returns a tuple. Update the Returns section to document both return values: the calculated mpp (float) and the scale factor (float), or change it to "tuple[float, float]".

Suggested change

	float:
	Microns per pixel required to scale the area to a fixed size.
	tuple[float, float]:
	A tuple containing:
	- Microns per pixel required to scale the area to a fixed size.
	- The scale factor used for scaling.

[Copilot]

The temporary directory created with tempfile.mkdtemp is never cleaned up after use. Unlike the segment_on_box function which calls rmtree to clean up temporary directories, this function leaves the temporary directory and its contents on disk. Add rmtree(tmp_save_dir) after the annotations have been processed to prevent accumulation of temporary files.

Suggested change

	update_ui_on_new_annotations(ann_types)
	update_ui_on_new_annotations(ann_types)
	rmtree(tmp_save_dir)

[Copilot]

This assignment to 'patch' is unnecessary as it is redefined before this value is used.

Suggested change

patch = img[63:191, 750:878, :]

[shaneahmed]

It's strange that you are getting FLK-E402 error on this.

[shaneahmed]

height and width is being overwritten on 1329 and 1330

[shaneahmed]

Please move this file to tests/engines

[shaneahmed]

Suggested change

	"""Torch method. Defines forward pass on each image in the batch.
	"""PyTorch method. Defines forward pass on each image in the batch.

[shaneahmed]

Do we need any of these inputs to match with the new EngineABC signature?

def __init__(
    self: EngineABC,
    model: str | ModelABC,
    batch_size: int = 8,
    num_workers: int = 0,
    weights: str | Path | None = None,
    *,
    device: str = "cpu",
    verbose: bool = False,

[shaneahmed]

Is there an output type option if a user wants to save the results?

[shaneahmed]

Suggested change

	"""
	"""

[shaneahmed]

Thanks @measty

This looks good. A few minor comments if you could address before we merge.

• I was not able to test it though on my workstation using --base-path argument.
• Also it would be good to show any progress bar or message on the visualization rather than only in the terminal if possible.
• Can we reset the model weights after the first run if possible?