SAM 3D Objects for Apple Silicon

Efficient 3D object reconstruction from images on macOS using Metal Performance Shaders (MPS). Gaussian splatting and color baking are not supported.

Original Image

Mask	3D Reconstruction

Using SAM 3D by Meta AI:

• Paper (arXiv)
• Official GitHub
• Model Weights (Hugging Face)

Installation

1. Clone and install dependencies:

   git clone <this-repo>
   cd Sam3D-MLX
   uv sync

2. Download checkpoints from Hugging Face and place them in checkpoints/hf/:

   mkdir -p checkpoints/hf
   # Download pipeline.yaml and all .pt/.safetensors files into checkpoints/hf/

3. Configure environment

   set -x PYTORCH_MPS_HIGH_WATERMARK_RATIO 0.0
   set -x SPARSE_BACKEND mps
   set -x SPARSE_ATTN_BACKEND sdpa
   set -x PYTHONPATH .

Usage

uv run python main.py \
    --image images/shutterstock_stylish_kidsroom_1640806567/image.png \
    --mask-dir images/shutterstock_stylish_kidsroom_1640806567 \
    --mask-index 0 \
    --mesh \
    --output outputs/reconstruction.glb

Key Arguments

Argument	Description
--image	Input image path
--mask-dir / --mask-index	SAM-style mask directory and index
--steps	Diffusion steps (default: 12)
--mesh	Output a full smooth GLB mesh
--output	File path for results (.glb, .stl)

Structure

checkpoints/hf/     # Model weights (download from HuggingFace)
images/             # Example dataset
outputs/            # 3D model results
.cache/             # Intermediate latent files
sam3d_objects/      # Core model logic

What Was Done

This port adapts the original CUDA-based SAM 3D Objects pipeline to run on Apple Silicon:

1. Removed CUDA dependencies: Replaced spconv-cu121, xformers, and other CUDA-specific packages.
2. MPS Backend: Rewired model loading and inference to use PyTorch's Metal Performance Shaders backend.
3. Metal Sparse Convolution: Custom Metal compute shaders for voxel processing:
   • sparse_conv.metal — GPU kernels
   • conv_metal.py — PyObjC wrapper
4. Metal Flash Attention: GPU-accelerated attention for sparse transformers:
   • flash_attn.metal — GPU kernels
   • metal_flash_attn.py — Python wrapper
5. Low-Memory Pipeline: Sequential model loading to fit within 48GB RAM.