ERE-CLIP

Figure 1. Overall architecture of ERE-CLIP.

ERE-CLIP is a vision–language model that learns semantic representations of visual entities based on subject–relation–object structures.
This repository provides the official implementation and instructions for reproducing the experiments reported in our paper.

---

Installation

This section describes how to set up the software environment and datasets required to reproduce our experiments on Windows.

Prerequisites

Git

Install Git for Windows from the official website:
https://git-scm.com/downloads/win

pyenv (Windows)

Install pyenv-win and configure the required environment variables by running the following commands in PowerShell:

git clone https://github.com/pyenv-win/pyenv-win.git "$HOME\.pyenv"

[System.Environment]::SetEnvironmentVariable("PYENV", $env:USERPROFILE + "\.pyenv\pyenv-win", "User")
[System.Environment]::SetEnvironmentVariable("PYENV_HOME", $env:USERPROFILE + "\.pyenv\pyenv-win", "User")
[System.Environment]::SetEnvironmentVariable(
  "PATH",
  $env:USERPROFILE + "\.pyenv\pyenv-win\bin;" +
  $env:USERPROFILE + "\.pyenv\pyenv-win\shims;" +
  [System.Environment]::GetEnvironmentVariable("PATH", "User"),
  "User"
)

# Restart PowerShell before executing the following command.
pyenv --version

Python

Install a specific Python version using pyenv:

pyenv install --list
pyenv install 3.13.5

pyenv global 3.13.5
pyenv rehash

pyenv versions

Python Packages

Create and activate a virtual environment, then install all required dependencies:

python -m venv ere_clip
ere_clip\Scripts\activate

python -m pip install --upgrade pip setuptools wheel

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

pip install numpy pillow opencv-python matplotlib psutil pandas schedulefree seaborn ujson wmi nvidia-ml-py3 natsort
pip install einops timm
pip install git+https://github.com/openai/CLIP.git
pip install open_clip_torch

deactivate

---

Datasets

ASER Dataset

1. Download the split archive files from the official source:
   KG.db.tar.0 through KG.db.tar.8
   https://hkustconnect-my.sharepoint.com/:f:/g/personal/xliucr_connect_ust_hk/Erraz2_KGjFHtbP9bh2-HMoBjCKGYX887MMzLX2y7xbs0w

2. Concatenate the split files into a single archive:

copy /b KG.db.tar.0+KG.db.tar.1+KG.db.tar.2+KG.db.tar.3+KG.db.tar.4+KG.db.tar.5+KG.db.tar.6+KG.db.tar.7+KG.db.tar.8 KG.db.tar

3. Extract KG.db.tar to obtain KG.db.

LVIS Dataset

Download the annotation files from the official LVIS website and extract them:

• lvis_v1_train.json.zip
• lvis_v1_val.json.zip
  https://www.lvisdataset.org/dataset

Download the COCO 2017 image sets and extract them:

• train2017.zip
• val2017.zip
• test2017.zip
  https://cocodataset.org/#download

---

Project Root Configuration

The project root and lightweight run snapshots directories are environment-dependent and may point to different locations.
Please modify the following fields in EnvConfig to match your local environment before running the code:

• EnvConfig.project_root → full project workspace (datasets, runs)
• EnvConfig.run_snapshots_root → lightweight copies of run outputs for remote viewing

Example:

self.project_root = Path("C:/path/to/your/project_root")
self.run_snapshots_root = Path("D:/path/to/your/run_snapshots_root")

Expected Directory Structure

Under project_root:

project_root/
├─ datasets/
│  ├─ aser/
│  │  └─ KG.db
│  └─ lvis/
│     ├─ train2017/
│     ├─ val2017/
│     ├─ test2017/
│     ├─ lvis_v1_train.json
│     └─ lvis_v1_val.json
└─ runs/
   ├─ exp_001/
   ├─ exp_002/
   └─ ...

Under run_snapshots_root:

run_snapshots_root/
├─ exp_001/
├─ exp_002/
└─ ...

---

Usage

The following commands outline the typical workflow for preprocessing data, training the model, building the final checkpoint, and running inference with ERE-CLIP.

1. Preprocess ASER Dataset

Extract (subject, relation, object) triplets from raw ASER data for Stage 1 training.

python scripts/preprocess/preprocess_aser.py

2. Stage 1: Conditional Text Representation Learning

Train ERE-CLIP to learn text embeddings from ASER triplets using contrastive learning conditioned on the relation.

python scripts/train/train_stage1.py

3. Stage 2: Region-Level Image–Text Contrastive Learning

Align image regions with text using LVIS bounding boxes and class names.

python scripts/train/train_stage2.py

4. Build Final Checkpoint

Combine the weights from Stage 1 and Stage 2 into the final ERE-CLIP model.

python scripts/train/build_final_checkpoint.py

5. Inference / Detection

Detect entities in images using ERE-CLIP scores.
No classification is performed; regions above a threshold are considered detected.

python scripts/predict/detect.py