RS-TA: Remote Sensing Task Agents via LLM and VFMs

Introduction

This repository introduces an innovative remote sensing task agent powered by LLM and VFMs. It's designed to understand remote sensing image commons and efficiently solve complex tasks.

Updates

• [2024.07.05]:

  • Add Llama3-8b-Instruct / Qwen2 LLM support, supporting English.
  • Update the LLM model to Llama3-8b-Chinese, base ,supporting EN/CH.

• [2024.07.15]:

  • Update the image captioning model to RS_BLIP, model has been finetuned on the rsicd dataset.
  • Add PAN image object detection and counting support. {--img_ch} 1 for PAN/Gray, 3 for MS/RGB

• [2024.07.17]:

  • Update Landuse Segmentation (Unet) support.
  • Update Object Detection (YOLOv5s) support.

• [2024.08.27]:

  • Suported Gradio interface.
  • Release the first demo of RS-Chat.

Requirements

pip install -r requirements.txt

Run Gradio app

python rschat_app_demo.py

Supported Model

Model	Hugging Face	github
Llama-3-8B-Instruct	meta-llama/Meta-Llama-3-8B-Instruct	llama3

Supported Function

Function	Description	Method	Pretrain Dataset	Model Weights
Image Captioning	Describe the remote sensing image	BLIP	BLIP Dataset	weight(github)
Scene Classification	Classify the type of scene	ResNet	AID Dataset	weight(Google) weight(Baidu)
Object Detection	Detect RS object from image	YOLO v5	DOTA	weight(Google) weight(Baidu)
Instance Segmentation	Extract Instance Mask of certain object	SwinTransformer+UperNet	iSAID	weight(Google) weight(Baidu)
Landuse Classification	Extract Pixel-wise Landuse Classification	HRNet	LoveDA	weight(Google) weight(Baidu)
Object Counting	Count the number of certain object in an image	YOLO v5	DOTA	Same as Object Detection
Edge Detection	Extract edge of remote sensing image	Canny	None	None

More funtions to be updated~

Citation

Please cite the repo if you use the data or code in this repo.

@article{RS ChatGPT,
	title = {Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models},
	shorttitle = {Remote Sensing ChatGPT},
	doi = {10.48550/ARXIV.2401.09083},
	author = {Guo, Haonan and Su, Xin and Wu, Chen and Du, Bo and Zhang, Liangpei and Li, Deren},
	year = {2024},
}

Acknowledgments

• Visual ChatGPT
• YOLOv5
• BLIP