TerraWatch: Deforestation Detection and Analysis Using Image Segmentation and Multimodal LLMs

Concept

TerraWatch is a proof of concept system developed during the TUM AI Hackathon 2024 to detect deforestation from satellite images and reason out the causes and potential environmental effects using computer vision models and multimodal large language models.

Below is the youtube demo:

Link to Full Video Demo on Youtube: Link

Architecture

Segmentation Model

We used a traditional UNet to train the data on annotated sattelite images from Ukraine. The details of the segmentation model training and results will be uploaded soon.

Frontend

The frontend is built using React with Elements UI. The necessary dependencies need to be installed using npm.

npm install

To start up the local frontend instance run the following command.

npm start

Running the LLM

1. Get your own API keys! Put them in .env

2. Create a virtual environment for example using virtualenv and activate it

python -m virtualenv tum-ai
source tum-ai/bin/activate

3. Install all necessary dependencies into the newly created environment using pip

pip install -r requirements.txt

4. Start the local Flask API server using uvicorn

python -m uvicorn api:app --reload

The API is now accessible through the following endpoints

GET /environmental-details/{latitude}/{longitude}
POST /deforestation
POST /segmentation

Future Work

1. Upgrade models: We would like to use ResUnet as it was observed that it achieves a faster training convergence desprite its high computational complexity. The residual connections present in its architecture facilitate information propagation and convergence speed. reference

2. Fine tune LLM models: The current LLM models have not been fine tuned and we would like to fine tune them based on available environmental datasets.