Skip to content

Commit 9c75dd3

Browse files
gooishingooishin
authored
Merge pull request #266 from intel/update-branch
724 user story initial check-in (#726)
2 parents 0ef78e7 + 0746585 commit 9c75dd3

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+4835
-0
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ edge-developer-kit-reference-scripts/
8282
5. [Digital Avatar](usecases/ai/digital-avatar/README.md)
8383
6. [Time Coordinated Computing (TCC)](usecases/real-time/tcc_tutorial/README.md)
8484
7. [Smart Parking](usecases/ai/smart-parking/README.md)
85+
8. [Video Summarization & Visual RAG](usecases/ai/video_summarization)
8586

8687
## Troubleshooting
8788

‎usecases/ai/video_summarization/.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*/__pycache__/*
2+
*/save_quantized_model/*
3+
*/MiniCPM*/
4+
*/backup/
5+
*/cache/
6+
*.pyc
7+
*.env

‎usecases/ai/video_summarization/README.md

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
## Introduction
2+
3+
This is a demo application that uses power of AI (Large Video Language model) for live video stream captioning for retail loss prevention scenario and Visual RAG (text-to-image retrieval) using Intel VDMS vector database.
4+
5+
![img](./resource/snapshot.gif)
6+
7+
## Pre-requisite:
8+
9+
- Ubuntu 24.04 LTS
10+
- Docker (https://docs.docker.com/engine/install/ubuntu/)
11+
- Miniforge Conda (https://conda-forge.org/download/)
12+
- Intel Openvino 2025.2.0 (https://docs.openvino.ai/2025/get-started/install-openvino.html)
13+
- Python 3.10+
14+
- tmux
15+
- Target Hardware:
16+
- Intel® Core™ Processors Platform (Alderlake, Raptor Lake, Arrow Lake, etc)
17+
- Intel® Arc™ A-series or Intel® Arc™ B-series graphics card (A770/B580)
18+
- Intel® Core™ Ultra Series Processor Platform (Lunar Lake, Arrow Lake-H)
19+
- Additional installation guide, see [link](https://github.com/intel/edge-developer-kit-reference-scripts?tab=readme-ov-file#quick-start).
20+
21+
## Environment Setup
22+
23+
1. Refer to the pre-requisite section, and follow the instructions from the website to install docker, Miniforge Conda.
24+
2. Install other ubuntu dependencies
25+
26+
```
27+
sudo apt update
28+
sudo apt install tmux
29+
```
30+
31+
3. Pull the code
32+
33+
```
34+
mkdir -p $HOME/work
35+
cd $HOME/work
36+
git clone https://github.com/intel/edge-developer-kit-reference-scripts edge-ai-devkit
37+
cp -rf edge-ai-devkit/usecases/ai/video_summarization ./video_summarization
38+
cd $HOME/work/video_summarization
39+
```
40+
41+
4. Create conda environment and install conda packages.
42+
43+
```
44+
conda create -n openvino-env python=3.11
45+
conda activate openvino-env
46+
conda update --all
47+
conda install -c conda-forge openvino=2025.2.0
48+
pip install -r requirements.txt
49+
```
50+
51+
52+
53+
## Model Preparation
54+
55+
Convert and Quantized MiniCPM-v-2_6 to OpenVINO IR format (INT8)
56+
57+
```
58+
mkdir -p $HOME/work
59+
cd $HOME/work
60+
optimum-cli export openvino -m openbmb/MiniCPM-V-2_6 --trust-remote-code --weight-format int8 MiniCPM_INT8
61+
```
62+
63+
64+
65+
## Starting Demo App
66+
67+
This demo app is comprising of the following modules/components:
68+
69+
- VLM API service (port 8000)
70+
71+
- Retriver API service (port 8001)
72+
73+
- Live Summarizer UI (port 8888)
74+
75+
- Video RAG UI (port 9999)
76+
77+
- Simple-RTSP-server (port 8554)
78+
79+
- VDMS Vector DB (port 55555)
80+
81+
82+
83+
There are 2 methods to run the demo, manual or via Docker. It is recommended to follow docker way of starting the Demo App for simplicity.
84+
85+
86+
87+
## Manually Starting Service
88+
89+
1. Activate conda environment and change directory to the project folder
90+
91+
```
92+
conda activate openvino-env
93+
cd $HOME/work/video_summarization
94+
```
95+
96+
2. Start API server
97+
98+
```
99+
uvicorn api.app:app --port 8000
100+
uvicorn api.retriever_api:app --port 8001
101+
```
102+
103+
3. Start Intel VDMS vector DB
104+
105+
```
106+
docker run --rm -d --name vdms-rag -p 55555:55555 intellabs/vdms:latest
107+
```
108+
109+
4. Start Live Summarizer:
110+
111+
``` streamlit
112+
streamlit run app_ov_test.py --server.port 8888 --server.address 127.0.0.1
113+
```
114+
115+
5. Start Video RAG UI:
116+
117+
```
118+
streamlit run app_rag.py --server.port 9999 --server.address 127.0.0.1
119+
```
120+
121+
6. Create chunks folder
122+
123+
```
124+
mkdir ./chunks
125+
```
126+
127+
7. Start virtual camera stream. You may use the utility script below to do that.
128+
129+
```
130+
./start_virtual_rtsp_cam0.sh /path/to/video.mp4 #replace /path/to/video.mp4 with the absolute path to your own video file
131+
```
132+
133+
> Note: The utility script only create one video stream, please copy and edit the script so more stream can be created. Make sure the video is posted to the URL as shown in the table below:
134+
>
135+
> | Camera Name | URL |
136+
> | ----------- | --------------------------- |
137+
> | CAM0 | http://localhost:8554/live |
138+
> | CAM1 | http://localhost:8554/live1 |
139+
> | CAM2 | http://localhost:8554/live2 |
140+
141+
> Utility Script:
142+
>
143+
> - clear_database.sh - use this script to clear the VDMS vector store
144+
>
145+
> - start_virtual_rtsp_cam0.sh - use this script to create a virtual camera stream. Expected input video format. Resolution: 1920x1080, Framerate: 15fps.
146+
>
147+
148+
149+
150+
## Start using docker
151+
152+
1. Name of the docker containers and its associated ports. Please make sure the ports are not used by any other locally hosted services.
153+
154+
| Container Name | Exposed Ports |
155+
| --------------- | ------------- |
156+
| vlm_api_service | 8000 |
157+
| rag_api_service | 8001 |
158+
| vector_store | 55555 |
159+
| rtmp_server | 8554 |
160+
| summarizer_ui | 8888 |
161+
| retriever_ui | 9999 |
162+
163+
164+
165+
2. create chunks folder (folder to hold the video chunks file), and change permission of the chunks folder.
166+
167+
```
168+
mkdir -p ../chunks
169+
chmod 777 ../chunks
170+
```
171+
172+
3. Launch linux terminal from desktop and cd to project directory:
173+
174+
```
175+
cd $HOME/work/video_summarization
176+
```
177+
178+
4. Run the command below:
179+
180+
```
181+
docker compose build
182+
docker compose up -d
183+
```
184+
185+
5. Start virtual camera stream. You may use the utility script below to do that.
186+
187+
```
188+
./start_virtual_rtsp_cam0.sh /path/to/video.mp4 #replace /path/to/video.mp4 with the absolute path to your own video file
189+
```
190+
191+
> Note: The utility script only create one video stream, please copy and edit the script so more stream can be created. Make sure the video is posted to the URL as shown in the table below:
192+
>
193+
> | Camera Name | URL |
194+
> | ----------- | --------------------------- |
195+
> | CAM0 | http://localhost:8554/live |
196+
> | CAM1 | http://localhost:8554/live1 |
197+
> | CAM2 | http://localhost:8554/live2 |
198+
199+
> Utility Script:
200+
>
201+
> - start_virtual_rtsp_cam0.sh - use this script to create a virtual camera stream
202+
203+
6. To stop the demo:
204+
205+
```
206+
docker compose down
207+
```
208+
209+
7. Other useful command
210+
211+
```
212+
tmux ls # use this command to check if virtual camera stream is running
213+
docker compose ls # use this command to check if all services of the demo is running
214+
docker compose top # use this command to check the container name
215+
docker compose logs [container_name] # use this command to retrieve the runtime logs of a specific container
216+
```
217+
218+
8. The docker-compose.yml uses environment variables to pass additional configuration parameters to the containers. Change this from
219+
220+
| Environment Variables | Containers | Default Value | Sample Value |
221+
| ----------------------------------------------------- | ------------------------------------------------------------ | ------------- | ---------------------------- |
222+
| **Proxy settings**: HTTP_PROXY, HTTPS_PROXY, NO_PROXY | vlm_api_service, rag_api_service, summarizer_ui, retriever_ui | None | http://proxy.domain.com:8080 |
223+
| * **AI backend:** DEVICE | vlm_api_service | GPU | GPU.1, GPU.0, GPU, CPU, NPU |
224+
225+
> ***Note:**
226+
>
227+
> 1. DEVICE=NPU is not supported yet
228+
> 2. You may also use this command to identify all AI accelerator supported in your HW. ```python -c "import openvino as ov; print(ov.Core().available_devices)"```
229+
230+
## Performance
231+
232+
1. Install qmassa. Follow instruction in https://github.com/ulissesf/qmassa.
233+
234+
1. Run qmassa in command line to view the GPU utilization.
235+
236+
```
237+
sudo $HOME/.cargo/bin/qmassa
238+
```
239+
240+
> Note: If it doesn't correctly show the gpu utilization for your intel GPU device, pass the parameter "-d bus:device:func" to qmassa. You may look up for the BDF (bus:device:func) of your Intel GPU card using the command 'lspci'.

‎usecases/ai/video_summarization/alert.png

15.4 KB
Loading

‎usecases/ai/video_summarization/api/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)