Skip to content

Commit b696289

Browse files
MSCetin37MSCetin37
committed
update compose.yaml
Signed-off-by: Mustafa <mustafa.cetin@intel.com> update the documents Signed-off-by: Mustafa <mustafa.cetin@intel.com> update the documents Signed-off-by: Mustafa <mustafa.cetin@intel.com> document update Signed-off-by: Mustafa <mustafa.cetin@intel.com> update the test file Signed-off-by: Mustafa <mustafa.cetin@intel.com> update the test file Signed-off-by: Mustafa <mustafa.cetin@intel.com> update README Signed-off-by: Mustafa <mustafa.cetin@intel.com> update README Signed-off-by: Mustafa <mustafa.cetin@intel.com>
1 parent 5c7b41c commit b696289

File tree

10 files changed

+509
-433
lines changed

10 files changed

+509
-433
lines changed

‎FinanceAgent/README.md

Lines changed: 35 additions & 147 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,30 @@
1-
# Finance Agent
1+
# Finance Agent Example
22

3-
## 1. Overview
3+
## Table of Contents
4+
5+
- [Overview](#overview)
6+
- [Problem Motivation](#problem-motivation)
7+
- [Architecture](#architecture)
8+
- [High-Level Diagram](#high-level-diagram)
9+
- [OPEA Microservices Diagram](#opea-microservices-diagram)
10+
- [Deployment Options](#deployment-options)
11+
- [Contribution](#contribution)
12+
13+
14+
15+
## Overview
16+
17+
The Finance Agent example showcases a hierarchical multi-agent system designed to assist users with financial document processing and analysis. It provides three main functionalities: summarizing lengthy financial documents, answering queries related to financial documents, and conducting research to generate investment reports on public companies.
18+
19+
Users interact with the system via a graphical user interface (UI), where requests are managed by a supervisor agent that delegates tasks to worker agents or the summarization microservice. The system supports document uploads through the UI for processing.
20+
21+
22+
## Problem Motivation
23+
Navigating and analyzing extensive financial documents can be challenging and time-consuming. Users often require concise summaries, answers to specific queries, or comprehensive investment reports. The Finance Agent addresses these needs by automating document summarization, query answering, and research tasks, thereby enhancing productivity and decision-making efficiency.
24+
25+
## Architecture
26+
### High-Level Diagram
27+
The Finance Agent system is structured as a hierarchical multi-agent architecture. User interactions are managed by a supervisor agent, which coordinates tasks among worker agents and the summarization microservice. The system supports document uploads and processing through the UI.
428

529
The architecture of this Finance Agent example is shown in the figure below. The agent is a hierarchical multi-agent system and has 3 main functions:
630

@@ -12,6 +36,7 @@ The user interacts with the supervisor agent through the graphical UI. The super
1236

1337
![Finance Agent Architecture](assets/finance_agent_arch.png)
1438

39+
### OPEA Microservices Diagram
1540
The architectural diagram of the `dataprep` microservice is shown below. We use [docling](https://github.com/docling-project/docling) to extract text from PDFs and URLs into markdown format. Both the full document content and tables are extracted. We then use an LLM to extract metadata from the document, including the company name, year, quarter, document type, and document title. The full document markdown then gets chunked, and LLM is used to summarize each chunk, and the summaries are embedded and saved to a vector database. Each table is also summarized by LLM and the summaries are embedded and saved to the vector database. The chunks and tables are also saved into a KV store. The pipeline is designed as such to improve retrieval accuracy of the `search_knowledge_base` tool used by the Question Answering worker agent.
1641

1742
![dataprep architecture](assets/fin_agent_dataprep.png)
@@ -30,154 +55,17 @@ The Question Answering worker agent uses `search_knowledge_base` tool to get rel
3055

3156
![finqa search tool arch](assets/finqa_tool.png)
3257

33-
## 2. Getting started
34-
35-
### 2.1 Download repos
36-
37-
```bash
38-
mkdir /path/to/your/workspace/
39-
export WORKDIR=/path/to/your/workspace/
40-
cd $WORKDIR
41-
git clone https://github.com/opea-project/GenAIExamples.git
42-
```
43-
44-
### 2.2 Set up env vars
45-
46-
```bash
47-
export ip_address="External_Public_IP"
48-
export no_proxy=${your_no_proxy},${ip_address}
49-
export HF_CACHE_DIR=/path/to/your/model/cache/
50-
export HF_TOKEN=<you-hf-token>
51-
export FINNHUB_API_KEY=<your-finnhub-api-key> # go to https://finnhub.io/ to get your free api key
52-
export FINANCIAL_DATASETS_API_KEY=<your-api-key> # go to https://docs.financialdatasets.ai/ to get your free api key
53-
```
54-
55-
### 2.3 [Optional] Build docker images
56-
57-
Only needed when docker pull failed.
58-
59-
```bash
60-
cd $WORKDIR/GenAIExamples/FinanceAgent/docker_image_build
61-
# get GenAIComps repo
62-
git clone https://github.com/opea-project/GenAIComps.git
63-
# build the images
64-
docker compose -f build.yaml build --no-cache
65-
```
66-
67-
If deploy on Gaudi, also need to build vllm image.
68-
69-
```bash
70-
cd $WORKDIR
71-
git clone https://github.com/HabanaAI/vllm-fork.git
72-
# get the latest release tag of vllm gaudi
73-
cd vllm-fork
74-
VLLM_VER=$(git describe --tags "$(git rev-list --tags --max-count=1)")
75-
echo "Check out vLLM tag ${VLLM_VER}"
76-
git checkout ${VLLM_VER}
77-
docker build --no-cache -f Dockerfile.hpu -t opea/vllm-gaudi:latest --shm-size=128g . --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy
78-
```
79-
80-
## 3. Deploy with docker compose
81-
82-
### 3.1 Launch vllm endpoint
83-
84-
Below is the command to launch a vllm endpoint on Gaudi that serves `meta-llama/Llama-3.3-70B-Instruct` model on 4 Gaudi cards.
85-
86-
```bash
87-
cd $WORKDIR/GenAIExamples/FinanceAgent/docker_compose/intel/hpu/gaudi
88-
bash launch_vllm.sh
89-
```
90-
91-
### 3.2 Prepare knowledge base
92-
93-
The commands below will upload some example files into the knowledge base. You can also upload files through UI.
94-
95-
First, launch the redis databases and the dataprep microservice.
96-
97-
```bash
98-
# inside $WORKDIR/GenAIExamples/FinanceAgent/docker_compose/intel/hpu/gaudi/
99-
bash launch_dataprep.sh
100-
```
101-
102-
Validate datat ingest data and retrieval from database:
103-
104-
```bash
105-
python $WORKDIR/GenAIExamples/FinanceAgent/tests/test_redis_finance.py --port 6007 --test_option ingest
106-
python $WORKDIR/GenAIExamples/FinanceAgent/tests/test_redis_finance.py --port 6007 --test_option get
107-
```
108-
109-
### 3.3 Launch the multi-agent system
110-
111-
The command below will launch 3 agent microservices, 1 docsum microservice, 1 UI microservice.
112-
113-
```bash
114-
# inside $WORKDIR/GenAIExamples/FinanceAgent/docker_compose/intel/hpu/gaudi/
115-
bash launch_agents.sh
116-
```
117-
118-
### 3.4 Validate agents
119-
120-
FinQA Agent:
121-
122-
```bash
123-
export agent_port="9095"
124-
prompt="What is Gap's revenue in 2024?"
125-
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --prompt "$prompt" --agent_role "worker" --ext_port $agent_port
126-
```
127-
128-
Research Agent:
129-
130-
```bash
131-
export agent_port="9096"
132-
prompt="generate NVDA financial research report"
133-
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --prompt "$prompt" --agent_role "worker" --ext_port $agent_port --tool_choice "get_current_date" --tool_choice "get_share_performance"
134-
```
135-
136-
Supervisor Agent single turns:
137-
138-
```bash
139-
export agent_port="9090"
140-
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --agent_role "supervisor" --ext_port $agent_port --stream
141-
```
142-
143-
Supervisor Agent multi turn:
144-
145-
```bash
146-
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --agent_role "supervisor" --ext_port $agent_port --multi-turn --stream
147-
148-
```
149-
150-
## How to interact with the agent system with UI
151-
152-
The UI microservice is launched in the previous step with the other microservices.
153-
To see the UI, open a web browser to `http://${ip_address}:5175` to access the UI. Note the `ip_address` here is the host IP of the UI microservice.
154-
155-
1. Create Admin Account with a random value
156-
157-
2. Enter the endpoints in the `Connections` settings
158-
159-
First, click on the user icon in the upper right corner to open `Settings`. Click on `Admin Settings`. Click on `Connections`.
160-
161-
Then, enter the supervisor agent endpoint in the `OpenAI API` section: `http://${ip_address}:9090/v1`. Enter the API key as "empty". Add an arbitrary model id in `Model IDs`, for example, "opea_agent". The `ip_address` here should be the host ip of the agent microservice.
162-
163-
Then, enter the dataprep endpoint in the `Icloud File API` section. You first need to enable `Icloud File API` by clicking on the button on the right to turn it into green and then enter the endpoint url, for example, `http://${ip_address}:6007/v1`. The `ip_address` here should be the host ip of the dataprep microservice.
164-
165-
You should see screen like the screenshot below when the settings are done.
166-
167-
![opea-agent-setting](assets/ui_connections_settings.png)
168-
169-
3. Upload documents with UI
170-
171-
Click on the `Workplace` icon in the top left corner. Click `Knowledge`. Click on the "+" sign to the right of `Icloud Knowledge`. You can paste an url in the left hand side of the pop-up window, or upload a local file by click on the cloud icon on the right hand side of the pop-up window. Then click on the `Upload Confirm` button. Wait till the processing is done and the pop-up window will be closed on its own when the data ingestion is done. See the screenshot below.
17258

173-
Note: the data ingestion may take a few minutes depending on the length of the document. Please wait patiently and do not close the pop-up window.
59+
## Deployment Options
60+
This CodeGen example can be deployed manually on various hardware platforms using Docker Compose or Kubernetes. Select the appropriate guide based on your target environment:
17461

175-
![upload-doc-ui](assets/upload_doc_ui.png)
62+
| Hardware | Deployment Mode | Guide Link |
63+
| :-------------- | :------------------- | :----------------------------------------------------------------------- |
64+
| Intel Gaudi HPU | Single Node (Docker) | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) |
17665

177-
4. Test agent with UI
66+
_Note: Building custom microservice images can be done using the resources in [GenAIComps](https://github.com/opea-project/GenAIComps)._
17867

179-
After the settings are done and documents are ingested, you can start to ask questions to the agent. Click on the `New Chat` icon in the top left corner, and type in your questions in the text box in the middle of the UI.
18068

181-
The UI will stream the agent's response tokens. You need to expand the `Thinking` tab to see the agent's reasoning process. After the agent made tool calls, you would also see the tool output after the tool returns output to the agent. Note: it may take a while to get the tool output back if the tool execution takes time.
69+
## Contribution
70+
We welcome contributions to the OPEA project. Please refer to the contribution guidelines for more information.
18271

183-
![opea-agent-test](assets/opea-agent-test.png)

‎FinanceAgent/docker_compose/intel/hpu/gaudi/README.md

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Deploy Finance Agent on Intel Gaudi HPU with Docker Compose
2+
This README provides instructions for deploying the Finance Agent application using Docker Compose on systems equipped with Intel Gaudi HPUs.
3+
4+
## Table of Contents
5+
6+
- [Overview](#overview)
7+
- [Prerequisites](#prerequisites)
8+
- [Start Deployment](#start-deployment)
9+
- [Validate Services](#validate-services)
10+
- [Accessing the User Interface (UI)](#accessing-the-user-interface-ui)
11+
12+
## Overview
13+
14+
This guide focuses on running the pre-configured Finance Agent service using Docker Compose on Intel Gaudi HPUs. It leverages containers optimized for Gaudi for the LLM serving component, along with CPU-based containers for other microservices like embedding, retrieval, data preparation and the UI.
15+
16+
## Prerequisites
17+
- Docker and Docker Compose installed.
18+
- Intel Gaudi HPU(s) with the necessary drivers and software stack installed on the host system. (Refer to Intel Gaudi Documentation).
19+
- Git installed (for cloning repository).
20+
- Hugging Face Hub API Token (for downloading models).
21+
- Access to the internet (or a private model cache).
22+
23+
Clone the GenAIExamples repository:
24+
25+
```shell
26+
mkdir /path/to/your/workspace/
27+
export WORKDIR=/path/to/your/workspace/
28+
cd $WORKDIR
29+
git clone https://github.com/opea-project/GenAIExamples.git
30+
cd GenAIExamples/FinanceAgent/docker_compose/intel/hpu/gaudi
31+
```
32+
33+
## Start Deployment
34+
This uses the default vLLM-based deployment profile (vllm-gaudi-server).
35+
### Configure Environment
36+
Set required environment variables in your shell:
37+
38+
```shell
39+
# Replace with your Hugging Face Hub API token
40+
export HF_TOKEN="your_huggingface_token"
41+
# Path to your model cache
42+
export HF_CACHE_DIR="./data"
43+
# Go to https://finnhub.io/ to get your free api key
44+
export FINNHUB_API_KEY=<your-finnhub-api-key>
45+
# Go to https://docs.financialdatasets.ai/ to get your free api key
46+
export FINANCIAL_DATASETS_API_KEY=<your-api-key>
47+
48+
# Optional: Configure HOST_IP if needed
49+
# Replace with your host's external IP address (do not use localhost or 127.0.0.1).
50+
# export HOST_IP=$(hostname -I | awk '{print $1}')
51+
# Optional: Configure proxy if needed
52+
# export http_proxy="your_http_proxy"
53+
# export https_proxy="your_https_proxy"
54+
# export no_proxy="localhost,127.0.0.1,${HOST_IP}" # Add other hosts if necessary
55+
56+
source ../../set_env.sh
57+
```
58+
59+
Note: The compose file might read additional variables from set_env.sh. Ensure all required variables like ports (LLM_SERVICE_PORT, TEI_EMBEDDER_PORT, etc.) are set if not using defaults from the compose file. For instance, edit the set_env.sh to change the LLM model:
60+
61+
### Start Services
62+
#### Deploy with Docker Compose
63+
Below is the command to launch services
64+
- vllm-gaudi-server
65+
- tei-embedding-serving
66+
- redis-vector-db
67+
- redis-kv-store
68+
- dataprep-redis-server-finance
69+
- finqa-agent-endpoint
70+
- research-agent-endpoint
71+
- docsum-vllm-gaudi
72+
- supervisor-agent-endpoint
73+
- agent-ui
74+
75+
76+
```shell
77+
docker compose up -d
78+
```
79+
80+
#### [Optional] Build docker images
81+
82+
Only needed when docker pull failed.
83+
84+
```bash
85+
cd $WORKDIR/GenAIExamples/FinanceAgent/docker_image_build
86+
# get GenAIComps repo
87+
git clone https://github.com/opea-project/GenAIComps.git
88+
# build the images
89+
docker compose -f build.yaml build --no-cache
90+
```
91+
92+
If deploy on Gaudi, also need to build vllm image.
93+
94+
```bash
95+
cd $WORKDIR
96+
git clone https://github.com/HabanaAI/vllm-fork.git
97+
# get the latest release tag of vllm gaudi
98+
cd vllm-fork
99+
VLLM_VER=$(git describe --tags "$(git rev-list --tags --max-count=1)")
100+
echo "Check out vLLM tag ${VLLM_VER}"
101+
git checkout ${VLLM_VER}
102+
docker build --no-cache -f Dockerfile.hpu -t opea/vllm-gaudi:latest --shm-size=128g . --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy
103+
```
104+
105+
106+
## Validate Services
107+
Wait several minutes for models to download and services to initialize (Gaudi initialization can take time). Check container logs (docker compose logs -f <service_name>, especially vllm-gaudi-server).
108+
```bash
109+
docker logs --tail 2000 -f vllm-gaudi-server
110+
```
111+
112+
### Validate Data Services
113+
Ingest data and retrieval from database
114+
115+
```bash
116+
python $WORKDIR/GenAIExamples/FinanceAgent/tests/test_redis_finance.py --port 6007 --test_option ingest
117+
python $WORKDIR/GenAIExamples/FinanceAgent/tests/test_redis_finance.py --port 6007 --test_option get
118+
```
119+
120+
### Validate Agents
121+
122+
FinQA Agent:
123+
124+
```bash
125+
export agent_port="9095"
126+
prompt="What is Gap's revenue in 2024?"
127+
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --prompt "$prompt" --agent_role "worker" --ext_port $agent_port
128+
```
129+
130+
Research Agent:
131+
132+
```bash
133+
export agent_port="9096"
134+
prompt="generate NVDA financial research report"
135+
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --prompt "$prompt" --agent_role "worker" --ext_port $agent_port --tool_choice "get_current_date" --tool_choice "get_share_performance"
136+
```
137+
138+
Supervisor Agent single turns:
139+
140+
```bash
141+
export agent_port="9090"
142+
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --agent_role "supervisor" --ext_port $agent_port --stream
143+
```
144+
145+
Supervisor Agent multi turn:
146+
147+
```bash
148+
python3 $WORKDIR/GenAIExamples/FinanceAgent/tests/test.py --agent_role "supervisor" --ext_port $agent_port --multi-turn --stream
149+
```
150+
151+
## Accessing the User Interface (UI)
152+
153+
The UI microservice is launched in the previous step with the other microservices.
154+
To see the UI, open a web browser to `http://${ip_address}:5175` to access the UI. Note the `ip_address` here is the host IP of the UI microservice.
155+
156+
1. Create Admin Account with a random value
157+
158+
2. Enter the endpoints in the `Connections` settings
159+
160+
First, click on the user icon in the upper right corner to open `Settings`. Click on `Admin Settings`. Click on `Connections`.
161+
162+
Then, enter the supervisor agent endpoint in the `OpenAI API` section: `http://${ip_address}:9090/v1`. Enter the API key as "empty". Add an arbitrary model id in `Model IDs`, for example, "opea_agent". The `ip_address` here should be the host ip of the agent microservice.
163+
164+
Then, enter the dataprep endpoint in the `Icloud File API` section. You first need to enable `Icloud File API` by clicking on the button on the right to turn it into green and then enter the endpoint url, for example, `http://${ip_address}:6007/v1`. The `ip_address` here should be the host ip of the dataprep microservice.
165+
166+
You should see screen like the screenshot below when the settings are done.
167+
168+
![opea-agent-setting](../../../../assets/ui_connections_settings.png)
169+
170+
3. Upload documents with UI
171+
172+
Click on the `Workplace` icon in the top left corner. Click `Knowledge`. Click on the "+" sign to the right of `Icloud Knowledge`. You can paste an url in the left hand side of the pop-up window, or upload a local file by click on the cloud icon on the right hand side of the pop-up window. Then click on the `Upload Confirm` button. Wait till the processing is done and the pop-up window will be closed on its own when the data ingestion is done. See the screenshot below.
173+
174+
Note: the data ingestion may take a few minutes depending on the length of the document. Please wait patiently and do not close the pop-up window.
175+
176+
![upload-doc-ui](../../../../assets/upload_doc_ui.png)
177+
178+
4. Test agent with UI
179+
180+
After the settings are done and documents are ingested, you can start to ask questions to the agent. Click on the `New Chat` icon in the top left corner, and type in your questions in the text box in the middle of the UI.
181+
182+
The UI will stream the agent's response tokens. You need to expand the `Thinking` tab to see the agent's reasoning process. After the agent made tool calls, you would also see the tool output after the tool returns output to the agent. Note: it may take a while to get the tool output back if the tool execution takes time.
183+
184+
![opea-agent-test](../../../../assets/opea-agent-test.png)

0 commit comments

Comments
 (0)