Decoupling Specialization and Intent in Multi-Agent Placement for the Cloud Continuum

Project Description

We present a two-stage framework for service placement problem in the cloud continuum that explicitly decouples behavioral specialization from high-level operational intent. The placement problem involves complex trade-offs between operational cost, latency compliance, infrastructure utilization, and network stability across heterogeneous edge–cloud environments. Our objective is to bridge abstract intent specification and runtime placement decisions in a scalable and adaptive way. The proposed solution combines two complementary mechanisms: Proximal Policy Optimization (PPO) and the multi-criteria decision analysis method TOPSIS. PPO is used to train multiple specialized agents under distinct reward weight configurations, inducing differentiated behavioral biases such as cost-oriented, latency-focused, or utilization-aware policies. Once trained, these agents generate candidate placement actions in real time without additional optimization overhead. TOPSIS operates as an intent-conditioned coordination layer. Given the actions proposed by the specialized PPO agents, it evaluates their projected system-level impact using weighted criteria aligned with the operator’s declared intent. By ranking alternatives according to their proximity to an ideal and anti-ideal solutions, TOPSIS selects the placement that best satisfies current policy priorities. This decouples behavioral learning from intent realization, allowing dynamic policy shifts without retraining. Experimental results over heterogeneous multi-region infrastructures demonstrate that the specialized agents exhibit clearly differentiated optimization tendencies, while the intent layer enables flexible and consistent with global objectives particularly under high-load and saturation regimes

---

This project is under the MIT License. However, we would like to know in which projects or publications you have used or mentioned it.

Please consider using the following citation when you use our package:

    Pending

---

Installing Dependencies with UV

The project uses UV for dependency management. Ensure UV is installed, then from the repository root run:

uv sync

This installs the locked dependencies from pyproject.toml and uv.lock into the project environment. To use the environment in your shell:

uv run python experiment/run_experiment.py

To create a virtual environment and install in editable mode:

uv venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate
uv pip install -e .

---

Individual Experiment Configuration File

Standalone experiments (Random, Greedy, and PPO) are driven by **experiment/experiment_conf.json**. For the evaluation of PPO models, firsts is necessary to train the agent.

Each experiment object has:

Field	Description
name	Short name of the experiment.
description	Human-readable description.
parameters	Simulation and policy parameters (see below).
results	Where to write outputs: path (directory) and file (metrics JSON).

Main parameters

Parameter	Description
network	Topology factory, e.g. create_small_network, create_large_network.
policy	Placement policy: rndPlacement (random), greedyPlacement (greedy), or ppoPlacement (DRL with Stable-Baselines3 PPO).
study_type	Optional study tag. Use drl to enable DRL-specific evaluation plots.
mec_node_random_policy	Whether MEC node selection is random (boolean).
services_per_step	Number of service requests per simulation step.
num_steps	Number of steps per run.
metric_capture_interval	Interval (in steps) at which metrics are recorded.
policy_config	Optional policy execution config. For ppoPlacement, use inference fields such as inference_only and load_model_path (saved checkpoint).
queue_delay	Queue delay model: method (mm1 or alpha_rho_k), alpha_seconds, k, TQ_MAX_seconds.

For ppoPlacement execution in experiment_conf.json, use:

• inference_only (bool): run only inference.
• load_model_path (str): path to a previously trained checkpoint (.zip).

PPO can be made congestion-aware during training/execution through policy_config:

• distance_congestion_weight (0-1): weight of link utilization in node path cost feature.
• congestion_penalty_weight: terminal reward penalty weight based on projected link utilization.
• preload_link_probability: probability of preloading an edge with traffic in training episodes.
• preload_link_max_utilization: max preload utilization ratio per edge in training episodes.

Example snippet:

{
  "scenario_seed": 4040,
  "experiments": [
    {
      "name": "Big Network - Random Policy",
      "parameters": {
        "network": "create_large_network",
        "policy": "rndPlacement",
        "services_per_step": 50,
        "num_steps": 100,
        "metric_capture_interval": 5,
        "queue_delay": {
          "method": "alpha_rho_k",
          "alpha_seconds": 0.1,
          "k": 10,
          "TQ_MAX_seconds": "inf"
        }
      },
      "results": {
        "path": "results/large_network_random",
        "file": "large_network_random.json"
      }
    }
  ]
}

Training the agent

When scenario_seed is set, all experiments inherit it unless they define parameters.random_seed explicitly.

run_experiment.py reads this file and runs each experiment; evaluation and plotting use eval_experiments.py with the same config.

For DRL training-only workflows, use **experiment/training_experiments.json** and run:

uv run python experiment/train_experiments.py

train_experiments.py is dedicated to training and persisting DRL models/checkpoints and their metadata. Training hyperparameters (e.g. train_timesteps, learning_rate, n_steps, batch_size) belong in training_experiments.json.

For cross-model comparisons (Random, Greedy, PPO), use **experiment/comparison_experiments.json** and run:

uv run python experiment/eval_experiments.py
uv run python experiment/compare_models.py

Each comparison writes artifacts under:

• results/comparativa/<comparison_name>/comparison_report.json
• results/comparativa/<comparison_name>/models_and_parameters.json
• results/comparativa/<comparison_name>/step_model_evolution.json
• results/comparativa/<comparison_name>/step_model_evolution.csv
• results/comparativa/<comparison_name>/total_response_mean_by_step_ms.png
• results/comparativa/<comparison_name>/total_response_p95_by_step_ms.png
• results/comparativa/<comparison_name>/network_congestion_over_time_percent.png
• results/comparativa/<comparison_name>/cpu_congestion_over_time_percent.png
• results/comparativa/<comparison_name>/total_response_final_step_boxplot_ms.png
• results/comparativa/<comparison_name>/total_response_boxplot_by_10_steps_ms.png
• results/comparativa/<comparison_name>/total_response_boxplot_by_service_type_per_model_ms.png
• results/comparativa/<comparison_name>/summary.md

Specialized Agents (C, U, and L)

In README.md file are explained all details about the parametrization of Cost-agent, U-utilization and L-latency agents used in the study.

---

Types of Graphs Generated

After running experiments and evaluation, plots are written under each experiment’s results path in a **plots/** subdirectory. Generated artifacts:

Resource and placement

File	Description
global_CPU_RAM_percent.png	Global CPU and RAM utilization over time.
number_of_vnfs_placed_over_time.png	Total VNFs placed per step.
CPU_utilization_per_node_type_over_time.png	CPU utilization per node over time.
RAM_utilization_per_node_type_over_time.png	RAM utilization per node over time.
success_rate_over_time.png	Placement success rate per step.
absolute_CPU_RAM_usage_over_time.png	Absolute CPU and RAM usage over time.
summary_statistics.png	Summary panel of key simulation statistics.

Latency

File	Description
latency_mean_scatter_by_step.png	Mean forward/backward service latency per step (dual axis).
latency_boxplot_by_step.png	Distribution of total service latency per step.
services_not_meeting_latency_requirement_by_step.png	Count of services exceeding latency limit per step.
absolute_processing_latency_mean_scatter_by_step.png	Mean processing latency per step.
absolute_processing_latency_boxplot_by_step.png	Distribution of processing latency per step.

Communication and links

File	Description
link_utilization_over_time.png	Link utilization time series.
services_per_link.png	Number of services using each link.
final_network_communication_metrics.png	Network graph with link occupation/congestion visualization.
link_occupation_summary.md	Text summary of link occupation and congestion (in plots/).

DRL diagnostics (PPO)

File	Description
ppo_policy_loss_vs_steps.png	PPO policy loss over training timesteps.
ppo_value_loss_vs_steps.png	PPO value loss over training timesteps.
ppo_entropy_vs_steps.png	PPO entropy over training timesteps (exploration signal).

These DRL diagnostic plots are generated only by train_experiments.py (training flow), not by eval_experiments.py (runtime evaluation flow).

DRL model artifacts (PPO)

When save_trained_model is enabled, each trained PPO model is stored in models/ with:

• <model_name>.zip: SB3 PPO checkpoint.
• <model_name>.json: metadata with hyperparameters, network context, and training diagnostics history.
• If a filename already exists, an automatic suffix (_v2, _v3, ...) is added to avoid overwriting.

---

Relevant Metrics

The pipeline collects and plots the following metric groups.

Latency

• End-to-end service latency: Forward and backward path latency per placed service (from placement history and communication metrics).
• Latency components: Processing (M/M/1 at VNFs), queueing (at links, configurable model), transmission, and propagation. Total must not exceed the service’s latency requirement (D_R).
• Meet latency requirement: Per-step count of services that exceed their latency limit.
• Processing latency: Total processing time per service (plotted as mean and distribution per step).

Queue (queueing delay)

• Queue delay model: Either M/M/1 or alpha_rho_k (configurable via queue_delay in experiment_conf.json). Queue delay is applied at links; it affects total latency and thus success vs. violation counts.
• Link occupation: Per-link used vs. capacity (arrival/service rates), used to derive utilization and queue delay.

Resource utilization

• Global: CPU and RAM utilization (%) over time.
• Per node: CPU and RAM utilization per node (and by node type) over time; absolute usage over time.
• Placement: Total VNFs placed, placement success rate (successful vs. attempted per step).

Communication and links

• Link utilization: Used bandwidth vs. capacity per link over time; identification of congested links (e.g. threshold 50%).
• Services per link: Number of services (or VNF pairs) using each link.
• Link occupation summary: Markdown report of link utilization and congestion from the final state.

Together, these metrics support analysis of placement quality, latency compliance, queueing impact, and network and compute load.

---

Experimentation Setup (Reference)

Parameter	Small Network (7 Nodes)	Large Network (29 Nodes)
Network topology	7-node graph (regional).	29-node graph (scalability).
Physical links	6 wavelengths, 10 Gbps each.	11 wavelengths, 10 Gbps each.
Link lengths	MEC–SWN 1 km; SWN–EDC 20 km; SWN–CDC 300 km; SWN–SWN 2 km.	MEC/EDC/CDC–SWN 1–300 km; SWN–SWN Metro 2 km; SWN–SWN Core 5 km.
MEC resources	280 CPU cores, 1120 GB RAM.	280 CPU cores, 1120 GB RAM.
EDC resources	1400 CPU cores, 5600 GB RAM.	1400 CPU cores, 5600 GB RAM.
CDC resources	140,000 CPU cores, 560,000 GB RAM.	140,000 CPU cores, 560,000 GB RAM.
Request load	100, 200, 300, 400 requests.	300, 600, 900, 1200 requests.

Physical link capacity: total bandwidth on a link cannot exceed (wavelengths × 10 Gbps). The model enforces this as B_w.

Topology (graph)

• Nodes (N): Compute nodes (N_v): MEC, EDC, CDC; switching nodes (N_SWN): routers, OXCs.
• Links (L_P): Optical links between nodes.

From / To	Distance	Network type
MEC ↔ SWN	1 km	Access
SWN ↔ SWN (metro)	2 km	Metro
SWN ↔ SWN (core)	5 km	Core
SWN ↔ EDC	20 km	Regional
SWN ↔ CDC	300 km	Core / cloud

Latency components

Component	Symbol	Definition	Where
Processing	D_P	M/M/1 processing time at the VNF.	Compute nodes (N_v).
Queueing	D_Q	Waiting time in node/link buffers.	Especially at MEC (limited resources).
Transmission	D_T	Time to send packet (T_S) over link bandwidth.	At lightpath origin.
Propagation	D_G	Signal travel time along fibre.	Along physical links (L_P).

VNFs and service classes

VNFs (e.g. NAT, TM, WO, FW, VT, ID) have CPU/RAM and scale attributes. Service classes (AR, mIoT, Video Streaming, VoIP, Web) define VNF chains, data rate, latency limit (D_R), and bandwidth. Traffic flows through the chain from source to destination; total latency (sum of the four components) must not exceed D_R.

Small network (7 nodes) – node roles and positions

Node	Role	Neighbours	Position
1	MEC	3	78×123
2	MEC	4	139×102
3	SWN	4, 5	194×129
4	SWN	3, 5	245×107
5	SWN	3, 4, 6, 7	319×121
6	EDC	5	319×66
7	CDC	5	414×125

Large network (29 nodes) – node roles and neighbours

Node	Role	Neighbours	Position
1	SWN	2, 6, 24	184×386
2	SWN	1, 3, 25	176×308
3	SWN	2, 4, 26	202×230
4	SWN	3, 5, 8, 23	243×253
5	SWN	4, 6, 13, 22	277×334
6	SWN	1, 5, 22	249×426
7	SWN	8, 12, 27	247×175
8	SWN	7, 9, 4, 23	284×211
9	SWN	8, 10	384×210
10	SWN	9, 11, 21, 18	457×207
11	SWN	12, 10, 21, 29	417×144
12	SWN	7, 11, 28	326×149
13	CDC	5, 18, 14	377×317
14	CDC	13, 18, 20, 15, 16	448×394
15	CDC	14, 16, 20	543×394
16	CDC	14, 15, 20, 17	558×316
17	CDC	18, 20, 16	497×270
18	CDC	10, 13, 14, 17	430×266
19	CDC	14, 17	465×318
20	CDC	15, 16	601×359
21	EDC	11, 10	526×165
22	EDC	5, 6	299×410
23	EDC	4, 8	300×254
24	MEC	1	119×383
25	MEC	2	115×281
26	MEC	3	119×199
27	MEC	7	179×125
28	MEC	12	256×106
29	MEC	11	436×101

Large network topologies

Metric	Large Network v1 (Base)	Large Network v2 (Var 1)	Large Network v3 (Var 2)	Large Network v4 (Var 3)
Total Nodes	29	27	34	28
Total Edges	39	42	78	49
CDC Nodes	8	5	8	2
EDC Nodes	3	4	4	8
MEC Nodes	6	6	9	6
SWN Nodes	12	12	13	12
CPU Cores	45,880	32,280	48,120	42,880
RAM (GB)	471,520	309,120	480,480	91,520
Wavelengths per Link	11	11	11	11
Bandwidth per Link (Gbps)	110	110	110	110
Total Network Bandwidth (Gbps)	4,290	4,620	8,580	5,390
Distance Range (km)	1–300	0–300	1–300	0–20
Average Distance (km)	20.82	41.10	36.99	7.39
Average Degree	2.69	3.11	4.59	3.50
Network Density	0.0961	0.1197	0.1390	0.1296
Is Connected	True	True	True	True
Avg Shortest Path Length	3.85	3.11	2.71	3.07
Network Diameter	8	7	5	8

VNF resources (CPU, RAM, scale)

VNF	Abbrev.	CPU	RAM	Scale
Network Address Translation	NAT	0.00092	0.00092	1
Traffic Monitoring	TM	0.0133	0.0399	1
WAN Optimizer	WO	0.0054	0.0108	1
Firewall	FW	0.0009	0.00135	1
Video Transcoder	VT	0.0054	0.0054	2
Intrusion Detection	ID	0.0107	0.0107	1

Service classes (composition and requirements)

These service classes are defined in **infrastructure/service_vnf.py**.

Class	Application	Share	Data rate	Latency (D_R)	BW	VNF chain
1	Augmented Reality (AR)	39.5%	100 Mbps	1 ms	High	TM → VT → ID
2	mIoTs	10.5%	100 Mbps	5 ms	Low	ID
3	Video Streaming (VS)	39.5%	4 Mbps	100 ms	High	TM → VT → ID
4	VoIP	1.5%	64 Kbps	100 ms	Low	TM → FW → NAT
5	Web Services	9.45%	100 Mbps	500 ms	Variable	TM → WO → ID

The user is assumed to be attached to an MEC node (traffic origin). Traffic flows forward from that MEC (s_k) through each VNF in the chain; some services also need to transmit back to the user, so both forward latency and backward latency are considered. The link from the user to the MEC node is negligible and is modelled as zero (not counted in latency). The sum of processing, queueing, transmission, and propagation latency on the forward and backward paths must not exceed D_R.

Agent decisions

Acknowledgment

• This work was supported by Grant PID2024-158637OB-I00, funded by MICIU/AEI/10.13039/501100011033 and by ``ERDF A way of making Europe'' (ERDF/EU).