Collecting Gateway Stats

[shanakama]

Summary

In business perspective and analytical perspective , we need to collect some information about how clients are using the API Platform. This discussion is initiated to get an idea about the information what we need to collect about how client uses self-hosted gateways with the API platform (like how many gateway replicas they have , how many connections established , any abnormal activities and so on).

Also these data would use to some operational activities like rate limiting/throttling.

currently we have identified following parameters to collect.

Parameter	Sample Value
number_of_gateway_controllers	1
last_connected_time	1222200002292
gateway_version	0.3.0
number_of_apis_deployed	10
org_UUID	e4cd8524-94ca-41a9-93a0-72b61e64664c
status	active

Storing the stat data

There are several options to store or publish stats.

1. publish data to moesif

We can publish these data to moesif and that provide more capabilities to visualize necessary data. But this method is not ideal for functions like rate limiting (If we think to use moesif data to make rate limiting decisions). Because data synchronization is not real time (~30 seconds delay).

2. Store stats in the platform API DB

Since we already having a DB associate with the platform API , we can use the same DB to store stat data.
Currently gateway ID (gatewayId) and gateway status (isActive) are persisted to the DB. Connection ID (connectionID) is capturing but does not persist. So , it is required to have a separate table to store stat data.

Add a new table to track connection history:

-- Connection statistics table
CREATE TABLE connection_stats (
    id VARCHAR(255) PRIMARY KEY,
    connection_id VARCHAR(255) UNIQUE NOT NULL,
    gateway_id VARCHAR(255) NOT NULL,
    connected_at TIMESTAMP NOT NULL,
    disconnected_at TIMESTAMP,
    is_active BOOLEAN DEFAULT true,
    gateway_version VARCHAR(10). NOT NULL, 
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    INDEX idx_gateway_id (gateway_id),
    INDEX idx_connection_id (connection_id),
    INDEX idx_is_active (is_active),
    INDEX idx_connected_at (connected_at)
);

3. Logging the stat and query from opensearch

We can follow some logging based mechanism to collect analytics data.

4. Store stat in a redis

Store the data in redis is also one option but If this gonna happen , we have to add a redis instance to the cluster as an additional component. But the control plane is scalable horizontally , then there might have multiple CP instances. Redis is better this kind of scenarios.

Main objective of this discussion is to know

1. What are the other values need to be collected other than above ? (may be in business point of view)
2. Using Redis as the metrics storage
3. Data retention period

Why do we need these data ?

1. For some internal security purposes like Rate limiting
2. As analytical data for business purposes
3. May be for visualize in dashboard for gateway owners (Number of active gateways, Number of APIs deployed in each gateway ..)