-
Notifications
You must be signed in to change notification settings - Fork 243
Feat/genapi add shared responsibility model #5125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,134 @@ | ||||||||||||||||||
--- | ||||||||||||||||||
meta: | ||||||||||||||||||
title: Security and Reliability in Generative APIs | ||||||||||||||||||
description: Learn more about shared responsibility in security and reliability practices for Generative APIs | ||||||||||||||||||
content: | ||||||||||||||||||
h1: Security and Reliability in Generative APIs | ||||||||||||||||||
paragraph: Learn more about shared responsibility in security and reliability practices for Generative APIs | ||||||||||||||||||
tags: generative-apis ai-data shared responsibility security reliability | ||||||||||||||||||
dates: | ||||||||||||||||||
validation: 2025-06-16 | ||||||||||||||||||
--- | ||||||||||||||||||
|
||||||||||||||||||
This page outlines key principles and best practices to help you ensure your applications' security and reliability when using Generative APIs. | ||||||||||||||||||
|
||||||||||||||||||
## Resilience | ||||||||||||||||||
|
||||||||||||||||||
Resilience ensures the continuity and availability of your applications and data, even in the face of disruptions or failures. In Generative APIs, you can promote resilience through the following pillars: availability, durability and performance. | ||||||||||||||||||
|
||||||||||||||||||
### Availability and durability | ||||||||||||||||||
|
||||||||||||||||||
Generative APIs SLAs target the following SLOs: | ||||||||||||||||||
|
||||||||||||||||||
| Processing Type | Configuration Details | Availability | | ||||||||||||||||||
| ------------ | ------------------- | ------ | | ||||||||||||||||||
| Standard | Standard synchronous HTTP calls to Generative APIs providing the generated content directly in HTTP response. These calls include stream and non-stream requests. | 99.9% | | ||||||||||||||||||
| Batch | Asynchronous processing of files sent to Generative APIs providing the generated content through files. | 99.9% | | ||||||||||||||||||
|
||||||||||||||||||
The detailed SLAs measurements and guarantees can be found on the [Service Level Agreement for Generative APIs](https://www.scaleway.com/en/database/sla/) page. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
As we do not store any data with Standard processing durability requirements do not apply. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
When processing data using Batch processing, your input data is stored only during processing time (24 hours): | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
- As input data storage is only temporary, no specific durability guarantee apply. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
- Output data (processing results) durability depends on the target storage system used (by default: Object Storage Standard Class) | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
## Performance | ||||||||||||||||||
|
||||||||||||||||||
Standard processing (synchronous HTTP calls): | ||||||||||||||||||
- Generative APIs runs models on mutualized infrastructure, and therefore ensures good performance in average utilization. We monitor and respond quickly to any drops in token generation throughput, but cannot guarantee strictly performance, especially during customer peak loads. As a consequence, [Rate Limits](https://www.scaleway.com/en/docs/generative-apis/reference-content/rate-limits/) apply, to ensure "fair use" of synchronous HTTP calls. Bigger requests volumes should be treated through Batch processing. | ||||||||||||||||||
- Guaranteed performance can be provided using dedicated resources on Managed Inference product. | ||||||||||||||||||
Comment on lines
+40
to
+41
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Batch processing (asynchronous file processing): | ||||||||||||||||||
- When using Batch processing, we handle scheduling of batch jobs to optimize both processing ressource allocation and processing time. Processing time is therefore only guaranteed to be lower than 24 hours and [Rate Limits](https://www.scaleway.com/en/docs/generative-apis/reference-content/rate-limits/) (larger than Standard processing) still apply. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
## Monitoring | ||||||||||||||||||
|
||||||||||||||||||
Monitoring is an essential pillar to ensure the security and reliability of your services. The practice provides real-time insights into the performance, security, and resource consumption of your Generative API usage. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
### Metrics and logs | ||||||||||||||||||
|
||||||||||||||||||
Generative APIs metrics and logs are stored inside [Cockpit](https://www.scaleway.com/en/docs/cockpit/). | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
This includes: | ||||||||||||||||||
- **Metrics**: Input and Output tokens and API requests . Metrics are refreshed every minute (some dashboards may aggregate data by the hour for accuracy reasons, but metrics can be queried at a finer rate using Cockpit Custom Dashboards) | ||||||||||||||||||
- **Logs**: No logs are currently stored inside Cockpit. | ||||||||||||||||||
Comment on lines
+55
to
+56
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
## Configuration and version management | ||||||||||||||||||
|
||||||||||||||||||
Configuration and version management are critical for maintaining reliability and performance across your services. | ||||||||||||||||||
|
||||||||||||||||||
### Configuration | ||||||||||||||||||
|
||||||||||||||||||
Currently, Generative APIs do not provide specific configuration properties stored within your account. All configuration parameters are the one you send through each API HTTP calls (such as `temperature`, `top_p` or `seed`) and you remain responsible for any change in outputs based on these parameters. | ||||||||||||||||||
|
||||||||||||||||||
Since Generative AI models are by definition non-deterministic, we cannot guarantee the same input will provide the same output over time (for example when used two different HTTP calls). If you want deterministic processing, we encourage you to use Managed Inference with a specific model and set all randomness parameters to deterministic level (for example using for instance `temperature`:`0` and a specific `seed` value). | ||||||||||||||||||
Comment on lines
+64
to
+66
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
### Version management | ||||||||||||||||||
|
||||||||||||||||||
#### Supported models | ||||||||||||||||||
|
||||||||||||||||||
Any changes to supported models and associated guarantees are detailed in our [Model Lifecycle Policy](https://www.scaleway.com/en/docs/generative-apis/reference-content/model-lifecycle/). | ||||||||||||||||||
|
||||||||||||||||||
Comment on lines
+72
to
+73
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
#### API versions | ||||||||||||||||||
|
||||||||||||||||||
Two types of API version updates may be performed: | ||||||||||||||||||
|
||||||||||||||||||
| Upgrade Type | Description| | ||||||||||||||||||
| ------------ | ------------------- | | ||||||||||||||||||
| Minor | These updates do not change the API current fields format and are backward compatible (no action is required on your side). New fields and feature can however be added. | | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
| Major | These updates change the API current fields or path. They may require action from your side. In this case, we will notify you with at least 3 month notice before deprecating significant feature that might break your application. | | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
#### Third party tools compatibility | ||||||||||||||||||
|
||||||||||||||||||
By following industry standards (such as targeting OpenAI API compatibility), we aim to provide compatibility with most AI ecosystem and tools by default. However, as ecosystem evolves quickly, we cannot guarantee strictly compatibility with third party tools, but provide extensive documentation: | ||||||||||||||||||
- Current API supported features are available in our API Documentation | ||||||||||||||||||
- Advanced errors and edge cases workaround in our [Troubleshooting section](https://www.scaleway.com/en/docs/generative-apis/troubleshooting/fixing-common-issues/). | ||||||||||||||||||
- Integration with third party tools is available in [Integrating Scaleway Generative APIs with popular AI tools](https://www.scaleway.com/en/docs/generative-apis/reference-content/integrating-generative-apis-with-popular-tools/#openai-client-libraries) | ||||||||||||||||||
Comment on lines
+85
to
+88
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
## Data protection | ||||||||||||||||||
|
||||||||||||||||||
Our Data protection measures are detailed in our [Privacy Policy](https://www.scaleway.com/en/docs/generative-apis/reference-content/data-privacy/). | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Overall: | ||||||||||||||||||
- We do not store sensitive data (such as the content of your prompt), unless we need to, to provide the service (such as temporarily when using Batch processing). | ||||||||||||||||||
- When data is stored, it is protected using state of the art method (such as encryption at rest). | ||||||||||||||||||
- During transit, your data is encrypted by relying on HTTPS protocol. | ||||||||||||||||||
Comment on lines
+94
to
+97
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
### Scaleway access | ||||||||||||||||||
|
||||||||||||||||||
In order to perform maintenance operations and guarantee the reliability of Generative APIs, or comply with local regulations, we need to access servers hosting Generative APIs service. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Most of these accesses are automatic actions, in cases where Scaleway needs update configuration or upgrade software versions. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Manual interventions might be required occasionally, and for troubleshooting reasons (such as specific customer requests generating errors or performing malicious activity), we may access temporarily complete HTTP requests content to identify root cause issue or any security vulnerability. All Scaleway accesses are authenticated and traced following industry security standards. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
## Identity and access management | ||||||||||||||||||
|
||||||||||||||||||
Identity and access management allows you to enable granular control over user permissions and to mitigate the risk of unauthorized access or data breaches. | ||||||||||||||||||
|
||||||||||||||||||
All accesses to Generative APIs are authenticated and authorized relying on [Scaleway IAM permissions sets](https://www.scaleway.com/en/docs/iam/reference-content/permission-sets/). | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
You are responsible for attributing these permissions to the relevant users or applications and reviewing these accesses frequently. | ||||||||||||||||||
|
||||||||||||||||||
## Compliance | ||||||||||||||||||
|
||||||||||||||||||
Several regulations apply to us directly, whereas other apply to your usage. Even in this case, we help you ease your compliance process by providing you with the information you need from your Cloud Provider. | ||||||||||||||||||
|
||||||||||||||||||
Comment on lines
+117
to
+118
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
### AI Act | ||||||||||||||||||
|
||||||||||||||||||
We (Scaleway) ensure our compliance with the AI Act within our scope of responsibilities. We also ensure that you have the information needed to comply with the requirements that apply to you. This means concretely: | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Provide link to the AI Act (this one maybe? https://artificialintelligenceact.eu/) |
||||||||||||||||||
- Gathering AI Model Providers information about their model (such as whether its training capacity is above 10²⁵ FLOPs, and falls into a specific category) and provide you with a link to these documents when they are made available by AI Model Providers. | ||||||||||||||||||
- Providing you with links towards licensing required by the AI Model Providers. | ||||||||||||||||||
Comment on lines
+122
to
+123
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Scaleway has no access to, nor knowledge of, any inputs and outputs generated by the Models. By using our AI products, you agree and acknowledge that you are (i) responsible for this use including any content integrated into the Models, and (ii) required to use the AI products in compliance with our General Terms of services. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Provide link to the General terms of services |
||||||||||||||||||
|
||||||||||||||||||
As a client of our AI products, you are likely to be considered an AI System Provider or Deployer under the AI Act. As such, it is your responsibility to ensure you comply with requirements that apply to you. | ||||||||||||||||||
|
||||||||||||||||||
### Additional Local Regulation | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
If you require additional information to comply with specific regulation, you can create a [support ticket](https://console.scaleway.com/support/tickets/create) or contact your account manager. | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe explain what "SLA", and "SLO" stand for, for people who aren't familiar with the terms.