-
Notifications
You must be signed in to change notification settings - Fork 136
Description
Is your feature request related to a problem? Please describe.
I am frustrated when attempting to use the synchronous file upload endpoint (post/pulp/api/v3/content/file/files/upload/) to fetch large files from external URLs. Currently, the API does not allow users to specify custom timeout parameters.
When pulling large datasets—for example, the OpenStreetMap planet file at https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf—the connection exceeds the default aiohttp limits, causing the upload to fail without any way for the user to extend the grace period for high-latency or high-volume transfers.
Describe the solution you'd like
I would like the post/pulp/api/v3/content/file/files/upload/ endpoint to accept the following optional fields in its request body to align with other Pulp download behaviors:
total_timeout: (number or null) Customizesaiohttp.ClientTimeout.total.connect_timeout: (number or null) Customizesaiohttp.ClientTimeout.connect.sock_connect_timeout: (number or null) Customizesaiohttp.ClientTimeout.sock_connect.sock_read_timeout: (number or null) Customizesaiohttp.ClientTimeout.sock_read.
These should default to null, allowing the system to fall back to standard aiohttp defaults unless explicitly overridden by the user.
Describe alternatives you've considered
I considered creating a Remote to handle the download, but this is not possible in this use case. A File Remote requires a PULP_MANIFEST file to be present at the source to index the content. The source in question (e.g., OpenStreetMap) provides direct links to large binary blobs without a Pulp-compatible manifest, making the synchronous file_url upload the only viable method.
Additional context
This feature would bring parity between the synchronous File upload API and other download-related tasks in Pulp that already utilize aiohttp timeout configurations.
Example payload structure requested:
{
"file_url": "https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf",
"relative_path": "planet-latest.osm.pbf",
"total_timeout": 3600,
"sock_read_timeout": 600
}