Skip to content

Support timeout customization for synchronous File upload API #7201

@vycius

Description

@vycius

Is your feature request related to a problem? Please describe.
I am frustrated when attempting to use the synchronous file upload endpoint (post/pulp/api/v3/content/file/files/upload/) to fetch large files from external URLs. Currently, the API does not allow users to specify custom timeout parameters.

When pulling large datasets—for example, the OpenStreetMap planet file at https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf—the connection exceeds the default aiohttp limits, causing the upload to fail without any way for the user to extend the grace period for high-latency or high-volume transfers.

Describe the solution you'd like
I would like the post/pulp/api/v3/content/file/files/upload/ endpoint to accept the following optional fields in its request body to align with other Pulp download behaviors:

  • total_timeout: (number or null) Customizes aiohttp.ClientTimeout.total.
  • connect_timeout: (number or null) Customizes aiohttp.ClientTimeout.connect.
  • sock_connect_timeout: (number or null) Customizes aiohttp.ClientTimeout.sock_connect.
  • sock_read_timeout: (number or null) Customizes aiohttp.ClientTimeout.sock_read.

These should default to null, allowing the system to fall back to standard aiohttp defaults unless explicitly overridden by the user.

Describe alternatives you've considered
I considered creating a Remote to handle the download, but this is not possible in this use case. A File Remote requires a PULP_MANIFEST file to be present at the source to index the content. The source in question (e.g., OpenStreetMap) provides direct links to large binary blobs without a Pulp-compatible manifest, making the synchronous file_url upload the only viable method.

Additional context
This feature would bring parity between the synchronous File upload API and other download-related tasks in Pulp that already utilize aiohttp timeout configurations.

Example payload structure requested:

{
  "file_url": "https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf",
  "relative_path": "planet-latest.osm.pbf",
  "total_timeout": 3600,
  "sock_read_timeout": 600
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions