Add a method in the HUB-SDK to generate pre-signed URLs for uploading larger datasets

[anata404]

Search before asking

• I have searched the HUB issues and found no similar feature requests.

Description

I’d like to request the addition of a method in the HUB-SDK to generate pre-signed URLs for uploading larger datasets programmatically. This functionality would complement the existing get_download_link method and enhance SDK usability for managing large datasets.

Use case

The current dataset.upload_dataset(file="<Dataset File>") method has a 10MB file size limit. Implementing a method to generate pre-signed URLs could be a viable solution to bypass this restriction.

Additional

No response

[UltralyticsAssistant]

👋 Hello @anata404, thank you for your feature request related to the Ultralytics HUB SDK 🚀! This is an automated response, but rest assured, an Ultralytics engineer will review your issue and assist you soon.

In the meantime, please be sure to visit our HUB Docs for helpful resources:

• Quickstart. Easily get started with training and deploying YOLO models.
• Datasets: Preparing and Uploading. Learn how to upload datasets in YOLO format and review current size limitations.
• SDK API Docs. Explore the full capabilities of the HUB-SDK for managing datasets, projects, and models programmatically.

We understand that improving SDK functionality for handling larger datasets is an important update, and your suggestion of implementing pre-signed URL generation could be a valuable addition. To help us evaluate this enhancement further, feel free to share more details or specific use cases if needed.

This request will be reviewed, and we’ll keep you updated on progress. Thank you for helping us improve Ultralytics HUB! 🚀✨

[anata404]

Here is a detailed discuss #963. And a great appreciate to @pderrenger

[pderrenger]

@anata404 thank you for your thoughtful feature request! Adding support for pre-signed URL generation to handle larger dataset uploads is a great idea and aligns well with enhancing the usability of the HUB-SDK. 🎉

Currently, the upload_dataset method has limitations, as you've noted. A potential enhancement could involve integrating a method that generates pre-signed URLs, allowing users to programmatically upload larger files. This functionality could complement the existing get_download_link method.

I'll forward this feature request to our development team for consideration. In the meantime, if you're open to contributing, feel free to submit a pull request to the HUB-SDK repository. We welcome community contributions and would love to collaborate with you on implementing this feature.

If you'd like to stay updated, please subscribe to this issue or the repository. Let me know if you have any further questions or if there's anything else I can assist you with. Thank you for your support and for being an active part of the YOLO community! 🚀