Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non-buffered streams for 'ResumableUpload'. #62

Open
krishna-kashyap opened this issue Mar 20, 2019 · 3 comments
Open

Support non-buffered streams for 'ResumableUpload'. #62

krishna-kashyap opened this issue Mar 20, 2019 · 3 comments
Labels
api: storage Issues related to the googleapis/google-resumable-media-python API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@krishna-kashyap
Copy link

krishna-kashyap commented Mar 20, 2019

Hi,

In get_next_chunk() method the stream is concluded to have ended on the read data being less than chunk_size rather than a empty chunk.

    if total_bytes is None:
        if num_bytes_read < chunk_size:
            # We now **KNOW** the total number of bytes.
    total_bytes = end_byte + 1

This could be a cause of concern especially in case of dynamically generated data (where nothing is written to file and data is being directly uploaded to GCS)

@krishna-kashyap krishna-kashyap changed the title Uploading of dynamically generated streams of data Detection of end of stream in ResumableUpload Mar 20, 2019
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Mar 21, 2019
@sduskis sduskis added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Apr 2, 2019
@tseaver tseaver added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Aug 28, 2019
@tseaver tseaver changed the title Detection of end of stream in ResumableUpload Support non-buffered streams for 'ResumableUpload'. Aug 28, 2019
@tseaver
Copy link
Contributor

tseaver commented Aug 28, 2019

Per the Python docs for io, "Interactive" streams as well as sockets / pipes can return "short" reads even when not at EOF. Supporting such streams for chunked / resumable uploads is a challenge: we need to be able to pass along a valid Content-Range header with each chunk.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/google-resumable-media-python API. label Mar 4, 2021
@offsetcyan
Copy link

I've just (mistakenly, I suppose) raised this at googleapis/python-storage#1185. The last update in 2019 references interactive streams - this can at least be fixed with respect to non-interactive streams by calling isatty(). It also allows for a clearer error message to be raised, that interactive streams are not supported.

@offsetcyan
Copy link

offsetcyan commented Nov 17, 2023

I've made a branch with a commit for the synchronous client that works on my machine™ to write to cloud storage using google/python-storage, but doesn't have tests (editing the test to use a non-interactive non-seekable stream didn't seem to just work...). Could be a starting point for a more competent engineer to start: main...offsetcyan:google-resumable-media-python:resumable-non-interactive-streams

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/google-resumable-media-python API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

5 participants