Response to HEAD should use Content-Length, not Offset

[jonhoo]

The protocol dictates that the response to a HEAD request on an incomplete upload should reply with an Offset header indicating how much data the server already has. Wouldn't it make more sense to use the Content-Length header here, as we are really telling the client how much data is already stored on the server?

[felixge]

The Content-Length must be set to whatever amount of data we return for GET. This may or may not be same amount that has been uploaded so far. A server may also decide to always advertise a Content-Length identical to the Final-Length, which means GET requests will be kept alive and filled with data as it arrives on the server.

This will be adressed in more detail in the protocol soon, but I think defining a new Offset header is required to adress this scenario. What do you think?

[jonhoo]

Why would the GET not return the amount of data that has already been uploaded?
Advertising a Content-Length which is equal to the Final-Length seems incorrect unless the upload has finished though... The spec says:

The Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient

and

Applications SHOULD use this field to indicate the transfer-length of the message-body

Which to me indicates that it should reflect the amount of bytes available for the entity, not how long it might potentially end up being? This would also work well should the protocol add support for entities where Final-Length is unknown when the upload starts.

I think defining an Offset header is the right thing to do for the PATCH request, but it doesn't seem semantically correct for a HEAD response, because that would imply the server's offset into the file. After all, a HEAD response is not the server asking the client for data from a given offset, it's the server telling the client what it already has. In that context, Offset doesn't really make sense?

[felixge]

Which to me indicates that it should reflect the amount of bytes available for the entity, not how long it might potentially end up being?

IMO the application defines what the entity is at any given point in time. In your case you may define the entity to be the data that has been received so far. However, another application may define it as the data it expects from the Final-Length.

Do you disagree with this?

[vayam]

+1 to remove Offset
This makes HEAD and GET consistent. It would be useful to add Entity-Length to HEAD and GET, so backend has ability to download while upload is in progress.

HEAD /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
Host: tus.example.org

Response:

HTTP/1.1 200 Ok
Content-Length: 70
Entity-Length: 100


GET /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
Host: tus.example.org

Response:

HTTP/1.1 200 Ok
Content-Length: 70
Entity-Length: 100

bytes

[jonhoo]

I think @vayam's suggestion is pretty good. Using Entity-Length to indicate the length of the entire entity (even though it might not be fully uploaded) and Content-Length as the length of the content of the reply. Keeping GET and HEAD consistent should also be a priority for the spec in my opinion, and the suggestion above addresses that too.

[felixge]

I guess I'm ok with this. The upload->download realtime streaming use case is IMO still interesting, but I'm willing to leave it unspecified unless there is a lot of interest from other people. (Anybody who needs it could still do it by using a different resource or a ?stream=true query argument or something.

I'll make the change.

[jonhoo]

I suppose the real time streaming could be indicated by a request header. Something like Expect: complete, which would make the server send Entity-Length as Content-Length and keep the connection open to continue streaming the file as it gets uploaded.

[felixge]

@jonhoo that's not very practical if you'd like to share the file url with somebody else (which is IMO the main use case here).

[jonhoo]

Hmm, true... I suppose the two could be reversed as the more common use case might be wanting the entire file, not just bits of it. Perhaps we could turn it around by saying that the default (if keep-alive is used) is to keep the connection open until the upload has finished. In that case you could get away with setting Content-Length to the size of the entity, and Entity-Length would no longer be needed. We could then cater for the other use case (downloading what has already been uploaded, but not keep the connection going) by using an explicit request header?

[felixge]

(if keep-alive is used)

What does keep-alive have to do with any of this?

We could then cater for the other use case (downloading what has already been uploaded, but not keep the connection going) by using an explicit request header?

My original idea was to define both GET strategies and leave it to the server to choose one or the other. However, the realtime download strategy is only doable with the "Offset" header, which is why I liked the idea of always including it for GET/HEAD.

Anyway, I think we should make a decision here one way or another. Are you +1 or -1 on removing the Offset header now?

[vayam]

@felixge @jonhoo I personally like the streaming use case. being able transcode while the video is being uploaded by just passing url. however to make tus generic, IMO we shud probably keep it out till we can think of good extension. whoever wants to implement can still do it like @felixge suggested

[jonhoo]

@felixge Sorry, poor wording on my part. I meant keep alive as in that the client keeps the connection open, not as in the keep-alive header flag.

Leaving the decision to the server will do nothing but add fragmentation of implementations.
I'd say +1 to removing Offset regardless of what we do as streaming, and then suggest we open another ticket to discuss how streaming support might be catered for by the standard as well.