Compliant use of tokens in blockwise transfer (RFC7959)

[boaks]

In the scope of LwM2M, a question about the usage of the token in relation with a blockwise transfer occurred.

Is it valid to use the client's token to relation a server side state to the request? In particular, to make it mandatory, that follow up requests of an blockwise transfer MUST use the same token?

For more background information, please read the comment in

Eclipse/Californium

and

Eclipse/Leshan

[chrysn]

I think it's rather clear that this is not how requests are supposed to be matched; IIRC there was wide agreement on that in the last interim.

While clients are free to reuse the tokens, the server can not rely on it. Given we are seeing evidence that people start relying on this just because it works with the one or two peers they are testing it, I'd recommend that clients start mixing up their block numbers just to keep their servers from relying on the pattern. (It's sad we have to resort to this, but apparently either people don't read the specs or the specs are not written well enough).

If there is any property of its interactions a server can only obtain when exchanging with peers that do use those constant-over-the-body tokens, please point me to it, but I'm very confident that the tools we have allow doing it properly. (That is not to say that 7959+9175 are perfect, far from it -- just that everything that can be done relying on constant tokens can also be done without).

[boaks]

either people don't read the specs or the specs are not written well enough

In some cases the people want to improve the specs. I remember converting RFC7641 into time series. For blockwise it was to support concurrent transfers of changed resources.
What is overseen frequently is the downside of such approaches. In that case it seems to be more the "concurrent transfers" what should be achieved. From my point, very crazy and dangerous. In the end a "break" will break 5 90% transfers instead of breaking one at 50% but have the other 4 done.

[SeppoTakalo]

Let me try to put up an example that might help to you to see what problems I'm seeing.

In LwM2M, a device holds multiple "objects" and those can have "resources" and all those are accessed using their CoAP URI path.
For example Device object

In LwM2M, you can send a query for a single or multiple resources. So for example, if you send query to path 3/0 you will get a payload that contains a content of all those resources mentioned in the link above. This is important detail, payload is generated, it is not static.

Now as you can see, one of the resource is "Current Time". So as you can expect, this changes as often as what is the accuracy of the system clock. Once a second, or once a millisecond.
So whatever payload we generate, will be different one second later. So only way to ensure the integrity of the paylod is that we keep a state of blockwise transfers.

So the problems, as I see it:

1. If I get query of GET 3/0 and the response is so big that it needs blockwise. I split the payload and send response with BLOCK2, N=0.
   what CoAP options do I use to tell the client that on the next query, ask for block from this specific payload generated on timestamp t=X. I could append Etag option, but client cannot use that as a paremeter to next query. If query contains Etag, I can only answer "2.03 Valid response". I cannot append Request-Tag either, as it can only appear in request, not response.

2. If I get again a similiar query of GET 3/0 and no Request-Tag or BLOCK2 tags, do I generate a new payload or send a first block of the previous payload?

3. If I get query of block N=0 and no Request-Tag, do I generate a new payload, or do I find an old one that has same URI?

4. If I get query of GET 3/0, BLOCK2, N=1, do I use a URI path to find a payload that was generated for block N=0 and send a N=1 block of that? Or what do I use to find the exact same query?

5. If I get query of block N=10 and I have already send N=11, is that an error or should I just respond normally? Out-of-order block numbers were already proposed here.

6. If out-of-order block transfer is fine, then I cannot use last block as any kind of meaningful indicator that transmission is over. When am I allowed to free the memory that I have reserved to a given payload?

--
In all of the cases above, I would have used Tokens to distinct an query that initiates generation of a new payload, or send block from already generated payload. Request-Tag seem obvious choice, but as the first example shows, it has the problem that if it is a server that initiates the block transfer, then it cannot append that option. If there is no knowledge on client side that response might be a block-wise, then how do you know when to append the Request-Tag?

[SeppoTakalo]

Regarding the security, I'm thinking that as long as the same pair of URI-path & block number does not get reused within the same token, I don't consider it as reusing of token.
So I can be seen as a long ongoing request that contains multiple packets.

But if you store one response packet, it will not match any further transfers as the token has already been used.

[chrysn]

So only way to ensure the integrity of the paylod is that we keep a state of blockwise transfers.

So far, that's a pretty common thing.

Quote from 7959

To avoid [changing ETags] happening all the time for a fast-changing resource, a server MAY try to keep a cache around for a specific client for a short amount of time. The expectation here is that the lifetime for such a cache can be kept short, on the order of a few expected round-trip times, counting from the previous block transferred."

what CoAP options do I use to tell the client that on the next query, ask for block from this specific payload generated on timestamp t=X.

None at all. If a later request for a later block comes in before your stored version expires, you send the stored version.

If I get again a similiar query of GET 3/0 and no Request-Tag or BLOCK2 tags, do I generate a new payload or send a first block of the previous payload?

If you get any request with Block2:0/-/x (or without a Block2), you can do either, and either should work. The difference will just be that if a client requests the same thing all over, not regenerating gives the client a stale view, so you'd need to be careful in picking that "short amount of time". Or, what I'd recommend, just regenerate.

On being "similar" (vs. "identical"), 7959 didn't have a good term, so I introduced "matchable" in RFC79175. You can only serve from the stored version if the request is "matchable" to the original one (which you can reasonably check by storing a hash of the client's address and the relevant option values). If the request is not matchable, you must not use that stored version. What you do precisely depends on how you store them -- you may have just a single "active transfer", or you may have a bunch of them, in which case keying them by that hash is a good option.

If I get query of block N=0 and no Request-Tag, do I generate a new payload, or do I find an old one that has same URI?

If I get query of GET 3/0, BLOCK2, N=1, do I use a URI path to find a payload that was generated for block N=0 and send a N=1 block of that? Or what do I use to find the exact same query?

Note that Request-Tag is not in any way special to the server. The rules for being "matchable" ("same endpoint pair, have the same code, and have the same set of options, with the exception that elective NoCacheKey options and options involved in block-wise transfer (Block1, Block2, and Request-Tag) need not be the same") make them an exception because it simplified phrasing the document for the client -- but on the server side, you just build a hash of the endpoint, code, and options unless NoCacheKey or Block1/Block2, and that's what needs to match. Also, the URI is not special there -- Uri-Path, Uri-Query and Accept are all options that need to match.

If I get query of block N=10 and I have already send N=11, is that an error or should I just respond normally? Out-of-order block numbers were already proposed here.

It is not an error, it is just recommended that the client fetches them in order. As long as the rest is good, you can just send the block they requested.

If out-of-order block transfer is fine, then I cannot use last block as any kind of meaningful indicator that transmission is over. When am I allowed to free the memory that I have reserved to a given payload?

Just because out-of-order is fine doesn't mean you have to go out of your way to accommodate it. But having sent the last block is also not a reliable indicator to free things, because you usually don't get an acknowledgement for the block. How smart you have to be depends on your constraints and requirements. Having one global "last transaction" block and a single timeout (that gets refreshed every time a request matches it) and overwriting it if anything else comes in requiring block-wise is fine. Having a small number of slots in an LRU cache is also fine and a bit better. If you have different authorization levels, some slots in the LRU may be reserved for admins. If your timeout is large, it may make sense to not go full LRU, but also consider whether you have transmitted the last block in an entry, and evict that even before it expires. But these are all optimizations that just modify when and to whom the error cases occur, and the error cases boil down to one situation:

The state which the client expected to continue with was overwritten. The client detects that using the ETag, and starts over.

security / token

The issue I'm taking with making this a token thing is not about security, it's about layering. The token is on the request/response layer, but block-wise happens in the application visible options, on top of that.