UDP and byte ranges

[dormando]

(I'll try not to re-echo some of Dustin's points).

> Add a field to the GET response akin to DNS's "there's more data but you
> need to ask for it". The first response packet will tell the client how
> long the entire key is in an extras field, and the common header will
> tell the client how long the data it got in the initial response is.
> 
> Add a new command, RGET (range-get), that defines a larger extras
> section with two additional fields, the offset and the length.
> 
> The client is explicitly allowed to ask for more data than can fit in a
> single UDP packet.

In order for this to work the client will have to do several roundtrips
to then further negotiate a potentially lossy connection. I read this
over a few times and I don't see, in theory, it having value over TCP. I
do agree with (most) of the implementation ideas... just feels like it's
not necessary.

Packing small commands into UDP packets for one-offs can have a huge
performance benefit on its own, but a clogged link will demolish the
efficiency from even a well thought-out system. I see it happen plenty
of times with online gaming, before the "smarter" ones switched back to
TCP :)

> The server sends as many RGET response packets as it needs to send, with
> each one containing enough information (offset and length) to reassemble
> the value on the client _without resequenceing the packets_!

It's kind of doing this implicitly? Unless I missunderstand :(

> If the client needs to rate limit the response, it can send separate
> RGET requests with each one asking for some length of data that the
> client can handle at that moment.

Final bit of my point. Having to manually rate limit the responses
without being able to rely on EAGAIN and EWOULDBLOCK means we'll
probably have to use some sense of a timer. Or perhaps some other magic
socket mechanism I'm not familiar with. At that point it sounds to me
like TCP is still a better fit.

> Yes, if the data-check mismatches during this operation, you've got to
> start asking all over again, but I think that's a fundamental problem to
> asking for a large value over UDP that might require re-requests (as
> opposed to TCP handling the re-transmit for you).

:)

> Thoughts? Comments?

I'm amazed and thankful at your work on the protocol documents so far!
I'm using it to write a demo application to prove to myself whether all
of the mechanics we discussed are as they should be. Should hopefully
finish tomorrow soon :)

I'm just thinking on this UDP thing and having way too many flashbacks
to horrific online game protocols and bizarre local network protocols
(nfs, et all).

-Dormando