"Given that TCP doesn't provide a way to read data from the middle of the receive buffer, the application has no idea whether the heartbeats are arriving or not" - isn't TCP feature of "urgent data" solving this problem exactly?
I've never used urgent data personally, but as far as I understand, how it works is that urgent bit is part of the bytestream. So, if data transfer is blocked by backpressure, even urgent data won't get through.
Feel free to correct me though.
Actually I have no idea if urgent data obeys TCP window or not. Seems like a good candidate for experiment :)
Apparently (according to Wikipedia) there are incompatibilities between TCP implementations which effectively limit OOB data to 1 byte. For heartbeat though it is enough. More troubling are network devices (the presentation linked in Wikipedia mentions Cisco PIX) which clear urgent pointer.
In any case, urgent byte was designed to send Ctrl+C over the network and is barely used anymore. I would be pretty concerned if I've seen a protocol using it for heartbeats.
Btw, I recall some horror stories from long ago about using urgent byte as message delimiter :)
Apparently you're right - RFC 6093 contains a thorough discussion on TCP urgent data. I was mislead by SIGURG which indirectly implies that by sending some urgent data it is delivered out-of-band.
Good to know! I had that impression for a long time but never really bothered to check.
As another try - you could decrease TCP keepalives timeouts (on Linux you could set them individually per socket, not globally) and use them as heartbeat mechanism? Don't know if it's portable though.
Nope, it's not. The keep-alive options are Linux-specific.
Also, there's another aspect that I've deliberately ommitted to keep the article easy to understand: If you want adaptable heartbeats that "just work" not depending on the link latency, you need something like SCTP's heartbeating algorithm. TCP keepalives won't do that for you. Even worse, you either need access to TCP's RTO value (no standard way to get it) or re-implement the RTO measurement on top of TCP.
It really gets messy once you start thinking about it in depth.
Apparently I cannot add links here so see gist catwell/5451026 ;)
This kind of problem is not limited to TCP, it's happening everywhere.
In networks, one of the most ridiculous examples IMO is L2TP. I mean, come on, a protocol explicitly designed to make loops in the hourglass model… Here the missing functionality (analogous to heartbeat in your example) is tunneling. The right solution to this problem would be something like MPLS, which despite a bad reputation has the right design: adding a layer (2.5). And the solution people use instead is… encapsulating layer 2 protocols inside IP!
Another example is mobile OSs. The problem is distribution, the Web solves it, so let's re-implement a whole OS in the browser, right? Encapsulate applications in a stack that was designed for online documents by twisting it as much as we can and pretend JavaScript is an assembly language. Mozilla criticized Microsoft because they could not implement a IE competitor efficiently on Windows 8. I wonder how efficient alternative browsers for Firefox OS will be without a way to write native code.
As you said all this happens because people who use a technology see its shortcomings and want to solve them, but they don't understand the technology and its design enough to change it, or they don't have enough leverage to make it happen. They know how to use it though, so they hack on top of it.
This is why we keep reinventing square wheels since the 70s. By layering stuff on top of flat tires instead of replacing them.
The question is: Do we have an alternative?
If you stick to the right thing (TM) you'll go the way of SCTP — you'll create an obscure niche solution.
Not accepting the political reality is invariably turns out to be lousy solution in the end. So, what's needed IMO is hacking simultaneously on the technical and political level. It's not at all obvious how to do that, but it's definitely a challenge.
I agree. For instance, I criticize things like Firefox OS on a technical level, but I still think it is very interesting because it solves the distribution problem. In an ugly way, but it does.
Regarding SCTP, the problem is even worse than that: now for lots of things we can't use TCP either because incompetent system administrators (or rather their clueless bosses) decide to only leave port 80 open. So we end up with HTTP as a transport protocol and applicative firewalls.
How can we change that? I agree that not accepting the political reality is not the solution for us hackers, and we have to do with what we have. But I think large players could, if they were bold enough. For instance, how could we solve the IPv6 migration problem fast? Deprecate IPv4 for all Google services with a 3 months deadline. A.k.a. solution "a dancing turtle is not enough incentive".
I am impatient to see what will happen with HTTP 2 / SPDY and maybe QUIC (Google's UDP replacement). Maybe they will turn to TCP next? :)
My feeling about it is that we should simply accept the process of protocol fossilisation and gradual movement up the stack. It seems to be a fact of life and something you cannot really fight.
However, when you look at the process closer, as layers are added on top, the waist of the hourglass seems to move upwards as well. Once it was at IP level, now it's almost on TCP level and heading upward towards HTTP.
The nice side effect is that any technology that gets *below* the waist (and if we accept the movement-upward model, every technology will ultimately get there) can be relatively easily replaced. Once, replacing L2 layer was a complex problem. Today, nobody really worries about replacing 1GbE by 10GbE although the two are pretty different beasts. The reason is that L2 is below the waist of the hourglass. You just don't have to care.
One day, TCP, HTTP and WebSockets will get below the waist as well. Then we can replace them with something that fixes the problems.
Of course, the time horizon here is measured in decades rather than years.
Interesting. Now the problem is that the upper layers have not been designed to do what they do, so it still makes the whole thing more complex. HTML5 is not a good application development platform and HTTP is not a good transport protocol.
But it's still nice to think we will be able to replace some of these things someday because people will standardize above them…
And now apparently DJB is trying to replace TCP+TLS ;) http://cr.yp.to/tcpip/minimalt-20130522.pdf
Hi Martin, thanks for the great article.
I have used the heartbeating technique for some time over both WebSockets and ZeroMQ. While the technique works well in both cases, it's far more pleasant over WebSockets since that protocol includes frame types for ping and pong (as well as allowing control frames, e.g. ping, to be interleaved between the many data frames of a WebSocket message).
Are you considering adding better support for heartbeating to nanomsg? It always feels like a hack when I add this to protocols built on top of ZMQ.
Hi Martyn,
The problem, as explained in the article, is that to implement *sane* heartbeats on top of TCP you have to re-implement a large part of TCP functionality. Which is a task that would require considerable amount of work.
Also, the system often used with ZeroMQ — passing heartbeats on a separate connection — seems to work quite well and makes investing in TCP re-implementation questionable.
One thing that's missing in ZeroMQ though, is the ability to funnel all the communication through a single open port. That's definitely on the roadmap for nanomsg.
As for websockets, I've tried to participate in the stadardisation with insights like those in this article, but I haven't had much time back then. My feeling at the time was that while the spec defined wire format for heartbeats and multiplexing, it was kind of short on specifying the actual semantics. Anyway, glad to hear it works for you.
I believe I mentioned this in #nanomsg, but this may be of interest to anyone who wants to learn more about the ossification of the transport layer: [Argh, it won't let me post links. Look up the DeDiS group at Yale, project 'Tng']
They suggest fixing what they term the 'transport logjam' by further subdividing the transport layer into:
Semantics (flow control, in-order vs out-of-order, e2e reliability) ["I want TCP/SCTP/etc"] over
Isolation (encryption, datagram integrity) [DTLS or similar] over
Flow (congestion control) [DCCP sans ports] over
Endpoint (ports) [UDP or similar]
That lets NAT deal with the endpoint layer, flow middleboxes like wireless optimizers deal with the flow layer, and everybody stays out of your transport semantics.
Isn't it kind of a chicken-and-egg problem? To beat the transport logjam they would have to beat the transport logjam to introduce new fine-grained layering at L4.
Not really - they use a system they're calling 'minion' in order to bypass that, by using existing protocols where possible. For instance, using UDP as the endpoint layer (thus it looks like a normal UDP packet) and sticking a portless DCCP on top of it as the flow layer is the trivial example.
One of their more ambitious ones is using COBS (Constant-Overhead Byte Stuffing, a record separation format) and a new sockopt (TCP_UNORDERED) in order to add out-of-order delivery to the TCP *API* without changing the wire protocol. That lets them use it as the endpoint and flow layers.
They took the same tack with TLS, too, so you can have what the network thinks is a normal TLS session but the API for the user is out-of-order unreliable datagrams.
Isn't there a userspace SCTP over UDP solution? That both plays well with firewalls and gives you the advantages of SCTP with just a small amount of extra overhead.
Oh, and as far as fossilization goes, don't forget SPDY which is basically the subset of SCTP that is useful for HTTP, plus an HTTP specific compression library implemented on top of TCP.
As for SCTP over UDP I though about it myself, but I am not aware of such thing actually being out there. If you have any pointers, I would love to check it out.
WebRTC DataChannels are using SCTP over UDP+DTLS. FWIW, I have a writeup @ hpbn.co/webrtc. A couple of deep-links into the chapter:
http://chimera.labs.oreilly.com/books/1230000000545/ch18.html#_real_time_network_transports
http://chimera.labs.oreilly.com/books/1230000000545/ch18.html#_delivering_application_data_with_sctp
Available in Chrome and Firefox today, which is pretty neat!
The generic SCTP-over-UDP thing would be extremely useful in business messaging scenarios. What's the actual availability of the solution? Is it an actual C library that can be used anywhere? Does it require a daemon to watch for incoming connections etc.?
