Martin, thanks for this. An interesting perspective - I never thought of Exchange-Binding-Queue as a leakage of the internal broker design. It makes sense when I remember how we built OpenAMQ.
You mention two recent articles on AMQP, I presume one of them is this one by @kellabyte, what's the other one?
There was @kellabytes' article and one by Pieter in past few days.
One thing about the exchange-binding-queue refactor is that it let us plug in arbitrary routing logic as exchanges, which did prove useful. There have been quite a few different exchanges, and this model also made it trivial (in the sense of taking several years to find the simple answer) to do pattern-independent federation, which was impossible with the JMS semantics, and missing from the 0MQ patterns.
Yes. As I said, the model made sense back then. The amount of different use cases just asked for some flexible model. It's pretty easy to spot that something is not flexible enough. Spotting that it is too flexible can take years.
This is an excellent piece. My only criticism is that you speculate about motivations. This does not add to your analysis.
True. It's a conspiration theory and thus worthless. Unfortunately, that's the best I have. Loosening the standard to accommodate vendors (not necessarily Microsoft) seems to be the only plausible explanation of what happened. Also, I don't believe people actually conspired to loosen the standard. It probably just followed from social dynamics of the working group.
By the way, the described scenario is nothing unheard of in standards development. In fact, it exists, to some extent, in any standardisation effort. See, for example, here: http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/ The only aspect where AMQP stands out is the scope of the phenomenon (dropping the whole semi-established standard).
I'd be curious to hear an alternative explanation though.
As a JMS vendor I can tell you that you're wrong with your statement that 0.9.1 is 1:1 JMS. 0.9.1 is quite incompatible with JMS and - beside that it is a version looking quite amateurish and lacking many features like exactly-once delivery, flow control and ack windowing - was the reason why existing messaging systems like those from Microsoft and JMS vendors did not implement it. This has changed with AMQP 1.0. We have now 2 JMS implementations (SwiftMQ and ActiveMQ) implementing it, we have a JMS-compliant client from Qpid (which passed the JMS TCK running against SwiftMQ and Qpid Java Broker over AMQP) and others will follow. I'm very confident now since the Department of Homeland Security joined the club. If you want to do business with them in future, be sure to provide AMQP 1.0, which, by the way, should not be a problem as we will have Proton which is a AMQP 1.0 toolkit which makes it a snap to add AMQP 1.0 support to anything in any popular language.
I don't know whether Microsoft was the reason to abandon the 0.9.1 model but you may be right here. Market power is what is important and if Microsoft initiated the move then I can only pass a big thanks to them. I don't think, however, that AMQP 1.0 will be changed that much in future. It is an OASIS Standard and it took about a year for the transition from the AMQP Working Group release with only editorial changes in the spec (no functional or protocol changes). Next step is the ISO Standard so every new release has to go through these stages. But even a change to make a protocol better is no problem for a versioned protocol.
Beside personal-psychological or business reasons to save an existing user base I really don't get why there is such a hate against AMQP 1.0. Don't spend your time with such a negative stuff. Instead be part of the move and just add AMQP 1.0 support to ZeroMQ and your fork. ;-)
@Andreas, I think one of Martin's points was exactly how easy it is to add AMQP/1.0 support to old products like SwiftMQ and ActiveMQ, and how little value that gives those who expected interoperability. You kind of underscored that argument.
I was not trying to be negative, just to summarise the history of AMQP as I remember it. Anyway, politics is boring. Hopefully, part II will be more interesting as I want to focus on technical aspects of messaging protocols in general and AMQP in particular.
(1) Your assertion that AMQP 1.0 does not address the 'broker model' layer
is false; it does.
The specification defines a 'distribution node' as 'a node that stores
messages for distribution', along with the logical message states
through which messages pass as they are distributed and two
'distribution modes' corresponding to competing and non-competing
consumer patterns.
There is therefore a standardised aproach to the basic pub-sub and
shared queue functionality ('Topics' and 'Queues' in JMS parlance).
(2) "I would say that Exchange-Binding-Queue model is rather a leakage
of internal broker architecture"
It is indeed! That is precisely the reason 1.0 moved away from that
model, to avoid tying the protocol to a particular internal
architecture.
Allowing existing messaging products to support the protocol without
having to undertake a complete rewrite seems to me perfectly
reasonable given the original goal as you describe it.
We are already seeing the fruits of that with new implementations of
AMQP offered or in the pipeline. Greater adoption provides users with
greater choice which is exactly what AMQP is supposed to achieve.
(3) "JMS […] defines a reasonable set of functionality […] the
real challenge of AMQP was to define a wire protocol that could
work as a communication medium between JMS clients and
servers. And early versions of AMQP did just that. The protocol
was basically reverse-engineered from JMS. You could browse
through the specification and map AMQP concepts to JMS concepts in
straightforward 1:1 manner."
This is a misleading picture. Early versions of AMQP were poor in
their support for JMS. The 'basic' class was a very naive translation
of the JMS API into an RPC-ish 'protocol' without sufficient
consideration of the functionality JMS offers. As more fully
functional JMS implementations were developed, the limitations became
clear.
I would argue the central 'concept' in JMS is that of the
'Destination', a symmetric redezvous point to which messages can be
sent and from which they can be received. The mapping of this to the
'distribution node' model is much more straightforward than it was to
the 'Exchange-Binding-Queue' model.
Hi Gordon!
Well, yes, there's the 'distribution mode' bit, but calling that a broker model is a bit exaggerated. Same way, ZeroMQ does a simple message delimitation on top of TCP, but I am not calling it a 'connection layer' just because of that.
And yup, early versions of AMQP had many deficiencies. However, it's not clear how AMQP/1-0, by stressing the connection layer and ignoring the broker model does a better job.
I would say that people working on messaging should focus on defining a generic, consistent and scalable broker model, but for some reason nobody is doing that. We are still stuck with JMS model. JMS is 11 years old now and the concepts it standardises are ~20 years old. No progress seems to happen. (Just recall how the Internet changed during that time!)
If you have any idea of how to move the broker model forward into XXI. century, I would love to discuss it with you. I have some ideas myself. I'll try to express them in a paper at some point.
Surely a standardised approach for queues (including browsing) and topics (in the pub-sub sense) *does* count as a 'broker model' under your definition? If not I think you need to more clearly describe the criteria by which you are excluding it.
Fundamental to any 'broker model' is a message abstraction and a standard way for producers and consumers/subscribers to interact with the broker. After all, the purpose of brokering in many cases is to decouple the producers and consumers so the producer shouldn't have to care what the broker does with the messages it sends. (It may of course want to verify that certain capabilities are supported on the node to which it is sending, and a means of doing this both for producers and consumers is important).
On top of this foundation you can then describe the behaviour of different node types (nodes are simply the sources/sinks for messages, i.e. the rendezvous points). For example you want to specify how a broker will allocate messages published to a node that has multiple subscribers - whether the subscribers compete for the message (as in typical queue behaviour) or whether they each get a copy (fanout or pub-sub behaviour). You will want to specify a set of states and transitions for messages through which storage models can be described (e.g. classic store and forward, reliable pub-sub). AMQP 1.0 does all this. In addition you would likely want selective subscriptions, where subscribers to the same node can specify interest in a subset of the messages published to that node (e.g. the wildcard matching in a pre 1.0 'topic exchange' as compared to the 'fanout exchange' or JMS style selectors). And of course there are other aspect of behaviour (nodes that retain the last value for a certain key, nodes that honour message priority when choosing which message to deliver etc).
Now, in my view, not every 'broker' is going to need/want to support all the same behaviours (e.g. with the pre 1.0 AMQP model support for transaction, durability or priorities was optional, as was support for the topic exchange type. Moreover some implementations decided not to support acknowledgements but to specialise in uses cases that didn't require them etc). I don't think the job of the standard is to force them to do so, but rather to provide standardised approaches for and expanding set of behaviours (perhaps aggregated into profiles for easing comparisons).
The 1.0 specification does provide a way to register extensions. This allows a catalogue of existing behaviours to be built up that then provides a good basis for further standardisation efforts, hopefully driven by real needs and working solutions. I suspect the first thrust here will be on the filters required for the JMS mapping. I can also see initiatives launched to standardise some of the existing extensions from ActiveMQ, SwiftMQ, Qpid etc (message groups, last value 'queues', sticky consumers, probably informally through the respective communities to begin with.
Also relevant here - a different dimension of 'broker' behaviour - is global addressing and standards for creating networks for co-operating components.
I'd certainly be very keen to hear your thoughts on what might be needed for a more 'modern' broker model!
(PS: Sorry this is rather a long post! as someone famous once said, I didn't have time to write a shorter one!).
