Last week I've refactored ØMQ codebase cutting off all the experimental features and making it more backward compatible with older versions. The refactoring resulted in a version, ØMQ/3.1, which was then selected by the community vote as a sanest way forward.
The refactoring meant throwing lot of work I've done in the past months into the trash. Why had it left me with a strange craving for a bottle of champagne and a tin of caviar then?
Well, it proved me wrong in assuming that ØMQ is a new experimental product where innovation is worth more than keeping the product stable. And those who know me know that proving me wrong requires exposing me to LOTS of evidence for a LONG time.
So, I was exposed to my daily dose of evidence for nearly half a year and here we are.
ØMQ is no longer an experimental project. It can no longer be randomly changed without hurting a lot of users. It's a stable product, used for serious business and a lot of people are depending on it. There are trading firms using ØMQ as a secret weapon in low-latency arms race. There's plethora of open-source projects using ØMQ for communication between components. There are HPC products using ØMQ. There's Storm, a real-time processing system based on ØMQ that was acquired by Twitter few months ago. CERN is considering ØMQ as a control infrastructure for Large Hadron Collider. ØMQ talks are popping up at conferences. There's even a whole user-organised conference about ØMQ going to take place in February.
Wow!
It kind of makes me reflect on the way we've been through since 2007…
I recall that night in San Jose, where, jet-lagged and full of tequila, we've — my friend Pieter Hintjens and me — drawn the very first simple diagram of how AMQP can be consoled with IP multicast on a napkin. The resulting whitepaper I've wrote can be still found here. It may seem to be a bit naive in the retrospect, however, for me it's veiled by nimbus of nostalgia. It was probably the first step towards ØMQ of today.
I recall writing the initial business analysis. We've been completely wrong there. We've focused on the financial services. In years to come we've got almost no help from that industry. Instead, ØMQ was adopted and helped with by large and generous open-source community which deserves my personal thanks at this point.
I recall creating a Delaware-based company to take care of the development ("American corporation sounds cool!") Thanks, Pieter, for putting the trust and money to such high-risk project! Also thanks to Peter Hodál, another friend and a lawyer, who took care of the paperwork, often doing so for free.
I recall my friend Peter Hajdin from komplot.sk creating the awesome ØMQ logo (For free! Thanks!) and saying "This logo is designed to be placed on top of the Chrysler Building in case you ever need that."
I recall creating initial benchmarking methodology; speaking to Cyril Václav, a friend who was doing his PhD in statistics at the time, and admitting that I've somehow managed to slip through the CS curriculum without passing a statistics course. Thanks for getting me introduced to the basics!
I recall measuring everything we possibly could, even such trifles as how the latency of HD writes varies with respect to the disk rotation speed. I recall the walls covered by multi-couloured benchmarking graphs.
I recall the hot summer in remote Bulgarian village, almost on the Turkish border, with car broken and no way to get home, where I devised the first version of ØMQ wire protocol. Thanks to the guy who fixed the car! Without him I would have been herding donkeys today. There would be no ØMQ.
I recall the coding. It haven't taken place in a Silicon Valley garage. Rather, it was a Baroque cellar with walled-up passages leading to the crypts of St. Martin's Dome in Bratislava. At the noon of Midsummer's day a single sun ray penetrated the deep yard, first passing the modern architecture on top, then XIX. century one in the middle and finally, for a short while, it cast a little speck of light through the cobwebbed window on the floor of our office. Thanks to Pavol Malošek who endured that and took care of half the coding and all of the infrastructure, including all the work with soldering iron! Thanks to his wife who baked a cake with shiny red sugar ØMQ logo for the first birthday of the project!
I recall the autumn of 2009 when the project was cancelled and I was left with basically dead and unfinished piece of software. I considered taking a full time job and abandoning ØMQ. However, the idea that messaging functionality can be exposed via BSD sockets was nagging at me at the time. I wasn't able to explain it to anybody back then! All I've got was lukewarm "Oh, I see," from everybody I've talked to. Finally, I decided to spend my almost non-existent savings to get ØMQ to the usable state. If I had decided otherwise back then, there would be no ØMQ today. Oh, my!
I recall how few months later I passed my first article about new, socket-based ØMQ to Martin Lucina to correct my distorted English. Martin is a friend — half Slovak and half New Zealander — who used to work with us on AMQP and OpenAMQ. At the time he suffered from a heavy AMQP-inflicted burnout. He had shared the office with us for two years and while working basically side by side, he wasn't willing to look at what we've done. When he passed the article back to me he was convinced. He was the first person to experience ØMQ epiphany!
I recall explaining the concepts of ØMQ to Pieter Hintjens in a worn-out glory of hotel Kyjev in Bratislava, among cheap stickers glued on marble walls and once-expensive furniture now silently falling apart. He had a lot of his hallmark "you are doing it completely wrong, you should do it this way," remarks and then he'd written up the marvellous and entertaining ØMQ guide.
I recall getting first patches from the community. Knowing that people are looking at your code and finding it worthy of patching is a fabulous experience. Special thanks to Steven McCoy who maintains OpenPGM, the library ØMQ uses for multicast, and Mikko Koppanen who took over the daunting task of maintaining ØMQ build system and who runs a build farm somewhere among the lakes of Central Finland.
I recall writing the first simple language bindings all by ourselves. Thanks to Matúš Hámorsky and Tamara Kuštárová, who had done a lot of initial work! Maybe the first independent binding was the Common Lisp one written by Vitaly Mayatskikh, a guy from distant Siberia, where people don't smile because — as they say — only an idiot smiles without a reason. At the time he was on his way through Europe, where people do smile sometimes, to America, where everyone is smiling all the time. Later on, we've handed out maintainership of existing proto-bindings to different members of the community. It proved to be the right move as each of these projects seeded a small community of its own. Few people to mention are Brian Granger and Benjamin Ragan-Kelley who made the Python binding a product as important as core ØMQ itself; Gonzalo Diethelm, a Chilean pipe organ player, whom I met in the jolly town of Amsterdam and who maintains the Java binding; Chuck Remes from Chicago maintaining Ruby binding; unfortunate Serge Aleynikov who created the first Erlang binding and is serving his time in a prison right now. Thanks to all the binding maintainers whom I haven't mentioned by name!
I recall the first person to mention ØMQ in a blog. It was Andy Piper from IBM, a WebSphere MQ and MQTT guy, whom I happened to meet in person last month. More people blogged and lectured about ØMQ later on: Ilya Grigorik, Nicolas Piël, Zed Shaw, Ian Barber and lots of others. Thanks!
Errh. Enough nostalgia! What's next?
Well, I would like to spend much more time on the project of standardisation of distributed messaging protocols (SP). If time permits, we should be able to create an IETF workgroup for it next year.
Maybe I'll give more love to the implementation of distributed messaging system, akin to ØMQ, in Linux kernel.
However, most importantly, I would like to tackle the problem of management of distributed messaging infrastructure. As those who repeatedly (and, alas, in vain!) asked me to add buffer size monitoring functionality to ØMQ know, I've tried to preserve the clean mechanism/policy separation and thus I've treated the problem of management as somehow out of scope of ØMQ project. However, the time is ripe for confronting it. It's no longer the theoretical exercise it used to be two years ago. Distributed messaging, in form of ØMQ, is in wide use today and people are facing the management challenge on a daily basis.
In overall, I believe the future is bright for the project. It's slowly getting into the mainstream. There are almost 1000 subscribers on the mailing list. Commercial opportunities are popping up and there's already a small community of developers offering ØMQ services.
So, while drinking the Champagne and eating the caviar I'm asking myself of four years ago about his opinion. And myself of four years ago replies: "Mission accomplished!"
November 10th, 2011