Website of Martin Sústrik, creator of ZeroMQ, nanomsg, libdill, libmill and ribosome.

Feuilletons about software design, economics, complex systems and psychology of programming.

All the opinions stated here are my own. Any resemblance to opinions of other people, living or dead, is purely coincidental.

Public Key Encryption for Kids

While everybody is using public key encryption, it's not at all obvious how it works. The whole idea is counter-intuitive. Everyone can encrypt a message but only one person can decrypt it? Really? Are you kidding me? However, when you try to gain some confidence in the system if for nothing else, just to make sure that it's not a hoax played on you by the cryptographers you are pointed to scientific papers: The system works like this: [complex math]. The system is secure because: [more...

Comments: 0

Backdoors in Encryption Standards and How To Fight Them

In the last week there a lot of discussion of malevolent parties putting backdoors into encryption standards and how to prevent it. There have been a long thread at IETF mailing list about the topic (starting here). NIST re-opened one of its standards due to worries that the backdoor was built in by NSA (see here). And so on. It may happen that security agency walks into a standards body and demands that the backdoor is built in. That is a rather trivial case and it can be handled by adjusting...

Comments: 0

nanomsg: Towards Full-Blown Configuration Management

There have been no administrative functionality in ZeroMQ, nor there is one in nanomsg. Adding such functionality is currently being discussed on nanomsg mailing list and I feel that it may prove confusing for those not directly involved in the discussion. This blog post thus tries to introduce the basic idea and the use cases without digging too deep into the implementation details. When you are connecting two components via nanomsg, you typically do it something like this: nn_connect (s, ...

Comments: 21

Event-driven architecture, state machines et al.

In my previous blog post I've described the problems with callback-based architectures and hinted that the solution may be replacing the callbacks by events and state machines. In this post I would like to discuss the proposed solution in more detail. Specifically, I am going to define what the events and state machines actually are and explain why they are useful. While the article may be used as an intro to nanomsg's internal architecture it can be also be though of as an opinion piece of...

Comments: 20

The Callback Hell

I've spent last month re-writing the code of nanomsg to use state machines internally, passing asynchronous events around instead of using random callbacks between the components. The change is complex, requires a lot of work and it is not visible to the end user, so the question is: Why do it at all? The time can be put to better use implementing sexy new features that would make the users happy. Instead, the progress on the library seems stalled and such major re-write may even result in...

Comments: 58

Getting Rid of ZeroMQ-style Contexts

I've argued several times (see for example here) that global state in a library should not be truly global, i.e. stored in C-style global variables, rather an instance of the global state should be created upon user request. The reason is that if the two modules within a process used the same low-level library, they would step on each other's toes: Imagine the library C has a global variable foo and setfoo() and getfoo() functions to access it. Now imagine the following sequence of events: ...

Comments: 8

TCP and heartbeats

Heartbeating is a common technique to check whether network connection is alive. The idea is that each end of the connection sends small packet od data called hearetbeat once in a while. If the peer doesn't receive a heartbeat for some time (typically a multiply of the interval between the heartbeats), the connection is considered broken. Interestingly, TCP protocol doesn't provide heartbeats (there are optional keep-alives that are operating on scale of hours, but these are not really useful...

Comments: 27

How to Write a Language Binding for nanomsg

FoldUnfold Table of Contents Reusing ZeroMQ bindings Forking ZeroMQ bindings Writing a new binding Plug-ins vs. native code Plug-ins Native code Run-time retrieval of constants Managing socket lifetime Sending and receiving messages Zero-copy Error codes Handling EINTR Polling Modularity Language bindings are pieces of infrastructure that allow nanomsg to be used from different programming languages. This article is meant to provide some hints that may be helpful for binding developers....

Comments: 0

Using Survey Protocol for High Availability

I've already wrote about survey scalability protocol in this blog. The article explained how the survey protocol can be used to collect information from a set of computers. This article, on the other hand, shows how to use survey protocol to combine high reliability with low latency. First, let's have a look how REQ/REP protocol handles reliability. The basic idea is that there are multiple instances of the service, so if one of them fails, others are still available for processing requests....

Comments: 0

Optimising Subscriptions in nanomsg

When I was writing ZeroMQ's subscription subsystem, my assumption was that there will be thousands or in the worst case tens of thousands subscriptions at any single time. The assumption reflected my background in financial services, where subscriptions are mostly used for subscribing for stock quotes. The topic you subscribe to is the name of the stock and there are, typically, some tens of thousands of those, even if you take into account derivatives like futures and options. However, it...

Comments: 8

Messaging & Multiplexing

I've originally written this article in September 2011. Given that I've had several discussions about multiplexing inside of messaging solutions in past few days, I've decided to re-publish a slightly improved version. Introduction Some messaging technologies (e.g. AMQP) allow for multiplexing several data streams on top of a single TCP connection. Distributed messaging systems such as ZeroMQ which assume lot of small independent services running on a single box or even inside a single process...

Comments: 0

Bus Messaging Pattern

The idea underlying bus messaging pattern is to provide the semantics similar to those of the hardware bus everyone connected to the bus gets any data sent to the bus just to do so on the higher layer. So, to use messaging terminology, everyone connected to the message bus gets any message sent to the bus. This pattern doesn't scale well at some point the number of applications connected to the bus will grow to the point where they will be overloaded by sheer amount of messages they...

Comments: 6

Implementing Network Protocols in User Space

Implementing network protocols in user space is not as uncommon as it may seem. The advantages when compared to implementing them in the kernel space range from easier development, more freedom to experiment and greater portability to less hassle with the distribution (the pain of getting the patch into the mainline kernel vs. simply uploading a user space library to GitHub), improved performance thanks to kernel by-pass and no way to provide a kernel space implementation for proprietary...

Comments: 20

Why Communication Infrastructure Should Use Permissive Licenses

In past days I was challenged couple of times about my decision to publish nanomsg under MIT/X11 license. Other folks, on the other hand, were extremely happy about the license. So, I written this article to give a short explanation of the licensing choice. First, there are two different meanings of freedom in software world. No, I don't mean free as in speech vs. free as in beer . What I mean is freedom as in freedom to run, inspect the code, redistribute and modify vs. freedom as in ...

Comments: 9

Priotitised Load Balancing with nanomsg

Load balancing is one of the typical features of messaging systems. Although some don't do it (MQTT), most of them provide some way to spread a workload among cluster of boxes. In ZeroMQ load balancing is done via REQ (requests that require replies) or, alternatively, by PUSH socket (requests that don't require replies). When designing it, I've opted for completely fair load-balancer. What it means is that if there are two peers able to process requests, first request goes to the first one,...

Comments: 0

Location-independent Addressing with Messaging Middleware

When I'm thinking about messaging, usually I assume is that its goal is to deliver nicely packaged tools for implementing different clever scaling algorithms. An example may be pub-sub pattern (or JMS topic) to do one-to-many data distribution, or req-rep pattern (or JMS queue) to implement a service provided by a cluster of stateless workers. However, messaging is quite often used in a very different manner. Instead of providing a distributed algorithm among N processes, it is used for a simple...

Comments: 5

EINTR and What It Is Good For

EINTR is one of the POSIX errors that you can get from different blocking functions (send, recv, poll, sem_wait etc.) The POSIX standard unhelpfully describes it as Interrupted function. Googling for EINTR returns mainly random questions like I am getting teh EINTR error. What now? answered mostly by Just restart the interrupted function. None of this helps much when you want to correctly handle EINTR, actually understand what you are doing and why. In this blog post I'll try to explain...

Comments: 28

The Merit of AMQP (part I)

Couple of AMQP-related blog posts have arrived lately. Reading it, I've realised that although I've been involved with AMQP from the very beginning I've never explicitly expressed my thoughts on it, on the split between AMQP/0-9-1 and AMQP/1-0 and related technical issues. Also, I believe that designing ZeroMQ gave me a pretty unique point of view on AMQP, that some of the readers may find interesting to explore. 8 Years in Making To start with, the goal of AMQP was breaking the IBM and TIBCO...

Comments: 13

Getting Rid of Domain Registrars Using Bitcoin Algorithm

Recently I've stumbled upon an though-provoking article here. What it says is that whatever the benefits of Bitcoin as a distributed currency, the biggest deal is that it provided us with a viable algorithm for reaching consensus in highly distributed global-scale systems. I would also say that actually proving that the algorithm works on global scale in real world is a great and almost unprecedented feat. The article goes on to hint that the Bitcoin algorithm may be used to fully decentralise...

Comments: 6

Enclave Pattern

In my last blog post I've ranted about problems with implementing intrusive containers in C++. The main problem is that to allow an object to be included into an intrusive container you have to modify the object itself, for example, add 'prev' and 'next' member variables to it. That in turn breaks the encapsulation principle. In rigorous object-oriented design the item is owned by the container, and thus the container should be aware of the item but not vice versa. There have been a lot of...

Comments: 38

Website powered by Wikidot.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License