250bpm

There's no problem with read and write themselves. They are fully implemented by the library and can do whatever the library implementer wants them to do.

The problem is with poll/select. These functions are global, not protocol-specific, i.e. they span across all the possible protocols: TCP, UDP, SCTP, myproto, yourproto etc.

What that means is that you, as the library implementer, have no control of how they work. Still, you need your protocol implementation to be pollable.

To do that you need a native file descriptor (poll/select won't accept anything else.)

The only ways to create file descriptors in user space is pipe, socketpair and eventfd (Linux-only). All of them suffer of the same problems:

1. There's no way to signal !POLLIN & !POLLOUT
2. There's no way to signal special conditions, such as POLLHUP or POLLERR

Yet one more try:

Imagine that you protocol implementation needs to both read and write more. Thus, the first function in your example returns MYPROTO_WANT_READ|MYPROTO_WANT_WRITE. The user retreives the file descriptor (which is pipe under the cover) and polls for POLLIN|POLLOUT.

However, there's no way to signal !POLLIN && !POLLOUT on the pipe. So, at least one of the two events must be signaled at any time. Thus, the poll(POLLIN|POLLOUT) is never going to block!

Yes, but this still doesn't explain why you need to signal !POLLIN & !POLLOUT in the first place.

In other words my question is this: Why isn't enough to have POLLIN mean "please call my library as soon as you can" and !POLLIN mean "chill"? Then after the user calls your library you can return all sorts of messages to the caller, like "EGAIN" or "you've got a new priority message" or "the other end hung up unexpectedly" etc.

Well, that's what everybody is doing. I've implemented that several times myself…

So it starts with this:

struct myproto_socket {
int in_pipe [2];
int out_pipe [2];
int err_pipe [2];
int hup_pipe [2];
…
};

One problem with that is the amount of fds used. Users start to hit the system fd limit pretty quickly.

So you change to implementation to use just one pipe and set of functions to test readability/writeability/error/hangup etc. That's rather weird API, but users are mostly still able to cope with it.

Then you start hitting performance limits, so you start using the pipe in edge-triggered mode rather than level triggered mode. So, the users have to cope with edge triggered mode. At this point most users are profoundly confused.

I can point out many discussions in ZeroMQ mailing list where people are confused about this API. I think I just saw one such email yesterday.

I see. So basically the idea is to cram 3 or 4 different kinds of events into a single fd instead of 1. I just don't see why this code:

void my_read_cb(int fd, void *data)
{
  struct myproto *s = data;
  /* handle POLLIN */
}

void my_write_cb(int fd, void *data)
{
  struct myproto *s = data;
  /* handle POLLOUT */
}

void my_err_cb(int fd, void *data)
{
  struct myproto *s = data;
  /* handle POLLERR */
}

is much easier than this

void my_cb(int fd, void *data)
{
  struct myproto *s = data;

  while (1) {
    switch (myproto_what(s)) {
    case MYPROTO_READ:
      /* handle POLLIN */
      break;

    case MYPROTO_WRITE:
      /* handle POLLOUT */
      break;

    case MYPROTO_ERR:
      /* handle POLLERR */
      break;

    case MYPROTO_EAGAIN:
      return;
    }
  }
}

And surely if you have performance problems with the last bit of code, you'll have the same problems with the first bit. It'll just be the eventloop's job fan out the codepaths depending on the event flags rather than the switch. Right?

Also the second version can easily be extended to protocols that have more interesting events than the TCP/stream derived POLLIN, POLLERR, POLLPRI and POLLHUP.

First of all, this post is about *API* problem. I am not arguing the thing cannot be done with pipe/socketpair/eventfd. What I am saying is that when doing so the API is convoluted and confusing. Check ZeroMQ mailing list for messages from confused users using the API.

What I want to achieve is to enable user of the library to write the following code:

int s = myproto_socket ();
struct pollfd fd = {s, POLLIN, 0};
poll (&fd, 1, -1);
assert (fd.revents & POLLIN);
...

Note the the above is pure POSIX, something that everyone is familiar with.

To put it in a different way: Thanks to this patch, user-space implementations of network protocols can expose exactly the same API to the user as kernel-space implementations.

" On the other hand I just want working software, now, on existing systems."

I have the working solution now. No problem with that. The problem is that it's ugly and confusing for the end users. That's what I am trying to solve.

"Yes, you need a kernel patch - on every operating system you want to support, and every user would need to have it."

Exactly. Linux seems to be a good starting point as it is rather widely deployed.

"For that you would need a higher I/O level interface instead of file descriptors, since file descriptors are OS-specific."

Why so? File descriptors are defined in POSIX. (Unless you are speaking of Windows, but nothing works as expected on Windows, so there's little point in caring.)

"For example take NSS/NSPR - you can use the NSS library to do SSL on top of absolutely anything as long as you can make it appear like TCP (reliable stream), in non-blocking mode (I've done it, it works)."

AFAICS the only way to make user-space implementation of the protocol behave exactly like TCP is to use the kernel patch. OK, there is still another option: Re-implement the whole BSD socket API in a library and overload the system functions by linking with library. Some products do that (SDP, OpenOnload) but it's kind of a brute-force approach.

Why so? File descriptors are defined in POSIX.

Yes, I have systems other than POSIX in mind. Including Windows, and systems with no operating systems at all like microcontrollers. A library that can't work on top of abstract interfaces is useless in such cases.

AFAICS the only way to make user-space implementation of the protocol behave exactly like TCP

It isn't the point to make it appear *exactly* like TCP. After all SSL isn't defined to work on top of TCP. From RFC 5246: "At the lowest level, layered on top of some reliable transport protocol (e.g., TCP), is the TLS Record Protocol."

Kernel implementations of protocols utilizing such abstract libraries also come to mind.

Ah, OK, I see.

Anyway, we've gone pretty far away from the topic of the original article. It was only about slighlty extending POSIX to make implementing protocols in user space easier. Implementing protocols in non-POSIX environment is a different, although extremely interesting, topic.

I think we are not on the same line here.

Specifics being put aside, the problem discussed is about generic interface for communication between different protocol implementations and different applications. It's a standardisation problem. It only works when all parties agree on the same interface.

I've opted for file descriptors because that's what everybody uses anyway and enhanced it to support one up to now unsupported corner case (user-space to user-space signaling). You can, of course, go for a different interface, but the more obscure it is, the less useful it will be.