The patch can be found at LKML (https://lkml.org/lkml/2013/2/8/67). I has not yet been merged in the mainline kernel as I don't have much time to actively push it. If you want to help with that, it would be great.

So yes, at the moment you have to apply the patch by hand and build the kernel yourself.

The patch was developed with 3.x kernel, but I guess backporting it to 2.6.38 should be easy, maybe even with no work needed, except for applying the patch.

No, nothing special has to be done with .config

Send, setsockopt etc. are not covered by the patch as you can implement those in user space easily with no special support from the kernel: just use geonetworking_send(), geonetworking_setsockopt() etc.

Actually I'm trying to implement a network protocol Geonetworkin in the user-space level and to make it working over the 802.11 Mac layer. I found that your post can be very helpful. still I want to know;
1- If the patch " EFD_MASK patch to Linux kernel" has to be installed first.
2- Is the patch compatible with the linux-2.6.38 kernel.
3- If yes, are there any recommended parameters to enable in .config file before compiling the kernel.
4- If possible to provide me with the remaining functions not implemented in this post (send, setsockopt, connect, bind ).
5- small description how the send() and receive() will communicate with the low level handlers of the sk_buff structure in PHY and MAC level.
6- Finally, It will be very helpful if you have an already implemented example you can give.

Thanks a lot.

Anyway, we've gone pretty far away from the topic of the original article. It was only about slighlty extending POSIX to make implementing protocols in user space easier. Implementing protocols in non-POSIX environment is a different, although extremely interesting, topic.

Why so? File descriptors are defined in POSIX.

Yes, I have systems other than POSIX in mind. Including Windows, and systems with no operating systems at all like microcontrollers. A library that can't work on top of abstract interfaces is useless in such cases.

AFAICS the only way to make user-space implementation of the protocol behave exactly like TCP

It isn't the point to make it appear *exactly* like TCP. After all SSL isn't defined to work on top of TCP. From RFC 5246: "At the lowest level, layered on top of some reliable transport protocol (e.g., TCP), is the TLS Record Protocol."

What I want to achieve is to enable user of the library to write the following code:

 int s = myproto_socket (); struct pollfd fd = {s, POLLIN, 0}; poll (&fd, 1, -1); assert (fd.revents & POLLIN); ... 

Note the the above is pure POSIX, something that everyone is familiar with.

To put it in a different way: Thanks to this patch, user-space implementations of network protocols can expose exactly the same API to the user as kernel-space implementations.

 void my_read_cb(int fd, void *data) { struct myproto *s = data; /* handle POLLIN */ } void my_write_cb(int fd, void *data) { struct myproto *s = data; /* handle POLLOUT */ } void my_err_cb(int fd, void *data) { struct myproto *s = data; /* handle POLLERR */ } 

is much easier than this

 void my_cb(int fd, void *data) { struct myproto *s = data; while (1) { switch (myproto_what(s)) { case MYPROTO_READ: /* handle POLLIN */ break; case MYPROTO_WRITE: /* handle POLLOUT */ break; case MYPROTO_ERR: /* handle POLLERR */ break; case MYPROTO_EAGAIN: return; } } } 

And surely if you have performance problems with the last bit of code, you'll have the same problems with the first bit. It'll just be the eventloop's job fan out the codepaths depending on the event flags rather than the switch. Right?

Also the second version can easily be extended to protocols that have more interesting events than the TCP/stream derived POLLIN, POLLERR, POLLPRI and POLLHUP.

Specifics being put aside, the problem discussed is about generic interface for communication between different protocol implementations and different applications. It's a standardisation problem. It only works when all parties agree on the same interface.

I've opted for file descriptors because that's what everybody uses anyway and enhanced it to support one up to now unsupported corner case (user-space to user-space signaling). You can, of course, go for a different interface, but the more obscure it is, the less useful it will be.

I have the working solution now. No problem with that. The problem is that it's ugly and confusing for the end users. That's what I am trying to solve.

"Yes, you need a kernel patch - on every operating system you want to support, and every user would need to have it."

Exactly. Linux seems to be a good starting point as it is rather widely deployed.

"For that you would need a higher I/O level interface instead of file descriptors, since file descriptors are OS-specific."

Why so? File descriptors are defined in POSIX. (Unless you are speaking of Windows, but nothing works as expected on Windows, so there's little point in caring.)

"For example take NSS/NSPR - you can use the NSS library to do SSL on top of absolutely anything as long as you can make it appear like TCP (reliable stream), in non-blocking mode (I've done it, it works)."

AFAICS the only way to make user-space implementation of the protocol behave exactly like TCP is to use the kernel patch. OK, there is still another option: Re-implement the whole BSD socket API in a library and overload the system functions by linking with library. Some products do that (SDP, OpenOnload) but it's kind of a brute-force approach.

The only problem is that for it to be useful it has to be standardised and widely adopted.

I strongly disagree with this. Sure, you need to implement the interface - but this is actually pretty easy as long as your program is not broken by design. If you use an existing event loop like libevent, implementing this interface is literally trivial.

Having such an interface standardized sure would be nice, but any interface is much better than no interface at all. It could be the difference between using an existing library by writing a hundred or so lines of glue code, and reinventing whatever you need just because the developers of the library didn't have such abstract usage in mind.

In the end, if you want your library to be completely portable, i.e. independent of any particular operating system and using only the C/C++ language, such user-implementable interfaces are the only way. For that you would need a higher I/O level interface instead of file descriptors, since file descriptors are OS-specific.

Some real libraries use such interfaces. For example take NSS/NSPR - you can use the NSS library to do SSL on top of absolutely anything as long as you can make it appear like TCP (reliable stream), in non-blocking mode (I've done it, it works).

So it starts with this:

struct myproto_socket {
int in_pipe [2];
int out_pipe [2];
int err_pipe [2];
int hup_pipe [2];
…
};

One problem with that is the amount of fds used. Users start to hit the system fd limit pretty quickly.

So you change to implementation to use just one pipe and set of functions to test readability/writeability/error/hangup etc. That's rather weird API, but users are mostly still able to cope with it.

Then you start hitting performance limits, so you start using the pipe in edge-triggered mode rather than level triggered mode. So, the users have to cope with edge triggered mode. At this point most users are profoundly confused.

I can point out many discussions in ZeroMQ mailing list where people are confused about this API. I think I just saw one such email yesterday.

In other words my question is this: Why isn't enough to have POLLIN mean "please call my library as soon as you can" and !POLLIN mean "chill"? Then after the user calls your library you can return all sorts of messages to the caller, like "EGAIN" or "you've got a new priority message" or "the other end hung up unexpectedly" etc.

Imagine that you protocol implementation needs to both read and write more. Thus, the first function in your example returns MYPROTO_WANT_READ|MYPROTO_WANT_WRITE. The user retreives the file descriptor (which is pipe under the cover) and polls for POLLIN|POLLOUT.

However, there's no way to signal !POLLIN && !POLLOUT on the pipe. So, at least one of the two events must be signaled at any time. Thus, the poll(POLLIN|POLLOUT) is never going to block!

The problem is with poll/select. These functions are global, not protocol-specific, i.e. they span across all the possible protocols: TCP, UDP, SCTP, myproto, yourproto etc.

What that means is that you, as the library implementer, have no control of how they work. Still, you need your protocol implementation to be pollable.

To do that you need a native file descriptor (poll/select won't accept anything else.)

The only ways to create file descriptors in user space is pipe, socketpair and eventfd (Linux-only). All of them suffer of the same problems:

1. There's no way to signal !POLLIN & !POLLOUT
2. There's no way to signal special conditions, such as POLLHUP or POLLERR

That's just another case in the switch above.

Anyway, we have one such standardised and widely adopted abstraction called "file descriptors". They work perfectly OK and there's no technical need to replace them by a different standard. There's only a little piece missing, namely the ability to create fully functional file descriptors in user space. My kernel patch solves this problem.

 #define EVENT_READ (1 << 0) #define EVENT_WRITE (1 << 1) struct MyLibEventLoop { /** * Registers a new file descriptor for notification of the given events. * When events are detected, implemented calls handler. The returned value * is an abstract handle for use in modify_fd and remove_fd. */ void * (*add_fd) (struct MyLibEventLoop *el, int fd, int events, void (*handler) (void *arg, int events), void *arg); /** * Modifies the notification events for a file descriptor registered using add_fd. */ void (*modify_fd) (struct MyLibEventLoop *el, void *fd_handle, int events); /** * Unregisters a file descriptor registered using add_fd. */ void (*remove_fd) (struct MyLibEventLoop *el, void *fd_handle); /** * Registers a timer. When it expires, the handler is called, and at this point * the timer is automatically removed. The timer can additionally be removed * before it has expired using remove_timer. Returns an abstract handle for use * in remove_timer. */ void * (*add_timer) (struct MyLibEventLoop *el, int milliseconds, void (*handler) (void *arg), void *arg); /** * Removes a running timer added using add_timer. After this is called it is guaranteed * that the timer's handler will not be called. */ void (*remove_timer) (struct MyLibEventLoop *el, void *timer_handle); }; 

Then make everything in your library do I/O via this abstract API. On the other side, the user of the library is free to implement the API however he wishes, be it an existing event loop library (libevent, libev, glib, Qt) or his own creation.

Well, exactly. However, there's no way for pipe or eventfd to signal !POLLIN & !POLLOUT.

Also there's no way to signal POLLERR, POLLHUP, POLLPRIO etc.

That's what the kernel patch allows you to do.

For non-blocking I/O you basically do this:

 ssize_t ret = myproto_recv(s, buf, BUFLEN)); if (ret >= 0) { /* yay! everything went well. now do something with the first ret bytes of buf, 0 means connection closed. */ return; } switch (ret) { case MYPROTO_WANT_READ: wait_for(myproto_getfd(s), POLLIN); break; case MYPROTO_WANT_WRITE: wait_for(myproto_getfd(s), POLLOUT); break; default: /* ohh noes! a protocol error occured */ break; } 

Once you detect POLLIN or POLLOUT you just run the code above again. This isn't much different from using raw sockets in non-blocking mode.

Also this should be doable in userspace. If your myproto implementation uses threads in the background, you just return a pipe or an eventfd filedescriptor that you control the other end of.
For blocking I/O it should be even simpler since myproto_recv() will only return
something positive or a protocol error.