How do you plan on shutdown?
In ZeroMQ for example, I often setup different threads to handle various sockets. The threads will just listen for data and process it (think something like a worker thread). In ZeroMQ, we currently have a context.term(). This interrupts any threads or pollers to throw a ZMQException (in Java). I can then catch the exception and call socket.close() in each thread and close out the loops.
A possible solution is for the socket.close() to be thread-safe. I can then close the sockets from my main thread.