- Lists can be treated as a separate abstract data type. As long as the client code (e.g., the code that works on my_entry above) respects this data abstraction, there is no encapsulation problem. (Programmers who cannot maintain this discipline should not be programming in C.)
- You eliminate code duplication (currently, you have to write the pointer manipulation code for each kind of object that might be on a list).
- You can change the type of list without having to change the client code (except to change the type of node).

Of course the my_entry code has to know about where its representative list_nodes are, so it's still intrusive in that sense, but that's okay.

This polymorphic linked structure trick is a standard C programming idiom (though known to distressingly few C programmers), and good programming practice.

First, I'm not sure what you mean by bad at resizing. What particularly about it is bad?

Re vector vs list, cache misses are a drastically significantly more important performance impact than anything else these days & you're guaranteed them when using a list unless you do lots of jumping through hoops to allocate everything in a contiguous array (but then why not vector?).

I highly recommend you watch http://channel9.msdn.com/Events/Build/2014/2-661 (the performance stuff starts roughly around the 20 minute mark) & is language agnostic (true for C++/C/Java, etc).

Additionally, in C++11, if you declare your move constructors noexcept (or use the default one or omit destructor, move/copy contructors & move/copy assignment operators), then resizes of vectors can be much faster (if the move constructor is faster than the copy one) since the container doesn't need to copy your objects, it'll just move them.

C++11 lets you also express your non-assignable types without resorting to pointers by declaring the type to be move only (thus Person can live on the stack or moved to live in the heap):

 class Person { public: ~Person(); Person(const Person& o) = delete; Person(Person&& o) noexcept : // initialize member variables with the resources from o { // modify o so that it no longer owns any resources // but leave it in a destructible state } Person& operator=(const Person&) = delete; Person& operator=(Person&& o) noexcept { Person tmp(std::move(o)); swap(*this, tmp); return *this; } private: friend void swap(Person&p1, Person& p2) noexcept; }; 

 class Person { public: Person(); // don't specify destructor or move/copy constructors/assignment operators // & this type is implicitly move-only due to _data being move-only, it's exception-safe // & everything is automatically annotated noexcept for you private: std::unique_ptr<PersonData> _data; }; 

It will not protect against all conditions, especially not a memory leak in another running process, but it could keep things going long enough to print dire warnings to the console, stderr, etc, and optimistically, allow your app to survive long enough to realloc() to restore the size of the "spare" memory pool.

What it will do is give you something useful to do to recover from malloc() when it returns a null - namely realloc() "spare" to something smaller to free up the memory you need. Hopefully.

Conclusion… never use std::list. But to go as backwards as to use C… I think it's taking it a bit far.

The solution given in the aforementioned article is to use so called "expression templates" which wrap the parts of the expression and solve the order of operations in the compile time, reordering them in a way that the compiler can then optimize it further. This compiles to the code which could be hand-crafted by a C programmer, but it requires a real mess of templates to emulate in C++ what I'd call "lazy-evaluated expressions". So it requires a lot of jumping through hoops to hack one's way around the encapsulation and to make the code efficient again, as it would be in C (but retain the type-safety of C++, which cannot be done in C, you need to admit). And I wonder, too, if this is really a flaw of OOP or just its implementation in C++.

Using list I get 24 bytes. I'm not sure where the extra few bytes comes from but that may be solved using a different implementation. Templates/oops doesn't cause extra allocations http://ideone.com/qGp2Am

Anyway, I am all for computing as much as possible in compile-time, however, there always remain couple of pesky little things that must be computed in run-time ;)

If by stressing the multi-paradigm character of C++ you mean that I should write plain old C and compile it with C++ compiler, sure, but then there's this pesky little dependence on libstc++ which gets pretty annoying, especially on embedded platforms.

Please mind that I am not against OO paradigm itself. The idea that people tend to think in terms of objects rather than functions is a profound insight, maybe the deepest one made in the history of programming languages. The problem is that the abstract notion of "object" is in reality downgraded to a simple "continuous chunk of memory" concept. While that works OK in most cases, there are cases where the abstraction leaks. For example: intrusive containers. Here a single chunk of data is split between two objects (container & the item). Another example: state machines. People tend to think about states as objects (they have well specified behaviours, each has a set of methods (events) etc. However, all the states in the state machine share the same data. Yuck!

So, I guess, the question is rather whether OO paradigm can be extended to take account of these cases.

Btw, given that you seem interested in various techniques to make things work in C++, have you ever come over some neat trick to implement state machines?

Here is your problem - you are trying to fit round peg into square hole. Despite of what you might have learned, and practised, C++ is not a "object oriented programming language"; from the horse's mouth www.stroustrup.com (slash) oopsla.pdf . It is a multiparadigm programming language. It supports concepts such as decoupling, reusability and interfaces via many different means of which OOP is just one. E.g. standard template library depends on implicit interfaces without inheritance (you know what "begin()" and "end()" do, without having to rely on hypothetical IContainer class), reusability in procedural style works surprisingly well in connection with templates, defining implementation detail classes in a implementation files rather than in headers does decoupling pretty well, selection of code execution paths based on non-type template parameters often gives better results than inheritance etc.

Never mind, you are an old dog now, it would be silly of me to try to teach you new tricks. Just say one thing: I used C++ some 20 years and almost gave up 10 years ago because OOP felt too limiting and clumsy (I've been programming in Lisp before). Then I read "Exceptional C++" series from Herb Sutter and this opened my eyes - there are more tools in C++ tool chest than in any other language, but somehow I've missed large proportion of those!

My conclusion after passing the whole way is that C++ is for rapid development, not for system-level code.

The fundamental problem, I believe, is that not all the problems align nicely with object-oriented model (e.g. state machines, rigorous error handling, low-level optimisation etc.) Of course, you can *not* use OO model in C++, but the it just looks silly. C, on the other hand, being so basic and low-level aligns well with any paradigm you can think of. And, of course, writing OO code in C is easy.

Btw, some interesting research was done in PARC. "Aspect oriented programming" — an attempt to capture various aspects of a program (e.g. object model, failure model, state machines) using different languages and then compose the partial descriptions of the system into a single executable. It's tricky though. However smart the guys in PARC were I think the problem was just too hard in this case.

Sorry, should have made if clear in my post earlier.

Martin, your problem with C++ is not a C++ problem. It is a problem with how certain (majority?) programmers use it. It is also a problem with inefficiencies in C++ standard library (admittedly, part of the language standard-wise, but you can write perfectly valid C++ without it). Back to your example: no-one prevents you from picking and using the parts of C++ which go in hand with the goals of your project, and simply banning (by means of coding standard) parts which you do not like. In case of your list<person> example: you can make your own intrusive list rather than use std::list. Simple as that. This can be done either by means of list defining internal template structure which either inherits from person or keeps it in a non-static member variable (either way, person becomes a subobject of internal implementation detail, thus giving exactly same object layout as your C example); alternatively you may impose that person shall expose two pointers person* next and prev (C++ gives you means to enforce this), this way allowing you to implement list in the same way you would do it in C.

One important thing that C users often forget about C++ is that you *do not have* to use all of its features. You get to use the features which make sense *for your project*, and if this subset excludes things like exceptions, dynamic polymorphism, standard library or whatever else, that's OK. You still get to benefit from things you left in (access control, perhaps? static polymorphism?). If this excludes almost all of C++ , then perhaps you just have to admit that you do not know C++ well enough to use if efficiently? In which case there is little point complaining about its inefficiencies …

However, I believe the problem is inherent: You can improve the code until it is cleanly implemented intrusive list. Then you have parts of the list object leaking into item object (next, prev). You need to ensure that these won't be used by anyone other than the list. So you'll probably go with something like the enclave pattern. At that point you should look on the code and ask: Are all these hacks used to get something that C++ promises out of the box (member access control) really worth it? Wouldn't it be cleaner to use raw C instead?

I thought we were talking about a linked list and that the problem was simply going to be one of choosing which intrusive linked-list implementation best fitted the problem.

This however is a much less well defined problem, but quite interesting.

What is going on in the code now is clearly not ideal.

That array class is crazy, just some thoughts so far, no need to respond:
1) why do we want random access to an array which does not
guarantee order is preserved? certainly it doesn't seem like a priority
that this is O(1)
2) pointers can be invalidated because deletion of objects does not remove them
from the array - this leads to a need to always check the pointers.
3) pointers can be invalidated because we are allowed access to the pointers
by reference
4) the array indexes can be invalidated because set_array_index is public.
5) The comments claim that insertion in O(1) but this is only average time.
Worst case time is bad because push may require new memory allocation and
copying of the array.

An intrusive linked list could actually solve all of these problems, though iteration would obviously be slower and if there is some reason why you need O(1) random access via index then that would be a problem too. Intrusive linked lists do not require any allocations to add existing nodes to a list, for this reason they are good for "real time" applications.

On the broader question of how to design the locking properly I am going to have to study the code more and review the literature in this area. I have a few ideas.

I'm gonna stop myself babbling, I will probably go quiet for a while but I will try to come back to you with something useful.

"that sounds like a job for typestate. Implementing some approximation of this with C++ metaprogramming sounds like a very interesting project (and as I think of it, some new C++11 features, in particular type erasure and variadic templates might help a lot)"

Go for it. It would be interesting to see what C++11 has to offer in terms of decoupling memory layout from accessibility settings.

"I'm sorry I just don't have the context to understand the problems with this sort of example. If you can point me to where I can find the code, which classes in particular are problematic (what corresponds to A B C and D) then I will try to deal with the complexity or admit defeat."

0MQ goes to great lengths to avoid this kind of problem. See the convoluted algorithm for managing list of pipes in lb_t and fq_t classes. The list stores pipe index (position in the list) inside of the pipe, however, pipe is not necessarily living in the same thread as the list. What you get is members that are living in different threads packed without any safeguards into a single class. There are lock-less algorithms involved etc. so it's really top complex system to demostrate theoretical issues on.

The ABCD snippet in the previous post is much cleaner example. The only goal is to ensure that individual variables are accessed only by their owners, i's by A, j's by B. Attempt to access say C::i from B or C should yield a compile-time error.

The only solution I can think of is along the lines of enclave pattern, I've described in my blog. However, that's hardly an idiomatic C++ and leaves me asking whether using plain old C is not a better solution.

"If possible, make compile complain when synchronisation is done incorrectly"
that sounds like a job for typestate. Implementing some approximation of this with C++ metaprogramming sounds like a very interesting project (and as I think of it, some new C++11 features, in particular type erasure and variadic templates might help a lot)

"0MQ code is to convoluted to be discussed here, but try adding proper encapsulation to this"
I'm sorry I just don't have the context to understand the problems with this sort of example. If you can point me to where I can find the code, which classes in particular are problematic (what corresponds to A B C and D) then I will try to deal with the complexity or admit defeat.

 class A { mutex m; /* uses and synchronises C::i and D::i */ }; class B { mutex m; /* uses and synchronises C::j and D::j */ }; class C /* doesn't use C::i or C::j */ { int i; int j; }; class D /* doesn't use D::i or D::j */ { int i; int j; }; 

As for the black magic, casting from member to the container is a standard way of doing things in C. Check e.g. Linux kernel sources.

"I've actually written a blog post about this"
Something like that but the next and previous pointers in the helper class need to point to person objects because you can't get to the person class from the helper class and all you are left with is a linked list of helpers (I guess strictly you could probably do some black magic where you find the offset of the helper data I'm assuming that's not what you wanted to do).

My problem is as follows:

I have many classes in the project, each has many members. I have also few mutexes. Each mutex synchronises some (but not all) members in some classes. Say mutex 1 synchronises A::a, A::b, B::i and C::x, while mutex 2 synchronises A::c and C::y etc.

Now, the challenge is:

1. How to keep track of which mutex synchronises which variables
2. If possible, make compile complain when synchronisation is done incorrectly

"I would say there's no way out, unless the language doesn't enforce tight coupling between objects and memory allocations."
Not necessarily but it is the easiest way of telling the compiler how the data should be laid-out. If you break the coupling between object and memory structure you will end up having to look up every property like Ruby and its dynamic bretheran, and that is a _lot_ slower.

Could you point me to the actual code where this is a problem. Talking in the abstract and with toy examples can end up going 'round in circles.

I wager I can find a C++ solution that will deliver better encapsulation, making your code safer, more expressive and easier to maintain, without compromising runtime performance when compared to the bare C solution you describe. And that even in this case to which you feel C++ is particularly unsuited.

I say this with all humility, knowing I am not an equal to you as a software engineer.

Ok, I guess you are into a more theoretical discussion. Here's my take then:

What OOP delivers is a way to decompose the universe into well-delimited objects. The appeal is that human brain is wired to deal with well-delimited objects and thus OOP allows common sense to kick-in (rather than pseudo-scientific reasoning required by non-object-oriented code).

In other words, if there's a box of chocolates, common sense assumes that these are two well-delimited objects and there's nothing chocolate-like about the box and nothing box-like about the chocolates.

So far so good. There's nothing wrong about OOP paradigm.

Enter C++. C++ have evolved from C and re-used C "struct" construct to define classes. What that means is that "well-delimited object" maps to "continuous chunk of memory" in C++.

This tight coupling between "object" and "memory allocation" works well in most cases, however, in few special cases it goes astray. When thinking about lists, the only way to separate box properties from chocolate properties is to allocate 2 memory chunks instead of a single one per chocolate in the box. That's basically what STL does.

Intrusive containers merge the two chunks into a single one, but then some of the properties of the box (prev,next) leak into chocolate object. Which, of course, violates the basic promise of the OOP paradigm (well-delimited objects).

I would say there's no way out, unless the language doesn't enforce tight coupling between objects and memory allocations.

This is exactly what the library-implemented templated intrusive-list solves (preferably something better thought out than the code I posted, boost probably has one).

The programmer has accepted that his object "is a" node in the list, in that sense that is part of what the object it but he still wants implement and use this in a safe, neat, reusable way.

The point is, you can write an intrusive list that is not intrusive in your client code. Yes you have to inherit from the node class but (my implementation is not full, but Google it someone has probably done it), but the important point is if you were to need to change the design and re-factor your code it would be relatively easy, depending what you are doing of course you will probably only have to touch the class definitions not the client code. It does not "degenerate to C style"

Talking about what "a C++ programmer" would do is really a straw man argument. The point is C++ generics provide the tools to represent any abstraction you might think of and do so without any run-time overhead. Yes the implementation of the patterns can look pretty scary (in large part to the verbose template syntax) but usage usually has acceptable to good syntax.

I have been reading a lot of language specifications and papers on different type systems and programming paradigms recently and the biggest surprise that I have had from the whole project is that it turns out most of the ideas can be implemented in a usable way as a C++ library. Think very carefully before you say something can't be done.

Finally if you have a case where your code is easier to read and maintain and more efficient in C style then do it, it certainly doesn't mean that your whole project has to "degenerate".

Why do I care enough to write this?
answer 1: C++ provides the tools to let you deal with abstractions at compile-time, C doesn't. In C you must therefore either do without the abstraction or use run-time abstractions leading to just the sort of indirection this article seeks to avoid.
answer 2: I'm a geek
answer 3: someone is wrong on the internet ;¬)

PS I have very much enjoyed using ZeroMQ

See, the object needs to know that it will be part of the list:

class important_object: public list_node<important_object>

Find code on codepad (dot) org (slash) MqAloPgJ

and whats more you get list elements that remove themselves when they are deleted resulting in cleaner, safer code.

Find code on codepad (dot) org (slash) MqAloPgJ

and whats more you get list elements that remove themselves when they are deleted resulting in cleaner, safer code.

It is like a proxy for the person class. that adds extra functionality to it.

Do you really believe that C++ requires extra chunk of memory in case of list?

Have a look into STL sources and see something like
==
/ @if maint Common part of a node in the %list. @endif
struct _List_node_base
{
_List_node_base* _M_next; /< Self-explanatory
_List_node_base* _M_prev; ///< Self-explanatory

static void
swap(_List_node_base& __x, _List_node_base& __y);

void
transfer(_List_node_base * const __first,
_List_node_base * const __last);

void
reverse();

void
hook(_List_node_base * const __position);

void
unhook();
};

/ @if maint An actual node in the %list. @endif
template<typename _Tp>
struct _List_node : public _List_node_base
{
_Tp _M_data; /< User's data.
};
==

Helpers do not exist!

It is the same object, it is one memory allocation, it is inheritance and it is C++ magic.

Enjoy C++, it is really cool.

Even if that wasn't the case, what would you do if out-of-memory condition happened? Any attempt to recover may run out of memory itself. For example, years ago I've seen closing a file fail in low-memory conditions.

Additionally, when you are running out of memory, OOM killer is probably already in progress and it may very well choose your application to be killed.

In short, memory allocation errors are the primary candidates for "fail fast" strategy.

"If any call to allocate memory fails, the application is terminated. This also means that there is no need to check if the call succeeded."

"no need" - OMG. So, every single program using GLib can fail at any time, either through its own stupidity or any *other* program's attempt to grab too much RAM. The irony is that the other program may not be using GLib, and could easily survive and go on to use the memory freed up by the GLib program imploding. Which offers an easy to to get more RAM - grab all that's left and wait for a bit voila! More memory.

GLib's memory exhaustion handling is a joke.

Btw. somebody nicked named "jtulach" (and in fact not me), provided effective and encapsulated C++ list on my http wiki(dot)apidesign(dot)org/wiki/Talk:Trait#jtulach_said_…_3
I guess it shows C++ is not as bad as was originally thought.

As for other OO languages, the problem is probably not even applicable as most of them don't let you mess with actual memory layout of your objects.

Finally, in the subsequent blog (250bpm.com/blog:9) I've proposed the "enclave" programming pattern which to some extent mitigates the intrucive container vs. encapsulation problem.

As a side note, it is generally accepted that multiple inheritance can be dangerous if more than one parent is not virtual.

Perhaps a more accurate wording would be that there are situations in which the OO paradigm imposes constraints that has a negative effect on code execution speed and/or memory allocation size. I offer as evidence the necessity to stipulate a very specific case in order to prove your argument.

To balance this out, I think it is fair to say that there are situations in which the OO paradigm provide significantly more benefits than modular/structured programming alone. In particular on projects with N (more the two or three) programmers, a well designed OO program can get you a lot closer to the hypothetical 1/N development time. (I'll state the obvious—badly designed programs make the choice of language and development paradigm a minor issue.)

This has been a very instructive experience—both the article and the resulting posts. I commend both Martin and the majority of the other posters for their informed focus on the issues and not on personalities. This is what good discussions should all be like.

"Imagine you want to store the same person in yet another container. How would you do that without breaking encapsulation?"

Well there are two ways to do that. If the other container is inherently related to my_person and will be handled in the same module, add extra fields to my_person:

struct my_person {
Person person;
struct my_person *next;
RbTreeNodeData tree_node;
};

On the other hand, if it's the *user of my_person* who needs to keep my_person's in another container, use the same technique as before, but on one layer above:

struct my_barking_person {
struct my_person person;
RbTreeNodeData tree_node;
};

I've written a lot of code like this. It's very manageable, and if you have a large "composition hierarchy", you save a lot of memory allocations (and memory).

"
2. No extra allocations should be made during the insert operation.
"
preallocate, or when in STL do as the std::vector does(I guess you know about how vector capacity changes when there is no space for push_back()
)
"
3. Make sure that encapsulation is not broken (i.e. adding the element to a new container doesn't require modifying the implementation of the element in question).
"
Since element cant be copied you mean adding a pointer to an element to a new container ?

Old saying goes you shouldn't try to fix social problems with technical solutions :)

That would imply private/protected/public keywords should be removed from C++ as they solve a social problem rather than a technical one.

Basically, on insert, your container will eventually require some memory. If you do not wish this memory to be outside of the entries, it has to be preallocated within the entries. So either it's inefficient or it break encapsulation. There's no way out.

Bingo!

The Person code in the library does not need to know anything about my_person, so the encapsulation is not broken in this perspective.

Imagine you want to store the same person in yet another container. How would you do that without breaking encapsulation?

Person p;
p.init();
…
p.deinit(); // assuming you don't want constructors/destructors :)

This approach allows a *user* to efficiently keep persons in his own data structures:

struct my_person {
Person p;
struct my_person *next;
};

The Person code in the library does not need to know anything about my_person, so the encapsulation is not broken in this perspective. The downside is it's prone to ABI breaks, but I'm not sure how relevant this is in your case.